User:GD/proposal - Revision history

GD: /* Which of the published tasks am I interested in? What do I plan to do? */

2018-05-06T11:49:53Z

Which of the published tasks am I interested in? What do I plan to do?

GD: /* Which of the published tasks am I interested in? What do I plan to do? */

2018-03-26T09:12:22Z

Which of the published tasks am I interested in? What do I plan to do?

GD: /* Which of the published tasks am I interested in? What do I plan to do? */

2018-03-26T07:29:04Z

Which of the published tasks am I interested in? What do I plan to do?

GD: /* Which of the published tasks am I interested in? What do I plan to do? */

2018-03-26T07:27:22Z

Which of the published tasks am I interested in? What do I plan to do?

GD: /* Which of the published tasks am I interested in? What do I plan to do? */

2018-03-25T11:24:12Z

Which of the published tasks am I interested in? What do I plan to do?

GD: /* Which of the published tasks am I interested in? What do I plan to do? */

2018-03-25T11:17:38Z

Which of the published tasks am I interested in? What do I plan to do?

GD: /* Am I good enough? */

2018-03-25T11:16:23Z

Am I good enough?

GD: /* Which of the published tasks am I interested in? What do I plan to do? */

2018-03-25T10:51:02Z

Which of the published tasks am I interested in? What do I plan to do?

GD: /* Which of the published tasks am I interested in? What do I plan to do? */

2018-03-25T10:18:46Z

Which of the published tasks am I interested in? What do I plan to do?

GD: /* Why Google and Apertium should sponsor it? How and who it will benefit in society? */

2018-03-25T09:47:56Z

Why Google and Apertium should sponsor it? How and who it will benefit in society?

@@ Line 88: / Line 88: @@
 The '''result''' of this work will be a tool that can check dictionaries and find new word-pairs that can be included in bidix. And generate insertions for dictionaries.
-See some examples of my ideas in [https://github.com/dkbrz/GSoC_2018/blob/master/Coding%20challenge.ipynb '''my Python notebook''']
+See some examples of my ideas in [https://github.com/dkbrz/GSoC_2018/blob/master/Proposal/Coding%20challenge.ipynb '''my Python notebook''']
 And there is a graph of released language pairs that shows possible ways of translation via other languages:

@@ Line 89: / Line 89: @@
 See some examples of my ideas in [https://github.com/dkbrz/GSoC_2018/blob/master/Coding%20challenge.ipynb '''my Python notebook''']
+And there is a graph of released language pairs that shows possible ways of translation via other languages:
+[[File:language_graph.png]]
 = Proposal =

@@ Line 87: / Line 87: @@
 The '''result''' of this work will be a tool that can check dictionaries and find new word-pairs that can be included in bidix. And generate insertions for dictionaries.
+See some examples of my ideas in [https://github.com/dkbrz/GSoC_2018/blob/master/Coding%20challenge.ipynb '''my Python notebook''']
 = Proposal =

@@ Line 62: / Line 62: @@
 '''Subgraphs''' A general graph consists of lots of connectivity components so while searching we need to take into account only a part of it. It really increases efficiency.
+'''Directed graphs'''
+<ul>
+<li> take into account LR and RL only cases </li>
+<li> avoid some cycles </li>
+<li> we can use directed in-edges for target language in translation subgraph to define it as a finite state  in finite-state machine. So we do not go outside the node because we have already found our translation (a simple path from word to target language word)</li>
+</ul>
+The last one is very important because it turned out that there is an endless loop problem that coul be solved by subgraphing but this is unefficient comparing to finite state solution for various resons: big n potential and logically it seems to be more natural.
 '''Vectorization''' Vectorizing functions and avoiding cycles really affects efficiency.
@@ Line 72: / Line 82: @@
 <li> get accuracy comparing with existing translations </li>
 </ul>
 So after running on different language pair, we get plenty of data to choose one or a composition.

@@ Line 74: / Line 74: @@
 So after running on different language pair, we get plenty of data to choose one or a composition.
+The '''result''' of this work will be a tool that can check dictionaries and find new word-pairs that can be included in bidix. And generate insertions for dictionaries.
 = Proposal =

@@ Line 32: / Line 32: @@
 '''List of main ideas:'''
 <ul>
-<li>Use classes to created the most appropriate type of information</li>
+<li>Use classes to create the most appropriate type of information</li>
 <li>Work with subraphs (connectivity components) to reduce the complexity of calculations</li>
 <li>Filtration algorithms to gain previous aim</li>
 <li>Vectorization to increase efficiency of all functions</li>
-<li>Developing didfferent metrics to reach quality of translation</li>
+<li>Developing different metrics to reach quality of translation</li>
 <li>Evaluation of these metrics </li>
@@ Line 59: / Line 59: @@
 </ul>
-'''Filtration''' Filtration is necesary to filter sets of word by their paramenters (in most cases, POS and language).
+'''Filtration''' Filtration is necessary to filter sets of word by their parameters (in most cases, POS and language).
 '''Subgraphs''' A general graph consists of lots of connectivity components so while searching we need to take into account only a part of it. It really increases efficiency.

@@ Line 14: / Line 14: @@
 <li> Programming (Python, R, Flask, HTML, xml, Machine Learning) </li>
 <li> Morphology, Syntax, Semantics, Typology/Language Diversity </li>
-<li> Mathematics (Discrete Mathemathics, Linear Algebra and Calculus, Probability Theory, Mathematical Statistics, Computability and Complexity, Logic, Graphs and Topology, Theory of Algorithms)</li>
+<li> Mathematics (Discrete Mathematics, Linear Algebra and Calculus, Probability Theory, Mathematical Statistics, Computability and Complexity, Logic, Graphs and Topology, Theory of Algorithms)</li>
 <li> Latin, Latin in modern Linguistics, Ancient Literature </li>
 </ul>

@@ Line 24: / Line 24: @@
 == Which of the published tasks am I interested in? What do I plan to do? ==
-I want to work on [http://wiki.apertium.org/wiki/Ideas_for_Google_Summer_of_Code/Bilingual_dictionary_enrichment_via_graph_completion Graph dictionaries]
+I would like to work on [http://wiki.apertium.org/wiki/Ideas_for_Google_Summer_of_Code/Bilingual_dictionary_enrichment_via_graph_completion Bilingual dictionary enrichment via graph completion]
+The main idea is to take a graph representation of dictionaries and create tools to work on translation via edges between words in this graph. Graphs are very hard to work on because the complexity of calculations is high. But there are some tools and libraries that are created specially for these purposes and are effective. The developer task is to apply these instruments to specific type of dictionary information.
+I worked with NetworkX as it is fully available for my current Windows, but I plan to work with Graph-tool that is much more efficient with large graphs.
+'''List of main ideas:'''
+<ul>
+<li>Use classes to created the most appropriate type of information</li>
+<li>Work with subraphs (connectivity components) to reduce the complexity of calculations</li>
+<li>Filtration algorithms to gain previous aim</li>
+<li>Vectorization to increase efficiency of all functions</li>
+<li>Developing didfferent metrics to reach quality of translation</li>
+<li>Evaluation of these metrics </li>
+'''Word object'''. Basic elements are lemma, language and POS information. Representation and String format can be modified according to developer needs. This one is like 'EN_first_adj' to check output of functions. One of important
+    class Word:
+        def __init__(self, lemma, lang, pos):
+            self.lemma = lemma
+            self.lang = lang
+            self.pos = pos
+        def __str__(self):
+            return (str(self.lang)+'_'+str(self.lemma)+'_'+str(self.pos))
+        __repr__ = __str__
+        def __eq__(self, other):
+            return self.lemma == other.lemma and self.lang == other.lang and self.pos == other.pos
+        def __hash__(self):
+            return hash(str(self))
+</ul>
+'''Filtration''' Filtration is necesary to filter sets of word by their paramenters (in most cases, POS and language).
+'''Subgraphs''' A general graph consists of lots of connectivity components so while searching we need to take into account only a part of it. It really increases efficiency.
+'''Vectorization''' Vectorizing functions and avoiding cycles really affects efficiency.
 = Proposal =

@@ Line 31: / Line 31: @@
 <p> I think there is a lot of math in language and graph representation of dictionaries is an exciting idea, because it adds some kind of cross-validation and internal system source of information. This information help to fill some lacunae that appear while creating a dictionary. This will improve a quality of translation as we manage to expand bidix. </p>
 <p> Graph representation is very promising because it represents a philosophical model of a metalanguage knowledge. Knowing several languages, I know that it could be hard to recall some rare word and it is easier to translate from French to English and only then to Russian - because I forgot the word-pair between Russian and French. This graph representation works just like my memory: we cannot recall what is this word from L1 in L2. Hmm, we know L1-L3 and L3-L2. Oh, that's the link we need. Now we know L1-L3 word-pair. So, as we work on '''natural''' language processing, let's use  '''natural''' instruments and systems as well.</p>
+<p>
+The main benefit of this project is reducing human labor and automatization of part of the dictionary development.
+<ul>
+  <li>Finding lacunae in created dictionary (what words are missing).</li>
+  <li>Dictionary enrichment based on algorithm that offer variants and evaluation of these variants.</li>
+  <li>A potential base for creating new pairs.</li>
+</ul>
+</p>
 == Coding Challenge ==