User:Techievena/Proposal - Revision history

Techievena: Content on the right pane

2018-05-19T04:33:26Z

Content on the right pane

Techievena: /* Contact Information */ Update mail address

2018-04-27T14:08:19Z

Contact Information: Update mail address

Techievena: /* Coding Challenges */ Final Update

2018-03-30T16:27:13Z

Coding Challenges: Final Update

Techievena: /* MILESTONES */ Final Update

2018-03-30T08:45:37Z

MILESTONES: Final Update

Show changes

Techievena: /* Implementation */ Final Update

2018-03-30T07:45:53Z

Implementation: Final Update

Techievena: /* Proposal Abstract */ Final Update

2018-03-30T07:39:32Z

Proposal Abstract: Final Update

Techievena: Final edits and tweaks

2018-03-26T16:21:44Z

Final edits and tweaks

Techievena: /* MILESTONES */ Add additional tweaks

2018-03-19T11:56:19Z

MILESTONES: Add additional tweaks

Techievena: Except timeline tweaks added

2018-03-19T08:17:00Z

Except timeline tweaks added

Show changes

Techievena: /* MILESTONES */ Add Coding Phase - 3 details (2)

2018-03-17T05:34:44Z

MILESTONES: Add Coding Phase - 3 details (2)

@@ Line 1: / Line 1: @@
+{{TOCD}}
 == Contact Information ==
 <b>Name:</b> Abinash Senapati<br/>

@@ Line 2: / Line 2: @@
 <b>Name:</b> Abinash Senapati<br/>
 <b>IRC Nickname:</b> Techievena <br/>
-<b>E-mail address:</b> techievena@users.sourceforge.net <br/>
+<b>E-mail address:</b> abinashsena@gmail.com <br/>
 <b>Website:</b> https://techievena.github.io <br/>
 <b>Github:</b> https://github.com/Techievena <br/>

@@ Line 192: / Line 192: @@
 == Coding Challenges ==
 <ul><li>Install Apertium.</li>
-<li>[https://github.com/Techievena/lexc2dix lexc2dix] → hfst lexc/twolc to monodix++ converter.</li></ul>
+<li>[https://github.com/Techievena/lexc2dix lexc2dix] → hfst lexc/twolc to monodix converter.</li></ul>
 == Other Commitments ==

@@ Line 74: / Line 74: @@
 <li><b>Writing a converter from HFST format to lttoolbox format</b></li>
 <p>There is a close resemblance between the currently used ‘.dix’ format in lttoolbox and the ‘.lexc’ file in HFST format. This is due to the fact that both of these files describe how morphemes in a language are joined together. They are basically the doppelganger of each other and only differ in the [http://wiki.apertium.org/wiki/Lttoolbox_and_lexc#Terminology terminologies] used. We need to parse through the lexc files and store the entries in the form of a python dictionary and rewrite them with appropriate xml tags to construct a corresponding dix file. This module can be further modified once we make any further alterations in the format of lttoolbox to support morphographemics and weights. I am already working on this module as a part of my coding challenge.</p>
+<li><b>Writing a module to support weights in lttoolbox</b></li>
+<p>Weights in lttoolbox will defined in the usual manner, smaller weights implying the better form to generate in case of a free variation. In a weighted finite-state transducer each arc and consequently each path is assigned a weight. The weight of a path in a transducer is usually some combination of the weights of the arcs (using the operators ⊕ and ⊗ for addition or multiplication). In lexc files we define weights for specific arcs in a special section after the continuation class. We need a similar approach in the dix files for assigning weights to the arcs. The weights are helpful in determining the most desirable analyses for a morpheme. We therefore have to come up with a module in lttoolbox to select and combine the paths with the least weights as long there is a valid transduction possible to generate the most appropriate lexicons after morphological analysis.</p>
+<p>As dictionary files in lttoolbox already have different sections, implementing weights in section is much easier than introducing arc-weights in the transducer. In this way we can have heuristics in sections with higher weights, that don't pollute the output in the "good" known analyses.</p>
+<li><b>Assigning weights using gold-standard corpora</b></li>
+<p>Every finite state transducer must specify an initial state, the final weights, arc and epsilon counts per states, an fst type name, the fst's properties, how to copy, read and write the transducer, and the input and output symbol tables. To assign weights to states and arcs in the lttoolbox transducer, an unsupervised training method will be implemented. They will be calculated by using a standard gold-tagged parallel corpora set up fully in a direction we intend to train for and at least until the pre-transfer stage in the opposite direction. A method similar to estimation of rules in lexical selection process will be deployed in the algorithm to train for estimating the weight values in the corpora. The training algorithm will be integrated with the filters present in the openfst library to determine the matches which will be allowed to proceed in composition or handle correct epsilon matching.</p>
 <li><b>Adding format for morphographemics</b></li>
 <p>As currently lttoolbox doesn’t support any kind of rule based character alterations it is practically not possible to write dictionary for languages with complex phonologies. Rules in hfst are declared in a separate text file and hfst-twolc combines lexicons based on these rules to produce a lexical transducer. Twol compilation is quite a complex task with a high tendency for every minute details to fail easily if not handled properly. Thus enabling two level rule formalism in lttoolbox from scratch is not efficient and advisable. We can fork essential features from the currently existing hfst-twolc for enabling rule based morphology in lttoolbox.</p>
@@ Line 80: / Line 85: @@
 <p>A new module in lttoolbox will be necessary for handling the new rule based morphographemics and finally combine the rules with the lexicons to generate results. This module will parse through the rules declared in the dix file and process them with the help of twol formalism backend support provided by hfst library. Finally it has to combine the results to generate a lexical transducer.</p>
 <p>We also need to modify the existing modules in lttoolbox like the lt-expand and the lt-trim in order to generate proper results in sync with the new alterations in lttoolbox.</p>
-<li><b>Adding OpenFST as backend to support weights</b></li>
-<p>OpenFst is a library for constructing, combining, optimizing, and searching weighted finite-state transducers. Weighted finite-state transducers are automata where each transition has an input label, an output label, and a weight. The weights can be used to represent the cost of taking a particular transition. Often a weighted transducer is used to represent a probabilistic model like the n-gram model. FSTs can be optimized by determinization and minimization, models can be applied to hypothesis sets or cascaded by finite-state composition, and the best results can be selected by shortest path algorithms. OpenFst library is a standard C++ template library which can be incorporated easily by including appropriate header files. Thus it can serve as backend in the lttoolbox to support weighted arcs and lemmas. There are a tonne of functions in the library to implement transducer composition, determinization and minimization using weighted acceptor versions of these algorithms.</p>
-<li><b>Assigning weights using gold-standard corpora</b></li>
-<p>Every finite state transducer must specify an initial state, the final weights, arc and epsilon counts per states, an fst type name, the fst's properties, how to copy, read and write the transducer, and the input and output symbol tables. To assign weights to states and arcs in the lttoolbox transducer, an unsupervised training method will be implemented. They will be calculated by using a standard gold-tagged parallel corpora set up fully in a direction we intend to train for and at least until the pre-transfer stage in the opposite direction. A method similar to estimation of rules in lexical selection process will be deployed in the algorithm to train for estimating the weight values in the corpora. The training algorithm will be integrated with the filters present in the openfst library to determine the matches which will be allowed to proceed in composition or handle correct epsilon matching.</p>
 <li><b>Backporting</b></li>
 <p>As of now there is no concrete prototype of what the new dix file and lttoolbox will look like after the completion of this project, but it is quite evident that changes in the codebase and the new format will be drastic after addition of the new features. Lttoolbox is an integral part of Apertium and we cannot afford to generate errors during any kind of morphological analysis and generation process using this module. For that we have to ensure that lttoolbox remains compatible with the older version of dix files. Thus, we must backport all the necessary changes to previous versions of lttoolbox for the smooth functioning of the lttoolbox module.</p>

@@ Line 53: / Line 53: @@
 === <u>Proposal Abstract</u> ===
-<p>The aim of this project is to implement the support for morphographemics and weights in the lttoolbox transducer. The proposal focuses on extending lttoolbox to perform the complex morphological transformations currently done in HFST and writing a module that translates the current HFST format to the new lttoolbox format.</p>
+<p>The aim of this project is to implement the support for morphographemics and weights in the lttoolbox transducer. The proposal focuses on extending lttoolbox to perform the complex morphological transformations and weight based analyses currently done in HFST and writing a module that translates the current HFST format to the new lttoolbox format.</p>
-<p><b>Possible Mentor(s):-</b> [http://wiki.apertium.org/wiki/User:Mlforcada User:Mlforcada], [http://wiki.apertium.org/wiki/User:TommiPirinen User:TommiPirinen], [http://wiki.apertium.org/wiki/User:Unhammer User:Unhammer]<br/>
+<p><b>Possible Mentor(s):-</b> [http://wiki.apertium.org/wiki/User:Francis_Tyers User:Francis Tyers], [http://wiki.apertium.org/wiki/User:TommiPirinen User:Tommi Pirinen], [http://wiki.apertium.org/wiki/User:Unhammer User:Unhammer]<br/>
 <b>Language(s) Used:-</b>  C++, Python, XSLT, XML</p>

@@ Line 38: / Line 38: @@
 == Interest in machine translation ==
-<p>It is quite uncomfortable to regard language just as a communication method. Huge experiences, history and views of the world human being have been soaked deeply into a language since the beginning of the world. I come from country that houses over 1600 languages of which around 150 have a sizeable speaking population even today. Most of them are on the verge of extinction and with ordered morphological rules, rule based machine translation is the most promising and viable option for their restoration.</p>
+<p>It is quite uncomfortable to regard language just as a method for communication. Huge experiences, history and views of the world human being have been soaked deeply into a language since the beginning of the world. I come from country that houses over 1600 languages of which around 150 have a sizeable speaking population even today. Most of them are on the verge of extinction and with ordered morphological rules, rule based machine translation is the most promising and viable option for their restoration.</p>
 <p>As I have taken up Formal Languages and Automata Theory course this semester in my university, I am familiar with the concepts of Deterministic Finite Automata, Non-deterministic Finite Automata, Weighted Transducers, Finite State Machines and other concepts essential to take up a project in machine translation.</p>
@@ Line 85: / Line 85: @@
 <p>Every finite state transducer must specify an initial state, the final weights, arc and epsilon counts per states, an fst type name, the fst's properties, how to copy, read and write the transducer, and the input and output symbol tables. To assign weights to states and arcs in the lttoolbox transducer, an unsupervised training method will be implemented. They will be calculated by using a standard gold-tagged parallel corpora set up fully in a direction we intend to train for and at least until the pre-transfer stage in the opposite direction. A method similar to estimation of rules in lexical selection process will be deployed in the algorithm to train for estimating the weight values in the corpora. The training algorithm will be integrated with the filters present in the openfst library to determine the matches which will be allowed to proceed in composition or handle correct epsilon matching.</p>
 <li><b>Backporting</b></li>
-<p>As of now there is no concrete prototype of what the new dix file and lttoolbox will look like after the completion of this project, but it is quite evident that changes in the codebase and the new format will be drastic after addition of the new features. Lttoolbox is an integral part of Apertium and we cannot afford this module to generate errors during morphological analysis and generation. Thus, we have to ensure that lttoolbox remains compatible with the older version of dix files.</p>
+<p>As of now there is no concrete prototype of what the new dix file and lttoolbox will look like after the completion of this project, but it is quite evident that changes in the codebase and the new format will be drastic after addition of the new features. Lttoolbox is an integral part of Apertium and we cannot afford to generate errors during any kind of morphological analysis and generation process using this module. For that we have to ensure that lttoolbox remains compatible with the older version of dix files. Thus, we must backport all the necessary changes to previous versions of lttoolbox for the smooth functioning of the lttoolbox module.</p>
 <li><b>Documentation</b></li>
 <p>When working in a collaborative environment, maintaining proper documentation is very important. Also users and new developers rely upon documentation tutorials and wiki to learn about the changes in codebase. Therefore a proper documentation which includes proper comments in between the code as well, has to be written that will help users and developers familiarize with the the new developments in the lttoolbox and their usage.</p>
-<p>The previously written wiki  and documentation will also require a proper review and may be a rewrite in case of any significant modifications in the features.</p>
+<p>The previously written wiki and documentation will also require a proper review and may be a rewrite in case of any significant modifications in the features.</p>
 <li><b>Testing and CI builds</b></li>
 <p>In software development and open source projects, automated testing is very important to maintain the huge codebase. A 100% code coverage policy for tests must be followed for having a completely error-free and efficient code. Apertium also uses Travis CI for continuous integration builds after its migration to Github platform. Test modules allow maintainers to keep a proper track of the existing codebase and prevent any non-mergeable code getting merged into main code-base, to  prevent build errors. Testing modules are therefore important and hence it is necessary to write tests for the addition of any chunk to the codebase.</p>
@@ Line 141: / Line 141: @@
 <ul>
 <li>Uptil now we mostly have a basic backend framework for playing with weights. We need to modify these tools and make it robust enough by debugging and handling error cases.</li>
-<li>The changes about the newly implemented OpenFST backend will be blogged during this period. Also documentation work has to be done during this period to let users know what aspect-classes are all about.</li>
+<li>The changes about the newly implemented OpenFST backend will be blogged during this period. Also documentation work has to be done during this period to let users know what the new OpenFST backend support for lttoolbox is all about.</li>
 </ul>
 <li><b>Week 7 (June 25 - July 1)</b></li>

@@ Line 109: / Line 109: @@
 <ul>
 <li>In this week, most of the work will be forking the essential features from hfst to be used for handling twol formalism in the lttoolbox.</li>
-<li>As hfst already has a working implementation of two level rule formalism, we can use it as library and build the twol implementation in lttoolbox on the top of it.</li>
+<li>As hfst already has a working implementation of two level rule formalism, we can use it as a library and build the twol implementation in lttoolbox on the top of it.</li>
+<li>The format of writing rules in the dictionary files and symbols as discussed in the community bonding period will be put on a test run to check its viability.</li>
 </ul>
 <li><b>Week 2 (May 21 - May 27)</b></li>
 <ul>
-<li>Now we need some kind of validation in lttoolbox for the new additions in the dix files, to prevent unwanted additions. We can make appropriate changes in the apertium-lint and in the apertium validation script to check the format of dictionary.</li>
+<li>Now we need some kind of validation in lttoolbox for the new additions in the dix files, to prevent unwanted additions. We can make appropriate changes in the apertium validation script to check the format of dictionary.</li>
 <li>Proper debugging and refinements will be done to have a well functioning twol handling module in lttoolbox.</li>
 <li>Well by the end of the second week, the refined version of the module for handling rules in lttoolbox transducer will be available. The GSoC blog will be written describing about this module and how it is going to help.</li>
@@ Line 120: / Line 121: @@
 <ul>
 <li>Now that we have a module for handling rules in lttoolbox, we need to align it properly with the semantics of lttoolbox.</li>
-<li>A proper parsing library has to be written to parse through the definition of the rules in the dix file and generate lexicons corresponding to them.</li>
+<li>A proper parsing library has to be written to parse through the definition of the rules in the dictionary file and generate lexicons corresponding to them.</li>
 </ul>
 <li><b>Week 4 (June 04 - June 10)</b></li>
@@ Line 158: / Line 159: @@
 <li><b>Week 9 (July 9 - July 15)</b></li>
 <ul>
-<li>We need to extend the utility of weights to rule-based analysis. The training for weights will be extended to rules in the new lttoolbox. Also there will be rules for allocating weights to stuffs in the monolingual dictionaries of Apertium.</li>
+<li>We now need to extend the utility of weights to rule-based analysis. The training algorithm for weights will be extended to rules in the new lttoolbox. Also there will be rules for allocating weights to stuffs in the monolingual dictionaries of Apertium.</li>
 <li>Proper test modules will be defined to reduce the likelihood of errors in the codebase.</li>
 </ul>
@@ Line 164: / Line 165: @@
 <ul>
 <li>To inform all the users and developers about all the major changes, they have to be well documented. So this week will be all about maintaining and having new documentations explaining the changes in the codebase. There will also be a tutorial explaining how to write and modify the new dix files, i.e. Starting a language with lttoolbox wiki will be modified.</li>
-<li>All the updates required in the will be done during this period, and all the previously written documentation will be reviewed.</li>
+<li>All the updates required in the wiki will be done during this period, and all the previously written documentation will be properly reviewed.</li>
 <li>The GSoC blog will also be maintained and all the exciting stuffs related to this project will be written down.</li>
 </ul>
@@ Line 170: / Line 171: @@
 <ul>
 <li>This week’s work will mostly be about adding backward compatibility and backporting the new changes in lttoolbox to make it compatible with the previous versions of Apertium pipeline.</li>
-<li>There are high chances that these additions will lead to compilation errors in the Apertium pipeline. Any new chunk of code causing any kind of errors and conflict with the previous format of lttoolbox will be examined thoroughly and fixed.</li>
+<li>There are high chances that these additions may lead to compilation errors in the Apertium pipeline. Any new chunk of code causing any kind of errors and conflict with the previous format of lttoolbox will be examined thoroughly and fixed.</li>
 </ul>
 <li><b>Week 12 (July 30 - August 5)</b></li>
 <ul>
-<li>All the work related to backporting and maintaining backwards compatibility will be completed in this week.</li>
+<li>All the work related to backporting and maintaining backwards compatibility will be completed by this week.</li>
 <li>This week will basically be the pre-final buffer week. All the previously made commits will be merged into master branch. Any minor changes if required for debugging or enhancement, will be made during this period.</li>
 <li>The final GSoC blog will be written explaining all about my work and commits made throughout the GSoC time period.</li>
@@ Line 185: / Line 186: @@
 </ul>
 <p>GSoC timeline is all about commitment and I really understand that doing things properly is more important than doing more things.</p>
-<p>Therefore I didn’t mention to support weights throughout the pipeline, analysis, disambiguation, lexical transfer/selection, translation, chunking in my GSoC timeline. The addition of weights will be mostly basic and will be just for monolingual dictionaries. After discussions with my mentors I will try to improve my timeline by including some more tasks or scraping off some, to make the best out the GSoC period.</p>
+<p>Therefore I didn’t mention to support weights throughout the pipeline, analysis, disambiguation, lexical transfer/selection, translation, chunking in my GSoC timeline. The addition of weights will be mostly basic and limited to the monolingual dictionary formats only. After discussions with my mentors I will try to improve my timeline by including some more tasks or scraping off some, to make the most out the GSoC period.</p>
 == Coding Challenges ==