Difference between revisions of "User:Amanmehta/Application"

From Apertium
Jump to navigation Jump to search
Line 55: Line 55:


=== Detailed week-wise workplan ===
=== Detailed week-wise workplan ===
{| style="border-spacing:0;width:18.627cm;"
|- style="border:1pt solid #000001;padding:0.176cm;"
| align=center| W
| align=center| Dates
| align=center| Goals
| align=center| Eval
| align=center| Accomplishments
| align=center| Notes
|- style="border:1pt solid #000001;padding:0.176cm;"
||
|| Post application period

[March 26 - May 3]
|| <div style="margin-left:0.661cm;margin-right:0cm;">*Mar-hin: </div>
** <div style="margin-left:1.296cm;margin-right:0cm;">Coding challenge: Finish with good WER (target ~65%)</div>
** <div style="margin-left:1.296cm;margin-right:0cm;">Get and analyse frequency list</div>
** <div style="margin-left:1.296cm;margin-right:0cm;">Add transfer rules (t1x) in decreasing frequency</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Learn about the scope of lttoolbox and other relevant tools and get acquainted</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Analyse opportunity to improve dictionaries (tag editing/expand dictionaries)</div>

||
||
||
|- style="border:1pt solid #000001;padding:0.176cm;"
||
|| Community Bonding Period

[May 5 - May 30]
|| * <div style="margin-left:0.661cm;margin-right:0cm;">Read documents thoroughly</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Study CG and rules for mar-hin.</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Discuss strategies/solutions with mentor to tackle loopholes.</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Add rules for transitive and intransitive verbs</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Prepare a corpora</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Testvoc and document state of the pair</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Add transfer rules (t1x)</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Test for (@,#,*) tokens</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">First regression testing.</div>

||
||
||
|- style="border:1pt solid #000001;padding:0.176cm;"
|| 1
|| May 30 - June 5
|| * <div style="margin-left:0.661cm;margin-right:0cm;">Even up nouns.</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Testvoc clean for noun</div>

* <div style="margin-left:0.661cm;margin-right:0cm;">Start working for pronouns, adverbs and adjectives</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Add appropriate rules/stems.</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Start working with chunking (t2x)</div>

||
||
||
|- style="border:1pt solid #000001;padding:0.176cm;"
|| 2
|| June 6 - June 12
|| * <div style="margin-left:0.661cm;margin-right:0cm;">Post-edit translated texts. Analyze and look for common rules and add rules</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Testvoc clean for all classes</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Corpus test, measure improvement, targets: </div>
** <div style="margin-left:1.296cm;margin-right:0cm;">Coverage ~67%</div>
** <div style="margin-left:1.296cm;margin-right:0cm;">WER <25%</div>

||
||
||
|- style="border:1pt solid #000001;padding:0.176cm;"
|| 3
|| June 13 - June 19
|| * <div style="margin-left:0.661cm;margin-right:0cm;">Add more rules</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Finish with t2x.</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Get done with t3x and t4x.</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Read CG, lrx.</div>

||
||
||
|- style="border:1pt solid #000001;padding:0.176cm;"
|| 4
|| June 20 - June 26
|| * <div style="margin-left:0.661cm;margin-right:0cm;">Get most frequent mistakes at this stage</div>

* <div style="margin-left:0.661cm;margin-right:0cm;">If measurable improvement could be made, make required plan</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Run corpus testing to analyse to improvement. Target :</div>
** <div style="margin-left:1.296cm;margin-right:0cm;">Coverage ~70%</div>
** <div style="margin-left:1.296cm;margin-right:0cm;">WER <= 20%</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Run final tests, look for major mistakes and scope for corrections</div>

||
||
||
|- style="border:1pt solid #000001;padding:0.176cm;"
|| 5
|| June 27 - July 3
|| * <div style="margin-left:0.661cm;margin-right:0cm;">Update documentation</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Run tests again, release this version ('''Deliverable #1''')</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Setup skeleton for mar-guj</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Start with expanding apertium-guj dictionary</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Improve morphological analyzer if possible</div>

||
||
||
|- style="border:1pt solid #000001;padding:0.176cm;"
|| 6
|| July 4 - July 10
|| * <div style="margin-left:0.661cm;margin-right:0cm;">Find good parallel corpora from wikipedia and add words and rules in decreasing frequency, especially in apertium-guj. (target ~500 entries)</div>
** <div style="margin-left:1.296cm;margin-right:0cm;">Continue for guj-hin bilingual dictionary expansion parallely.*</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Even up nouns</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Testvoc nouns</div>

||
||
||
|- style="border:1pt solid #000001;padding:0.176cm;"
|| 7
|| July 11 - July 17
|| * <div style="margin-left:0.661cm;margin-right:0cm;">Even up verbs</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Even up adjectives, adverbs</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Add transfer rules</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Work over a ~500 word story for better analysis.</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Calculate WER, PER</div>

||
||
||
|- style="border:1pt solid #000001;padding:0.176cm;"
|| 8
|| July 18 -

July 24
|| * <div style="margin-left:0.661cm;margin-right:0cm;">Evaluate the system and identify most frequent errors</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Improve mar-guj bilingual dictionary adding to its coverage. </div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Add more transfer rules for verbs, adjectives, adverbs and pronouns, Target:</div>
** <div style="margin-left:1.296cm;margin-right:0cm;">Coverage (~60%)</div>
** <div style="margin-left:1.296cm;margin-right:0cm;">WER (<=30%)</div>

||
||
||
|- style="border:1pt solid #000001;padding:0.176cm;"
|| 9
|| July 25 - July 31
|| * <div style="margin-left:0.661cm;margin-right:0cm;">Testvoc cleanup for all classes</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Work over a large corpora</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Regression and corpus testing</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Work on CG and disambiguation</div>

||
||
||
|- style="border:1pt solid #000001;padding:0.176cm;"
|| 10
|| August 1 - August 6
|| * <div style="margin-left:0.661cm;margin-right:0cm;">Continue working on disambiguation and its solutions.</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Add required transfer/lexical rules to improve WER, PER.</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Begin with chunking, t3x and t4x.</div>

||
||
||
|- style="border:1pt solid #000001;padding:0.176cm;"
|| 11
|| August 7 - August 13
|| * <div style="margin-left:0.661cm;margin-right:0cm;">Add words and rules, starting with most frequent words (in decreasing frequency) for '''guj-hin pair.'''</div>

* <div style="margin-left:0.661cm;margin-right:0cm;">Get most frequent mistakes for '''mar-guj pair'''</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Begin final testing for '''mar-guj pair'''</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Check for any major feedbacks to '''mar-hin pair''', work over it.</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">User acceptance testing, gisting evaluation.</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Mar-guj & mar-hin pairs ready for or close to trunk and guj-hin improved. '''(Deliverable #2 and #3)'''</div>

||
||
||
|- style="border:1pt solid #000001;padding:0.176cm;"
|| 12
|| August 14 - August 17
|| * <div style="margin-left:0.661cm;margin-right:0cm;">Documentation, final release of all three pairs.</div>
* <div style="margin-left:0.661cm;margin-right:0cm;">Detailed analysis on what further improvement could be made for the pairs, for future help to apertium.</div>

||
||
||
|-
|}

Revision as of 11:35, 2 April 2017

Contact details

Name: Aman Mehta
E-mail: amanmehta1997@gmail.com
Svn: aman-mehta
IRC-nick: amanmehta
Mobile: +91 8329139961
Timezone: UTC+05:30
Github link: https://github.com/amanmehta-maniac
I stay online on IRC for most of my time so as to be easily accessible


Interest in machine translation

I am passionate about computers. Automation of tasks such as translation fascinates me. The core problem that translation of a text from one language to other can’t be solved by simple substitution of words, catches my interest. The idea of building a translation system and automating translation intrigues me. As MT gives people opportunity to access knowledge in multiple languages, it can play a pivotal role in education for all mission, not only in India but also across the globe. It adheres to the idea that knowledge should be free and accessible to all, which even I believe in strongly. Working on and around machine translation, serves my interest as well as my motivation.

Interested published tasks and project goals

I plan to “Adopt an unreleased language pair”, or to be precise, three language pairs: mar-hin, guj-hin, mar-guj. Mar-hin and guj-hin pairs are in incubator and mar-guj pair is still unreleased. My goal is to bring incubator pair: mar-hin and an unreleased pair: mar-guj, both to release quality. I also plan on expanding dictionaries for guj-hin pair and make further improvements to coverage and WER to the extent possible.


Interest in Apertium

Given my interest in machine translation, I decided to contribute to Apertium and enjoy adding my contribution to Apertium. I developed my interest in Apertium project in last couple of months during which I spent my time on resolving few svn bugs as well as on improving mar-hin pair. It is, at present, one of the best open-source machine translation platforms. Spending my summer to work for this platform would give me an opportunity to add my contribution in an area that fascinates me.


Reasons for Google and Apertium to sponsor

The mar-hin and mar-guj pairs can be brought to a production quality without much effort due to lexical similarities. I am very well acquainted with apertium as well as with the language pairs I am proposing to work on. The odds of finding a polyglot who could add these pairs to Apertium in a single summer would probably be low. If successful, this would add a couple of more language pairs to Apertium which would triple the number of Indian language pairs. The release of these pairs could also help Apertium in expanding language pairs for many other Indian languages. It has been ~2 months since I have joined Apertium and I am very much familiar with it. I have fixed quite a few bugs on svn. I have been working around mar-hin pair and I have been successful in adding coverage for adverbs and adjectives by scraping <avy> tags. It has been around a month and hence I have a very good gist on what all is needed to bring this pair to release quality. For detailed information about my tasks completed, refer to the section “Tasks completed till date”.


Who it will benefit in society and how

  • Who?
    • Over 70 million Marathi speakers
    • Over 50 million Gujarati speakers
    • People belonging to non-native lingual state
      • Eg: A gujarati speaker in Maharashtra (like myself)
  • How?
    • Translator available to learn languages
    • Access to Hindi information
      • Hindi media/newspapers
    • Improved coverage of Hindi books to Marathi and Gujarati and vice-versa.

Eventually helping people of different native languages to share space and reduce communication gap.


Workplan

I plan to work on mar-hin pair for which I have already started working. My goal is to develop mar-hin pair to close to release quality for roughly the first month. Mar-hin pair is already decent in the mono dictionaries and morphological analyzers and hence in this one month I would focus on bilingual dictionary, building a good translator, adding transfer and lexical selection rules.

  • Target WER<=20%
  • Target coverage~70%

For the remaining two months, my main focus will be to release the unreleased mar-guj pair and develop it to close to release quality and parallely expand and improve dictionaries for guj-hin pair, as much as time permits.
For mar-guj pair:

  • Target WER <=20%
  • Target coverage~65%

Detailed week-wise workplan