Difference between revisions of "User:Eden"

From Apertium
Jump to navigation Jump to search
(2 intermediate revisions by the same user not shown)
Line 11: Line 11:
   
 
== My goal ==
 
== My goal ==
I’m planning to start the ‘English-Lingala’ language pair.
+
I’m planning to start the ‘English-Lingala’ language pair.<br/>
  +
At first I was only planning to work in one direction(eng-lin), but following ''firespeaker'' 's suggestion, I will also work in the other direction to make my mentor's job easier.
   
 
== Why am I interested in Apertium? ==
 
== Why am I interested in Apertium? ==
Line 36: Line 37:
 
'''Update 1: Apr/1/19'''<br/>
 
'''Update 1: Apr/1/19'''<br/>
 
1. Added ~95% of all words from this [https://sourceforge.net/p/apertium/svn/HEAD/tree/branches/xupaixkar/rasskaz/ story]. <br/>
 
1. Added ~95% of all words from this [https://sourceforge.net/p/apertium/svn/HEAD/tree/branches/xupaixkar/rasskaz/ story]. <br/>
2. From the ''493''-word story, my final translation has ''74'' unknown words(*) and ''63'' words with the wrong final form(#). Most of them are verbs, adj and adv. Original story is [https://github.com/thefreezer/GSOC-apertium-eng-lin/blob/master/story.txt here] and [https://github.com/thefreezer/GSOC-apertium-eng-lin/blob/master/output.txt here] is the final output.<br/>
+
2. From the ''493''-word story, my final translation has ''74'' unknown words(*) and ''63'' words with the wrong final form(#). Most of them are verbs, adj and adv. Original story is [https://github.com/thefreezer/GSOC-apertium-eng-lin/blob/master/story_eng.txt here] and [https://github.com/thefreezer/GSOC-apertium-eng-lin/blob/master/eng-lin-output.txt here] is the final output.<br/>(eng-lin)
 
3. Added 8 rules which give me correct translations for:<br/>
 
3. Added 8 rules which give me correct translations for:<br/>
 
* prn/np vblex/vbhaver/vbser det n (eg. I see a house) with correct present and past(saw) verb tenses
 
* prn/np vblex/vbhaver/vbser det n (eg. I see a house) with correct present and past(saw) verb tenses
Line 52: Line 53:
   
 
== Work plan ==
 
== Work plan ==
(this page will frequently change as I get more familiar with Apertium)
 
 
community bonding period
 
community bonding period
- reading more about transfer-rules and creating a doc for Lingala rules
+
- reading more about transfer-rules and creating a doc for eng-lin lin-eng rules
 
- build a better frequency list of Lingala words
 
- build a better frequency list of Lingala words
 
- reading more about the HFST
 
- reading more about the HFST
  +
- continue work and achieve a WER<50% from at least one direction
   
 
Week 1:
 
Week 1:
 
- adding nouns(from frequency list) in the lin transducer
 
- adding nouns(from frequency list) in the lin transducer
- adding verbs in the lin transducer
+
- adding verbs with correct tenses in the lin transducer
 
- constraint grammar
 
- constraint grammar
   
 
Week 2:
 
Week 2:
 
- adding pronouns and adjectives in the lin transducer
 
- adding pronouns and adjectives in the lin transducer
- adding adverbs, conjunctions, prepositions, etc
+
- also adding adverbs, conjunctions, prepositions, etc
 
- constraint grammar for prn and adj
 
- constraint grammar for prn and adj
   
 
Week 3:
 
Week 3:
- polishing the transducer to give better analyses(eg.
+
- polishing the transducer to give better analyses
 
- filling nouns and adjectives in bilingual dictionary,
 
- filling nouns and adjectives in bilingual dictionary,
 
- regression testing
 
- regression testing
   
 
Week 4:
 
Week 4:
- transfer rules for nouns and adjectives
+
- transfer rules for nouns and adjectives(both directions)
 
- disambiguation rules
 
- disambiguation rules
   
Line 82: Line 83:
 
- continue work on bilingual dictionary,
 
- continue work on bilingual dictionary,
 
- main work will be on verbs
 
- main work will be on verbs
- transfer rules for verbs
+
- transfer rules for verbs in both directions
   
 
Week 6:
 
Week 6:
 
- filling pronouns, adverbs, and others in the bidix
 
- filling pronouns, adverbs, and others in the bidix
 
- work on compound Lingala words
 
- work on compound Lingala words
  +
- transfer rules for pronouns, adverbs and compound nouns(both directions)
   
 
Week 7:
 
Week 7:
- adding pronouns, determinants, adverbs, and others in the bidix
+
- adding determinants and more adjectives in the bidix
- WER < 25% on a 500 word story
+
- WER < 35% on a 500 word story
 
- add/polish rules for concordance between verbs and pronouns
 
- add/polish rules for concordance between verbs and pronouns
   
 
Week 8:
 
Week 8:
 
- continue work on transfer rules in .t2x and t3x files
 
- continue work on transfer rules in .t2x and t3x files
- work on disambiguation,
+
- work on disambiguation(eng-lin, lin-eng)
 
- lots of testing and improvement of bilingual dictionary
 
- lots of testing and improvement of bilingual dictionary
- WER < 25% on a 1000 word story
+
- WER < 30% in both directions on a 1000 word story
   
* '''Deliverable #2''' Advanced bilingual dictionary and transfer rules
+
* '''Deliverable #2''' Advanced bilingual dictionary(~5,000 words) and transfer rules
   
 
Week 9 :
 
Week 9 :
- continue work on disambiguation
+
- continue work on disambiguation(both directions)
 
- testvoc and improvements
 
- testvoc and improvements
  +
- filling bidix(common nouns)
   
 
Week 10:
 
Week 10:
 
- work on transfer rules,
 
- work on transfer rules,
- goal is WER < 20% on a story greater > 1000 words(is this achievable?)
+
- goal is WER < 30% on a story greater > 1000 words(is this achievable?)
   
 
Week 11:
 
Week 11:
 
- continue work on transfer rules and testing,
 
- continue work on transfer rules and testing,
 
- Wikipedia article translations
 
- Wikipedia article translations
  +
- continue filling bidix
   
 
Week 12:
 
Week 12:
  +
- filling bidix with miscellaneous words
 
- detailed analysis of work completed(wiki),
 
- detailed analysis of work completed(wiki),
 
- evaluation of results and documentation
 
- evaluation of results and documentation
   
* '''Project completed''' Goal is to have a WER < ~25% on most texts
+
* '''Project completed''' WER~30%(with ~7,000 words in bidix) in both directions on most texts
   
 
== Skills and qualifications ==
 
== Skills and qualifications ==
 
Ongoing major: first year Computer Science students with a minor in Statistics<br />
 
Ongoing major: first year Computer Science students with a minor in Statistics<br />
Relevant technical skills: python(online data mining, inferential statistics, numpy, pandas, matplotlib), c++(proficient), sql(elementary), git(proficient), bash(proficient), html5/css3(advanced)<br />
+
Relevant technical skills: python(online data mining, inferential statistics, numpy, pandas, matplotlib), c++(elementary), sql(intermediate), git(intermediate), bash(intermediate), html5/css3(advanced)<br />
 
Work experience: as an intern created static and dynamic websites<br />
 
Work experience: as an intern created static and dynamic websites<br />
 
Languages: French(native), Lingala(native), English(Fluent) , Swahili(proficient), Tshiluba(proficient), Twi(elementary)<br />
 
Languages: French(native), Lingala(native), English(Fluent) , Swahili(proficient), Tshiluba(proficient), Twi(elementary)<br />
   
 
== Non-Summer-of-Code plans ==
 
== Non-Summer-of-Code plans ==
Traveling to Ontario for 5 days from June 29, but that will not affect my work. I’m committed to put it at least 40+ hours a week for the duration of the project.
+
Traveling to Ontario for 5 days from June 29, but that will not affect my work. No other commitments which will allow me to put it at least 40+ hours a week for the duration of the project.
   
   

Revision as of 07:10, 8 April 2019


Contact Information

Name: Eden-Grace Muamba
Location: Alberta, Canada
University: University of Alberta
E-mail: nzambieden@gmail.com
IRC: eden__
Timezone: UTC -7
Github: https://github.com/thefreezer

My goal

I’m planning to start the ‘English-Lingala’ language pair.
At first I was only planning to work in one direction(eng-lin), but following firespeaker 's suggestion, I will also work in the other direction to make my mentor's job easier.

Why am I interested in Apertium?

Apertium is at the intersection of computers and languages, which are two of my passions. This will be my first ever contribution to an open source project. For the short amount of time I have been on the IRC and the mailing list, the Apertium community has made it a fun and enjoyable experience for me. I hope, not only to develop an English-Lingala pair but also, to become a long-time contributor to Apertium, mainly by creating new English/French-African Language pairs.

Who will benefit and why should it get sponsored

African languages are poorly represented in Apertium and even other commercially available options are usually quite lacking. Given that Lingala, and most African languages do not always have a lot of digitized content accessible, it's hard to use any machine learning or NLP tools to build translators since massive amount of data for these languages do not exist. In such cases, a rule-based MT tool like Apertium becomes the most viable option.

Lingala is a Bantu Language, mainly used as a lingua franca, in central Africa(mainly in the Democratic Republic of Congo and to some extent in Angola and the Republic of Congo) with over 70 million speakers(https://en.wikipedia.org/wiki/Lingua_franca). Developing an English-Lingala pair will, I believe, positively contribute to the technological and economic development of these underserved places. Hopefully this translator will serve a lot of people and organizations. From Wikipedia contributors, to casual users, and to other open source software that might need a Lingala translator.

Lingala resources

Here is a list of open and public domain resources(dictionaries, grammar books, texts, etc) for the Lingala language:
- Crubadan text corpus A text corpus sorted by word frequency
- The excellent, Grammar and dictionary of Bangala
- Universal Declaration of Human Rights - Lingala (tones)
- Lingala. Livre du formatteur Lingala teacher's manual (I will have to confirm if this book is in the public domain)
- Bible and Quran can be used as parallel texts.
- Notions de Lingala - Another dictionary plus common Lingala sentences

Coding challenge

All my work is in my repo: https://github.com/thefreezer/GSOC-apertium-eng-lin
Update 1: Apr/1/19
1. Added ~95% of all words from this story.
2. From the 493-word story, my final translation has 74 unknown words(*) and 63 words with the wrong final form(#). Most of them are verbs, adj and adv. Original story is here and here is the final output.
(eng-lin) 3. Added 8 rules which give me correct translations for:

  • prn/np vblex/vbhaver/vbser det n (eg. I see a house) with correct present and past(saw) verb tenses
  • prn/np vblex/vbhaver/vbser pr det adj n(eg. Mary eats in the beautiful garden)
  • and other rules for dealing with the infitive form of a verb, and handling the pro-drop behavior of the language.

I will try to implement a rule for dealing with the future tense(eg. I will play ...)
Note: a lot of these rules are inspired from the eng-fra pair

It would be good to be able to evaluate WER, so a correct Lingala version of the story would be very useful —Firespeaker (talk) 03:50, 8 April 2019 (CEST).

Update 2: Apr/7/19
1. Added the full Lingala translation here.
2. lin-eng: 75.27% WER
3. eng-lin: 85.65% WER (I mostly focused on lin-eng, which explains why this direction is higher)

Work plan

community bonding period 
- reading more about transfer-rules and creating a doc for eng-lin lin-eng rules
- build a better frequency list of Lingala words
- reading more about the HFST
- continue work and achieve a WER<50% from at least one direction
Week 1: 
- adding nouns(from frequency list) in the lin transducer
- adding verbs with correct tenses in the lin transducer
- constraint grammar
Week 2:
- adding pronouns and adjectives in the lin transducer 
- also adding adverbs, conjunctions, prepositions, etc
- constraint grammar for prn and adj
Week 3:  
- polishing the transducer to give better analyses
- filling nouns and adjectives in bilingual dictionary, 
- regression testing
Week 4:  
- transfer rules for nouns and adjectives(both directions)
- disambiguation rules
  • Deliverable #1 Advanced Lingala transducer with basic bilingual dictionary
Week 5:  
- continue work on bilingual dictionary,
- main work will be on verbs
- transfer rules for verbs in both directions
Week 6:  
- filling pronouns, adverbs, and others in the bidix
- work on compound Lingala words
- transfer rules for pronouns, adverbs and compound nouns(both directions)
Week 7: 
- adding determinants and more adjectives in the bidix
- WER < 35% on a 500 word story
- add/polish rules for concordance between verbs and pronouns
Week 8: 
- continue work on transfer rules in .t2x and t3x files
- work on disambiguation(eng-lin, lin-eng) 
- lots of testing and improvement of bilingual dictionary
- WER < 30% in both directions on a 1000 word story
  • Deliverable #2 Advanced bilingual dictionary(~5,000 words) and transfer rules
Week 9 :
- continue work on disambiguation(both directions)
- testvoc and improvements
- filling bidix(common nouns)
Week 10:
- work on transfer rules, 
- goal is WER < 30% on a story greater > 1000 words(is this achievable?)
Week 11:
- continue work on transfer rules and testing, 
- Wikipedia article translations
- continue filling bidix
Week 12:
- filling bidix with miscellaneous words 
- detailed analysis of work completed(wiki),
- evaluation of results and documentation
  • Project completed WER~30%(with ~7,000 words in bidix) in both directions on most texts

Skills and qualifications

Ongoing major: first year Computer Science students with a minor in Statistics
Relevant technical skills: python(online data mining, inferential statistics, numpy, pandas, matplotlib), c++(elementary), sql(intermediate), git(intermediate), bash(intermediate), html5/css3(advanced)
Work experience: as an intern created static and dynamic websites
Languages: French(native), Lingala(native), English(Fluent) , Swahili(proficient), Tshiluba(proficient), Twi(elementary)

Non-Summer-of-Code plans

Traveling to Ontario for 5 days from June 29, but that will not affect my work. No other commitments which will allow me to put it at least 40+ hours a week for the duration of the project.