Difference between revisions of "User:Aboelhamd/progress"

From Apertium
Jump to navigation Jump to search
Line 101: Line 101:
 
Position-independent word error rate (PER): 58.09 %<br />
 
Position-independent word error rate (PER): 58.09 %<br />
   
---
+
----
   
 
Average score of our system :<br />
 
Average score of our system :<br />
Line 117: Line 117:
 
Position-independent word error rate (PER): 60.89 %<br />
 
Position-independent word error rate (PER): 60.89 %<br />
   
---
+
----
   
 
Statistics about input files<br />
 
Statistics about input files<br />
Line 133: Line 133:
 
Position-independent word error rate (PER): 61.86 %<br />
 
Position-independent word error rate (PER): 61.86 %<br />
   
---
+
----
   
 
''' Day 1 (Friday April 26) '''
 
''' Day 1 (Friday April 26) '''

Revision as of 22:42, 2 May 2019

GSOC 2019 progress : Extend weighted transfer rules

The code is uploaded regularly in this repo [1].


My working days will be everyday except for Thursday, 5 hours per day, at least for the first phase only.


Contents

Phase 1 (April 19 : May 16)

From April 19 to May 16 and from June 21 to June 28.
It's shifted because of my exams.


Week 1 (April 19 : April 25)

Day 1 (Friday April 19)
Latest evaluation scores were lower than traditional apertium's LRLM resolution, by far, unfortunately.
Debugged the code to see what's the cause of such low evaluation score.
Figured out that there was a bug in normalizing the LM scores of the target ambiguous sentences. The LM score is a log base 10 the probability of the sentence, and as the magnitude gets higher, the sentence normalizing probability should get lower, and I was doing the inverse of that.
The easiest solution was to modify the score-sentences script, and instead of getting the score, will get its reciprocal.


Day 2 (Saturday April 20)
The evaluation results were better, but still not better than traditional apertium's LRLM resolution.
Debugged the code to see why the score still worse than apertium's.
Found a bug in generating the ambiguous combinations and still working on solving it.

The bug resulted some incomplete sentences like:
Sentence : Reciprocal of negative LM score
Resumption of the period of sessions : 0.0552026445652
Resumption of session period : 0.0739337147853
Resumption of the session period : 0.0753641871191
Resumption of the period of : 0.0757469108192
Resumption of period of : 0.0684245152522
Resumption of the period : 0.0809504851312

And as shown, the best score went to incomplete sentence, which is one of the reasons why we got bad evaluation score.


Day 3 (Sunday April 21)
The bug of some incomplete sentences was a problem of pointers. Before that, I was depending on pointers and the program was working good but a memory leaks problem emerged when I had done some changes, so instead of solving the leaks problems I turned to use stack instead of the heap, and with not enough testing after that, we got this bug.
Now, I will remove that stack solution again and turn to pointers and try to solve the leaks problem, so the program will work as well as previous version.
The bug was solved, and started training spa-eng pair again.


Day 4 (Monday April 22)
Continue spa-eng training.
Read the paper Neural Machine Translation with Extended Context. [2]


Day 5 (Tuesday April 23)
Continue spa-eng training.
Debugged kir-tur pair for possible bug, as the chunker output has only default and unknown chunks. But still needs more debugging


Day 6 (Wednesday April 24)
Training is finished.
Modifying some of the code regarding evaluation, to update them with the last bug fixed.
Evaluation is finished, and the result scores still lower than apertium's.

Using 100% of training data, 6-gram LM, max entropy models, sampling ambiguous combinations
Apertium LRLM Apertium ambiguous
WER (Word Error Rate) 78.41 76.93
PER (Position-independent word Error Rate) 61.86 57.62
BLEU (Bi-Lingual Evaluation Understudy) 14.13 13.72



Day 7 (Thursday April 25)
Working on training with 10%, 25%, 50%, 75% of the data, to evaluate the results with respect the data size.

Week 2 (April 26 : May 2)

Evaluating with 1000 sentences. The source and all target files are uploaded here[3] .

Best score of our system :

Number of words in reference: 24701
Number of words in test: 25406
Number of unknown words (marked with a star) in test: 1200
Percentage of unknown words: 4.72 %

Results when removing unknown-word marks (stars)

Edit distance: 18675
Word error rate (WER): 75.60 %
Number of position-independent correct words: 11058
Position-independent word error rate (PER): 58.09 %


Average score of our system :

Number of words in reference: 579172
Number of words in test: 622563
Number of unknown words (marked with a star) in test: 26178
Percentage of unknown words: 4.20 %

Results when removing unknown-word marks (stars)

Edit distance: 453716
Word error rate (WER): 78.34 %
Number of position-independent correct words: 269929
Position-independent word error rate (PER): 60.89 %


Statistics about input files

Number of words in reference: 24701
Number of words in test: 26486
Number of unknown words (marked with a star) in test: 1200
Percentage of unknown words: 4.53 %

Results when removing unknown-word marks (stars)

Edit distance: 19369
Word error rate (WER): 78.41 %
Number of position-independent correct words: 11206
Position-independent word error rate (PER): 61.86 %


Day 1 (Friday April 26)

Day 2 (Saturday April 27)

Day 3 (Sunday April 28)

Day 4 (Monday April 29)

Day 5 (Tuesday April 30)

Day 6 (Wednesday May 1)

Day 7 (Thursday May 2)


Week 3 (May 3 : May 9)

Day 1 (Friday May 3)

Day 2 (Saturday May 4)

Day 3 (Sunday May 5)

Day 4 (Monday May 6)

Day 5 (Tuesday May 7)

Day 6 (Wednesday May 8)

Day 7 (Thursday May 9)

Week 4 (May 10 : May 16)

Day 1 (Friday May 10)

Day 2 (Saturday May 11)

Day 3 (Sunday May 12)

Day 4 (Monday May 13)

Day 5 (Tuesday May 14)

Day 6 (Wednesday May 15)

Day 7 (Thursday May 16)

Week 5 (June 21 : June 27)

After my exams.

Day 1 (Friday June 21)

Day 2 (Saturday June 22)

Day 3 (Sunday June 23)

Day 4 (Monday June 24)

Day 5 (Tuesday June 25)

Day 6 (Wednesday June 26)

Day 7 (Thursday June 27)



Phase 2 (June 28 : July 25)

Week 1 (June 28 : July 4)

Day 1 (Friday June 28)

Day 2 (Saturday June 29)

Day 3 (Sunday June 30)

Day 4 (Monday July 1)

Day 5 (Tuesday July 2)

Day 6 (Wednesday July 3)

Day 7 (Thursday July 4)

Week 2 (July 5 : July 11)

Day 1 (Friday July 5)

Day 2 (Saturday July 6)

Day 3 (Sunday July 7)

Day 4 (Monday July 8)

Day 5 (Tuesday July 9)

Day 6 (Wednesday July 10)

Day 7 (Thursday July 11)

Week 3 (July 12 : July 18)

Day 1 (Friday July 12)

Day 2 (Saturday July 13)

Day 3 (Sunday July 14)

Day 4 (Monday July 15)

Day 5 (Tuesday July 16)

Day 6 (Wednesday July 17)

Day 7 (Thursday July 18)

Week 4 (July 19 : July 25)

Day 1 (Friday July 19)

Day 2 (Saturday July 20)

Day 3 (Sunday July 21)

Day 4 (Monday July 22)

Day 5 (Tuesday July 23)

Day 6 (Wednesday July 24)

Day 7 (Thursday July 25)



Phase 3 (July 26 : August 19)

Week 1 (July 26 : August 1)

Day 1 (Friday July 26)

Day 2 (Saturday July 27)

Day 3 (Sunday July 28)

Day 4 (Monday July 29)

Day 5 (Tuesday July 30)

Day 6 (Wednesday July 31)

Day 7 (Thursday August 1)

Week 2 (August 2 : August 8)

Day 1 (Friday August 2)

Day 2 (Saturday August 3)

Day 3 (Sunday August 4)

Day 4 (Monday August 5)

Day 5 (Tuesday August 6)

Day 6 (Wednesday August 7)

Day 7 (Thursday August 8)

Week 3 (August 9 : August 15)

Day 1 (Friday August 9)

Day 2 (Saturday August 10)

Day 3 (Sunday August 11)

Day 4 (Monday August 12)

Day 5 (Tuesday August 13)

Day 6 (Wednesday August 14)

Day 7 (Thursday August 15)

Week 4 (August 15 : August 19)

Day 1 (Friday August 15)

Day 2 (Saturday August 16)

Day 3 (Sunday August 17)

Day 4 (Monday August 18)

Day 5 (Tuesday August 19)