User:Kamush/GSoC2021ProgresReport - Revision history

Kamush: /* Progress Report */

2021-08-31T11:51:53Z

Progress Report

Kamush: /* Progress Report */

2021-08-31T11:50:57Z

Progress Report

Kamush: /* Progress Report */

2021-08-31T11:44:25Z

Progress Report

Kamush: /* Progress Report */

2021-08-01T17:10:46Z

Progress Report

Kamush: /* Progress Report */

2021-08-01T09:54:31Z

Progress Report

Kamush: /* DONE (+Notes & Comments) */

2021-07-25T16:20:36Z

DONE (+Notes & Comments)

Kamush: /* ONGOING */

2021-07-25T15:53:41Z

ONGOING

Kamush: /* TODO */

2021-07-25T15:52:33Z

TODO

Kamush: /* Progress Report */

2021-07-25T11:36:57Z

Progress Report

Kamush: /* Progress Report */

2021-07-25T11:08:22Z

Progress Report

@@ Line 146: / Line 146: @@
 | style = "text-align: center;" | 86.02 %
 | 71.37% / 64.81%
-| 61.93% / 56.05%|
+| 61.93% / 56.05%
+|
 * More post-editing done
 |-

@@ Line 134: / Line 134: @@
 |Focus on testvoc
 | style = "text-align: center;" | +1000
-| style = "text-align: center;" | -
+| style = "text-align: center;" | 86.02 %
+| 71.37% / 64.81%
-| -
+| 61.93% / 56.05%
-| -
-| -
+|
+* More frequent words added to the dictionary
 |-
 |Week 10
@@ Line 143: / Line 144: @@
 |Finalize work
 | style = "text-align: center;" | +400
-| style = "text-align: center;" | -
+| style = "text-align: center;" | 86.02 %
+| 71.37% / 64.81%
-| -
+| 61.93% / 56.05%|
-| -
+* More post-editing done
-| -
 |-
 |}

@@ Line 123: / Line 123: @@
 | style = "text-align: center;" | +500
 | style = "text-align: center;" | 85.17 %
+| 69.21% / 60.69%
-| -
+| 67.10% / 59.89%
-| -
 |
 * Some additions to apertium-kaz & apertium-uzb
@@ Line 133: / Line 133: @@
 August 1-7
 |Focus on testvoc
-| style = "text-align: center;" | -
+| style = "text-align: center;" | +1000
 | style = "text-align: center;" | -
 | -
@@ Line 142: / Line 142: @@
 August 8-14
 |Finalize work
-| style = "text-align: center;" | -
+| style = "text-align: center;" | +400
 | style = "text-align: center;" | -
 | -

@@ Line 126: / Line 126: @@
 | -
 |
+* Some additions to apertium-kaz & apertium-uzb
 * Some transfer rules
-* Some more lexical selection rules
+* Some more lexical selection rules & bidix
-* Some more bidix
 |-
 |Week 9

@@ Line 121: / Line 121: @@
 July 25-31
 |Focus on transfer rules
-| style = "text-align: center;" | -
+| style = "text-align: center;" | +500
-| style = "text-align: center;" | -
+| style = "text-align: center;" | 85.17 %
-| -
 | -
 | -
+|
+* Some transfer rules
+* Some more lexical selection rules
+* Some more bidix
 |-
 |Week 9

@@ Line 159: / Line 159: @@
 == DONE (+Notes & Comments) ==
+* Made a script to calculate WER for each line of the kaz-uzb.txt file to see what sentences are causing the most problems.
+* Translating big Kaz text into Uzb
+** Aside from those small texts obtained from parallel corpora
+** For better WER/PER calculation
+** For checking transfer rules
+** Started with readily available small JaM Story
+*** Azmat and Oygul story in our case
+*** Added 47 sentences from that.
+** Chose Nur-Sultan(capital city) article of Kazakh Wiki for that.
+*** Split the article into sentences, translated into uzb manually.
+*** Added 112 sentences out of Nur-Sultan text.
+** Used QED parallel corpus form internet.
+*** Manually selected sentences and corrected their translations.
+*** Added 90 sentences from this.
+** Made 250 sentences in total, stopping this here.
+* Tried finding parallel corpora:
+** ''https://opus.nlpl.eu/''
+*** KDE4: Not good: ''https://opus.nlpl.eu/KDE4/v2/kk-uz_sample.html''
+*** MozillaI10: Nope
+*** '''QED''':
+**** kk-uz.tmx file
+**** Good, Gonna use this, but needs manual selection and corrections on translations.
+*** TED: the same file with QED.
+*** The rest is also not good, just one-word translation mostly.
+* Lexical selection rules for kaz-uzb
+** Created a script to analyse bidix and find ambiguous stems.
+** Created a script that generates lexical selection rules from manually chosen stems
+** Rules added in kaz-uzb.lrx for 1565 words.
+** Rules added in uzb-kaz.lrx for 1295 words.
+* '''Apertium-dixtools:'''
+** ''https://wiki.apertium.org/wiki/Apertium-dixtools''
+** Fixing bidix(deduplicating, removing empty lines):
+    apertium-dixtools fix apertium-kaz-uzb.kaz-uzb.dix apertium-kaz-uzb.kaz-uzb.dix.fixed
+** Sorting bidix(aligning too):
+    apertium-dixtools sort -alignBidix -ignorecase apertium-kaz-uzb.uz-uzb.dix apertium-kaz-uzb.kaz-uzb.dix.sorted
+* Sorting+Deduplicating the bidix for future purposes:
+** Bidix count before deduplication: 11008
+** Fixing the bidix:
+*** Removed 1501 entries in section main
+** Bidix count after deduplication: 9507
+** Sorted+Regrouped the dix.
 * Made a script to calculate DixCount, Coverage, WER/PER at once
 * '''Calculating WER/PER:'''
 ** Apertium-eval-translator:
 *** ''https://github.com/apertium/apertium-eval-translator''
-    apertium-eval-translator -ref uzb.txt -test kaz-uzb.txt
+    $ cat kaz-big.txt | apertium -d ../ kaz-uzb > kaz-uzb.txt
+    $ cat uzb-big.txt | apertium -d ../ uzb-kaz > uzb-kaz.txt
+    $ apertium-eval-translator -ref uzb-big.txt -test kaz-uzb.txt
+    $ apertium-eval-translator -ref kaz-big.txt -test uzb-kaz.txt
 ** '''Parallel text:'''
 *** JaM Story:

@@ Line 153: / Line 153: @@
 == ONGOING ==
-* Lexical selection rules for kaz-uzb
+* Transfer rules
+* Some lexical selection rules
-* Translating big Kaz text into Uzb
-** For better WER/PER calculation
-** For checking transfer rules
-** Chose Nur-Sultan(capital city) article of Kazakh Wiki for that.
-** Made 112 sentences out of Nur-Sultan.
 * Collecting more bidix
+** Adding only most-frequent words from kazakh wiki hitparade until  85% coverage is reached
 == DONE (+Notes & Comments) ==

@@ Line 148: / Line 148: @@
 == TODO ==
+* More transfer rules
-* Writing lexical selection rules for uzb-kaz
-* Transfer rules
 * Testvoc
 == ONGOING ==

@@ Line 109: / Line 109: @@
 July 18-24
 |Test the kaz-uzb translator
-| style = "text-align: center;" | +193
+| style = "text-align: center;" | +219
-| style = "text-align: center;" | 84.50 %
+| style = "text-align: center;" | 84.58 %
 | 69.21% / 60.69%
 | 67.10% / 59.89%