User:Elmurod1202/GSoC2020Progress

From Apertium
Jump to navigation Jump to search

The Project Proposal can be seen here

The Final Report can be seen here

Status table

Week Stems Tur-Uzb Naïve Coverage Progress
№ Dates uzb tur-uzb WER PER uzb tur-uzb Evaluation Notes
0 May 4 - May 31 34375 2412 90.80 % 81.60 % 89.57 % 72.14 % Initial evaluation As of the end of May
5 June 29 - July 5 34373 2445 84.45 % 76.80 % 90.23 % 72.14 % First Evaluation End of June - ~July 3
9 July 27 - Aug 2 34424 4191 78.70 % 68.34 % 90.23 % 72.74 % Second Evaluation As of July 31 - Aug 1
10 July 3 - Aug 9 35621 5639 78.70 % 68.64 % 90.28 % 80.14 % Weekly evaluation Week #10
11 Aug 10 - Aug 16 37649 8154 78.70 % 68.64 % 90.46 % 83.08 % Weekly evaluation Week #11
12 Aug 17 - Aug 23 57406 13023 78.70 % 68.64 % 90.91 % 86.02 % Weekly evaluation Week #12
13 Aug 24 - Aug 30 58757 12861 78.70 % 68.64 % 90.94 % 86.03 % Final evaluation As of Aug 31

Apertium Notes

TODO

  • TESTVOC
  • Writing script to automatically make Lexc rules(for entries in bidix)

ONGOING

  • Insert entries from the Word Frequency List
  • Checking and correcting the entire bidix
  • Work on Lexical Selection rules.
  • Review: Nouns
  • Review: Proper nouns
  • Review: Postposition
  • Review: Pronouns
  • Review: Verbs
  • Review: Punctuation
  • Review: Numerals

NOTES

  • On this day of August 8:
    • 1876: Thomas Edison invents Autographic Printing;
    • 2020: Me hardly passes 80% barrier on Trimmed Coverage :D
  • Creating a Word Frequency List from Corpus:
    • Aka: Hitparade;
  cat corpus.tr.wiki.20200601.txt | apertium-destxt | lt-proc -w ../apertium-tur-uzb/tur-uzb.automorf.bin | apertium-retxt | sed 's/\$\s*/\$\n/g' | grep '\*' | sort | uniq -c | sort -rn > tur-uzb.parade.txt
  • Review: Interjections - DONE!
  • Review: Determinatives - DONE!
  • Review: Conjunctions - DONE!
  • Long lagging stuff:
    • Fix Trimmed coverage - DONE!.
  • Review: Adverbs - DONE!
  • Add Section “Regexp” to bidix - DONE!
  • Add Section “Unchecked” to bidix - DONE!
  • Review: Adjectives - DONE!
  • Review: Abbreviations - DONE!
  • Combining checked and unchecked sections of bidix : DONE
  • Fixing the bidix, deduplication(24.07.2020):
    • Before the fix: Entries:4418
    • After the fix: Entries: 4000
  • Firespeaker:
    • elmurod1202: you already have a trimmed transducer
    • elmurod1202: just use it to analyse a corpus and generate a frequency list of unanalysed forms
    • elmurod1202: the trimmed transducer is the monolingual transducer limited to the words in bidix, it's tur-uzb.autmorf.bin and uzb-tur.autmorf.bin
    • there's scripts all over that do this already
    • it's a few lines of bash
  • Install Apertium-dixtools -DONE
  • Tur-Uzb Translation example:
  cat texts/tur.txt | apertium -d ../../apertium-tur-uzb/ tur-uzb

  awk 'FNR==NR{a[$1]=$2;next} ($1 in a) {print $1,a[$1],$2}' en-tr.sorted en-uz.sorted > tr-uz.txt
  • Update JaM Story - DONE!.
  • Remember: “Every time a set of modifications is made to any of the dictionaries, the modules have to be recompiled. Type make in the directory where the linguistic data are saved”
    • To do so, there is an easy way:
    • Just run this in apertium-tur-uzb directory: make langs
    • Even better?, Use: make -j3 langs
  • Helpful Quotes( :-) ):
    • “The best script is maybe(often) no script”(@Firespeaker)
    • “There is no stupid question, only stupid students”(@Spectei)

EVALUATION

  sh lt-covtest.sh tur-uzb ../apertium-tur-uzb ../corpora/tur.SETimes.en-tr.txt.bz2
    • As of Aug 1: 72.74 %
  • Calculating Coverage:
    • apertium-quality
  git clone https://github.com/apertium/apertium-quality
      • It works \o/, phew.
  aq-covtest texts/corpus.wiki.20200520.txt uzb_guesser.automorf.bin - 94.14%
  aq-covtest corpus.wiki.20200520.txt ../apertium-uzb/uzb.automorf.bin  - 89.36%

  bash hfst-covtest.sh uzb ../apertium-uzb/ ../corpora/corpus.wiki.20200520.txt

  perl apertium-eval-translator-line.pl -ref ./texts/uzb.txt -test ./texts/tur-uzb.txt
  • Counting Stems:
    • Counting dix stems
      • Apertium-tools - dixcounter.py
  python3 dixcounter.py ../apertium-tur-uzb/apertium-tur-uzb.tur-uzb.dix
      • Stems:
        • Beginning: 2412
        • Before fix: 2468
        • After deuplication: 2065
    • Counting Lexc Stems
  python3 lexccounter.py ../apertium-uzb/apertium-uzb.uzb.lexc
      • Apertium-uzb.uzb.lexc: 34375
      • Apertium-tur.tur.lexc : 21634
      • Apertium-tur-uzb.uzb.lexc: 3922
      • apertium-tur-uzb.tur.lexc : 11206


SETTING UP INSTRUCTIONS

  • Apertium-dixtools:
    • Fixing bidix(deduplicatiing, removing empty lines):
  apertium-dixtools fix apertium-tur-uzb.tur-uzb.dix apertium-tur-uzb.tur-uzb.dix.fixed
    • Sorting bidix(aligning too):
  apertium-dixtools sort -alignBidix -ignorecase apertium-tur-uzb.tur-uzb.dix apertium-tur-uzb.tur-uzb.dix.sorted
  • Extracting Wikipedia corpus:
  wget https://dumps.wikimedia.org/uzwiki/20200520/uzwiki-20200520-pages-articles.xml.bz2

  python3 apertium-WikiExtractor.py --infn uzwiki.xml.bz2

  echo 'Salom dunyo' | apertium -d . uzb-tur
  • Translation of a text file:
  cat texts/tur.txt |apertium -d . tur-uzb > ./texts/tur-uzb.txt
  cat texts/uzb.txt |apertium -d . uzb-tur > ./texts/uzb-tur.txt

./autogen.sh --with-lang1=../apertium-tur --with-lang2=../apertium-uzb make

      • For a mono, it’s just:
./autogen.sh
 make
  • Forked(&Installed) necessary repos on GitHub:
    • Apertium-uzb
      • Monolingual package for Uzbek
    • Apertium-tur
      • Monolingual package for Turkish
    • Apertium-tur-uzb
      • Bilingual package for Turkish-Uzbek translation
    • apertiumpp
      • For parallel corpora used in evaluation
    • Apertium-quality
      • Name says it, to evaluate
        • aq-covtest - checks coverage
      • Also, for extracting wiki, but it didn’t work for me
        • aq-wikiextract - extracts wiki corpora from wiki dump
    • apertium-eval-translator
      • To evaluate WER/PER
    • apertium-viewer
      • To have a nice GUI, but you basically don’t need that
    • Apertium-dixtools
      • To sort/fix dictionaries

GitHub Stuff

  • Branch naming convention:
    • Upstream - the original repo you want to have a slice of
    • Master - your branch/fork from upstream
    • Origin - a local copy of the branch in your machine
  • To synch with all remotes:
    • git fetch -all
  • To see the remote links:
    • git remote -v
  • To be able to sync with the original repo:
    • Adding an upstream remote:
    • With fetch & merge:
      • Fetching the upstream to your project:
        • git fetch upstream
      • Merging changes:
        • git merge upstream/master
    • Or, just do:
      • git pull upstream master
    • Apparently, “git pull” contains:
      • git fetch
      • git merge
  • To reset the repo to previous version:
    • git reset --hard <commit hash>
    •  !!! You loose all commits made after
    • Discard all local changes to all files permanently:
      • git reset --hard
  • To merge(squash) last two commits:
    • git rebase --interactive HEAD~2
    • Then choose the last commit and edit push to squash, save, exit
    • Give the commit message you want
    • Done.

Terminal stuff

  • Screen(Working with multiple screens):
    • Advantage: Ability to access from both SSH and local machine
    • Installation:
      • sudo apt-get install screen
    • Listing the active screens:
      • screen -ls
    • Starting a new session:
      • screen -S screenName
    • Exiting the session:
      • exit
    • Deactivating current screen:
      • Ctrl+A, press D
    • Returning to the background (deactivated) screen:
      • screen -r screenName

Useful Bash commands

  • Reading file line by line and printing line with other format:

while IFS= read -r line; do 

   echo "Text read from file: $line"

done < my_filename.txt

  • Printing specific line (given line number) of a text:
    • $ sed -n 5p file
    • Returns 5th line
  • Printing a specific column of a text file:
    • awk -F":" '{print $1}' file.txt
    • -F”:” specifies the separator.


NOTE FOR THE READER:

'Here it started, notes are being created from bottom to top, so the last action comes first.'

Retrieved from "https://wiki.apertium.org/w/index.php?title=User:Elmurod1202/GSoC2020Progress&oldid=72931"