Difference between revisions of "Turkish and Kyrgyz/Kymorph article"

From Apertium
Jump to navigation Jump to search
Line 8: Line 8:
* Which corpora to use?
* Which corpora to use?
** Wikipedia
** Wikipedia
**# [http://paste.pocoo.org/show/2KXepqcTTiWDWLOFPRlR/ punktgen.py] ky.crp.txt ky.pickle
**# aq-wikicrp -x -t ky.pickle kywiki-20110923-pages-articles.xml kywp.xml
** [[Turkish and Kyrgyz/Making a corpus from azattyk|Azattyk]]
** [[Turkish and Kyrgyz/Making a corpus from azattyk|Azattyk]]
* concerns
* concerns

Revision as of 07:36, 5 October 2011

Outline

Morphotactica

Morphophonologia

Corpora

  • Which corpora to use?
    • Wikipedia
      1. punktgen.py ky.crp.txt ky.pickle
      2. aq-wikicrp -x -t ky.pickle kywiki-20110923-pages-articles.xml kywp.xml
    • Azattyk
  • concerns
    • Wikipedia is messy; should we have an automated cleaning process or get stats as-is?
      • Use aq-wikicrp, this way it is reproducible .

Numbers

size of corpora
wikipedia azattyk
num words 271005
xml file size >3.8MB