The Right Way to count dix stems

This page documents how to count stems in dix files. There's a ready-made script that does it available at http://svn.code.sf.net/p/apertium/svn/trunk/apertium-tools/dixcounter.py.

We want to import xml data first in the tree form. Getting xml file information via regular expression would be cumbersome and often undoable. There are much neater ways how to accomplish this and XML was specially designed for storing and retrieving information easily.

Here is a quick guide how to process xml in python

import xml.etree.ElementTree as xml
root = xml.fromstring("dictionaryXML")

Dictionary tag is going to be the root of the tree and its children tags are going to be child nodes of the root.

For bilingual dictionaries we want to count the number of word pairs. We have <l></l> and <r></r> tags for a pair inside the <e></e> tag.

For monolingual dictionaries we want to count the number of lemmas. That means <e> tag with lm attribute.

We use root.findall() to get all occurrences and then len() to get the size of the resultant array.

So the command for the bilingual dictionaries:

len(tree.findall("*[@id='main']/e//l"))

(we choose all <l> tags which are inside <e></e> tags inside section with 'main' id)

and for the monolingual dictionaries:

len(tree.findall("*[@id='main']/*[@lm]"))

(we choose all e tags with 'lm' attribute inside section with 'main' id)