Apertium-quality/Quickstart - Revision history

Andiqu: /* Getting apertium-mt-he */

2018-10-24T17:06:32Z

Getting apertium-mt-he

Andiqu: /* Getting apertium-mt-he */

2018-10-24T16:57:40Z

Getting apertium-mt-he

Andiqu: Removed SVN references

2018-10-24T14:19:59Z

Removed SVN references

Gallaecio: /* Installing apertium-quality */ Add ‘sudo’ to ‘make install’ commands

2016-07-12T07:02:51Z

Installing apertium-quality: Add ‘sudo’ to ‘make install’ commands

Ilnar.salimzyan: /* Vocabulary testing */

2015-02-24T01:15:42Z

Vocabulary testing

Ilnar.salimzyan: /* Dictionary testing */

2015-02-24T01:10:12Z

Dictionary testing

Ilnar.salimzyan: /* Regression testing */ fix a typo

2014-02-08T19:22:35Z

Regression testing: fix a typo

Ilnar.salimzyan: /* Installing apertium-quality */ fix a typo

2014-02-08T19:17:06Z

Installing apertium-quality: fix a typo

123.243.206.102: Created page with '== Introduction == The Apertium Quality Control Framework, or `apertium-quality`, is a framework and toolkit for unit testing Apertium dictionaries, and to a lesser extent, HFST …'

2011-08-30T11:02:21Z

Created page with '== Introduction == The Apertium Quality Control Framework, or `apertium-quality`, is a framework and toolkit for unit testing Apertium dictionaries, and to a lesser extent, HFST …'

New page

== Introduction ==
The Apertium Quality Control Framework, or `apertium-quality`, is a framework and toolkit for unit testing Apertium dictionaries, and to a lesser extent, HFST finite state transducers, and recording detailed statistics for later analysis. The toolkit portion of the project consists of tools such as regression testing, coverage testing, vocabulary testing and statistics storage, while the framework is robust enough to allow extensive modification through standard interfaces and detailed XML schemas.

== Prerequisites ==
You must have, at the very least, the following applications and libraries installed:

* Autotools
* Python >= 3.1
* Subversion
* Apertium >= 3.2 (or SVN version)

Assuming Debian or Ubuntu, you can most likely get these by running:
<pre>
apt-get install automake autoconf subversion apertium python3 python3-lxml
</pre>

== A typical use case/walkthrough — apertium-mt-he ==
=== Getting the goodies ===
==== Installing apertium-quality ====
First of all, we need to actually install `apertium-quality`, so in the terminal, enter:
<pre>svn co https://apertium.svn.sourceforge.net/svnroot/apertium/trunk/apertium-tools/apertium-quality</pre>

It may ask you to accept a security certificate. Feel free to accept permanently. Once it has downloaded, <code>cd</code> into the directory and run the usual commands:
<pre>
cd apertium-quality
./autogen.sh && make && make install && make install-nltk
</pre>

If you see any "errors" relating to libyaml, ignore them. They're not errors, it simply isn't finding the C libyaml libraries, and instead uses pure Python ones.

Add `--prefix` wherever you please, and if Python 3 isn't detected for some reason, prefix the `./autogen.sh` command with `PYTHON=/path/to/python3`.

Before running any of the tools you must export the line specified by the Makefile after you build the module if you installed to a prefix, otherwise Python wont be able to find the module.

''PLEASE NOTE: if you cannot install lxml, you may come across a few bugs, as lxml is preferred ove the built-in ElementTree library for compatibility and speed reasons. If you do come across any bugs however, please report them!''

==== Getting apertium-mt-he ====
`apertium-mt-he` is a dictionary pair that was being developed during the GSoC period while I was coding `apertium-quality`, so why not try it out and see how it's going?

Let's download it:
<pre>
svn co https://apertium.svn.sourceforge.net/svnroot/apertium/incubator/apertium-mt-he
cd apertium-mt-he
</pre>

So assuming that the aq tools were installed correctly and are visible on your `$PATH`, we can begin testing. Otherwise, set up your $PATH correctly.

First of all, we should pick a few revisions to test so we can get out some pretty stats, so the logs today show this for me:
<pre>
$ svn log
------------------------------------------------------------------------
r33535 | unhammer | 2011-08-21 00:03:20 +1000 (Sun, 21 Aug 2011) | 1 line

new summary, only 51 verb _forms_ missing :)
------------------------------------------------------------------------
r33534 | unhammer | 2011-08-20 23:57:00 +1000 (Sat, 20 Aug 2011) | 1 line

minor typo
------------------------------------------------------------------------
r33525 | n0nick | 2011-08-20 19:13:12 +1000 (Sat, 20 Aug 2011) | 1 line

adding more missing verbs to he.dix
------------------------------------------------------------------------
r33524 | n0nick | 2011-08-20 19:04:58 +1000 (Sat, 20 Aug 2011) | 1 line

adding some missing verbs
------------------------------------------------------------------------
r33523 | n0nick | 2011-08-20 18:43:58 +1000 (Sat, 20 Aug 2011) | 1 line
</pre>

Let's start by checking out an earlier revision and work our way forward:
<pre>
svn up -r 33402
</pre>

=== Herein lies the testing ===
So you've successfully gotten up to this point? Wonderful! Let's start doing some testing. But first, a few things:

# Never do testing on a dictionary without committing changes first. It most likely won't let you in order to guarantee the integrity of the data.
# Unless you want to save statistics, don't use -X. Test first, statistic save later.

First, let's compile the dictionary.
<pre>
./autogen.sh && make
</pre>
''If you have installed Apertium to a prefix, make sure you prepend ./autogen in the above code with <code>PKG_CONFIG_PATH=/path/to/lib/pkgconfig</code>!''

==== Ambiguity testing ====
This kind of test allows you to see the naive ambiguity of your dictionaries by counting the forms of each word, then averaging the result. We do this by running `aq-ambtest` on each dix. So let's do it!

<pre>
aq-ambtest apertium-mt-he.he.dix -X
aq-ambtest apertium-mt-he.mt.dix -X
aq-ambtest apertium-mt-he.mt-he.dix -X
</pre>

See that dangling `-X` on the end? That's the flag to save statistics. By default it saves to a file called `quality-stats.xml`. If you add a filename to the end of the `-X`, it'll save in that file. By putting it at the end, we can just use the default, and we will! All tests use `-X` for saving stats, so easy to remember.

==== Regression testing ====
Dictionaries are complex beings. Sometimes you want to write some tests to prove you haven't broken something, or you've met a milestone. Luckily, this dictionary has quite a few regression tests already written for our testing pleasure, so let's have at it then!

<pre>
aq-regtest -d . mt-he http://wiki.apertium.org/wiki/Special:Export/Maltese_and_Hebrew/Regression_tests -X
aq-regtest -d . mt-he http://wiki.apertium.org/wiki/Special:Export/Maltese_and_Hebrew/Pending_tests -X
</pre>

You should get a lot of output saying WORKS everywhere for the first one, with 24/24 passes. 100%, yay! Pending however is a bit more disappointing. 4/33? How disheartening. Not to matter, maybe it gets better in the future!

==== Wikipedia corpus extractor ====
So in order to do some of the tests like generation testing or coverage testing, we need corpora, right? Have no fear, for `aq-wikicrp` is here! Let us get a Maltese wikipedia dump and make a lovely little corpus with it, but first, we must get a sentence tokeniser compatible with our version of Python.

* Python 3.1: `wget http://apertium.bbqsrc.net/static/maltese-3.1.pickle`
* Python 3.2: `wget http://apertium.bbqsrc.net/static/maltese-3.2.pickle`

''For information on how to generate your own tokenisers, check out dev/punktgen.py in the apertium-quality svn repo.''

And now, assuming Python 3.2, we run the following:
<pre>
wget http://dumps.wikimedia.org/mtwiki/latest/mtwiki-latest-pages-articles.xml.bz2 && bunzip2 mtwiki-latest-pages-articles.xml.bz2
aq-wikicrp mtwiki-latest-pages-articles.xml mt.wikipedia.crp.txt -t ./maltese-3.2.pickle
aq-wikicrp -x mtwiki-latest-pages-articles.xml mt.wikipedia.crp.xml -t ./maltese-3.2.pickle
</pre>

Have a look at both of the output files. One of them is purely plain text, with a sentence per line. The other file is an XML corpus, separated into "entries". The XML format allows for some more advanced parsing and searching, but is so far hardly used with this toolkit, other than coverage testing, where both formats are supported. Feel free to use whichever you're more comfortable with.

==== Coverage testing ====
Coverage testing does what you'd expect, tests the dictionary for coverage. Using our newly created corpora, we can test the coverage! Feel free to use either one, but be consistent; only use one of them.

<pre>
aq-covtest mt.wikipedia.crp.txt mt-he.automorf.bin -X
# OR
aq-covtest mt.wikipedia.crp.xml mt-he.automorf.bin -X
</pre>

This command should run fairly quickly, and give you the number of tokenised words, coverage percentage and top unknown words. It also displays the speed it translated the whole corpus, and it's all stored nice and cosy in the stats file.

==== Dictionary testing ====
There are a few other tests that can be done on the dictionaries that don't really need their own command, so these functions are in `aq-dixtest`. They include counting entries and counting rules, but will soon be extended to count CG rules and transfer rules. It can be run:
<pre>
aq-dixtest mt-he -X
</pre>

This will show the count of rules per file, and the entry count per file. It also shows the totals and unique entries.

==== Vocabulary testing ====
This test allows you to see the manner that your dictionary's transfer rules are working, and get some useful output. By default it will save the output in voctest.txt, so it's worth having a look at this output to find transfer bugs and what not.
<pre>
aq-voctest mt-he -X
</pre>

You should get some pretty line counts, # counts and @ counts. Sweet!

==== Generation testing ====
Generation testing isn't working at the moment sadly, so stay tuned.

==== Morph testing ====
Morph testing isn't supported by the language we're using, but it is as simple to run as regression testing. One simply runs a configuration file like:
<pre>
aq-morftest tests.yaml -X
</pre>

This command will do very similar things to the regression test, but for HFST dictionaries. It allows you to test morphology in both directions to find bugs and regressions. Pretty schmick.

=== What now? ===
Now we've run pretty much every test. What now? Didn't you say something about pretty graphs? Yes, I did! But first, we need more statistics, and who really wants to type all of these commands all of the time unless testing for output?

==== Introducing aq-autotest ====
`aq-autotest` takes an .aqx file as input. It is an XML configuration file, defined at http://github.com/bbqsrc/apertium-schemas in the aqx.rnc file. For this walkthrough, here's a ready-made configuration file just for you!

<pre>
<config xmlns="http://apertium.org/xml/quality/config/0.1">
<commands>
<command>./autogen.sh</command>
<command>make clean</command>
<command>make</command>
</commands>
<coverage>
<corpus language="mt-he" path="mt.wikipedia.crp.txt"/>
</coverage>
<regression>
<test language="mt-he" path="http://wiki.apertium.org/wiki/Special:Export/Maltese_and_Hebrew/Regression_tests"/>
<test language="mt-he" path="http://wiki.apertium.org/wiki/Special:Export/Maltese_and_Hebrew/Pending_tests"/>
</regression>
</config>
</pre>

''If you have installed Apertium to a prefix, make sure you prepend ./autogen in the above file with <code>PKG_CONFIG_PATH=/path/to/lib/pkgconfig</code>!''

For this example, simply save the file in the current directory as `quality.aqx`. Before running it, let's delete `quality-stats.xml` so we don't duplicate data unnecessarily, and feed the aqx file to `aq-autotest` like so:
<pre>aq-autotest quality.aqx -X</pre>

This has much less output than the other frontends, due to its primary purpose being to generate statistics as easily as possible.

Now to make this rather short, I have made a simple script so that you can generate the remaining revisions in a timely fashion for the other revisions:
<pre>
for i in 33402 33403 33404 33405 33406 33421 33423 33424 33425 33426 33427 33471 33523 33524 33525 33534 33535 33681 33682 33683; do svn up -r $i; aq-autotest quality.aqx -X; done
</pre>

You might notice that some of the testing fails and bails out. This is normal. It means that revision of the dictionary was buggy and won't compile correctly, so is simply skipped. As it should be.

So, once that finishes running, hurray! We have enough stats to do something fun. What's that, you ask?

==== Welcome to pretty graphs — aq-htmlgen ====
We can generate the preliminary version of some pretty graphs with a very simple command:
<pre>
aq-htmlgen quality-stats.xml out
</pre>

This will output a bunch of JS, CSS, and HTML files to the directory specified, which here is `out`. Open `index.html` in `out` and enjoy the graphs! They are a bit limited right now, but can be easily improved with a drop-in replacement of raphael_linegraph.js which is on the list of things to do!

=== Conclusion ===
I hope the walkthrough wasn't too tedious and highlights how using this framework can improve your testing life. Developer documentation will come very soon so if you wish to extend or improve the framework, you will be able to do it with ease.

Documentation about the configuration formats and XML schemas can be found on the wiki at http://wiki.apertium.org/wiki/apertium-quality.

@@ Line 49: / Line 49: @@
 <pre>
 $ git log
+commit efc73a5695e3c549ccfa4a66bb6a63d2fd830e8c
-commit d52aaf072e6d76f951e9ed7a46ec10335aaf11c6
-Author: Kevin Brubeck Unhammer <unhammer@fsfe.org>
+Author: Francis M. Tyers <ftyers@prompsit.com>
-Date:   Sun Aug 21 00:03:20 2011 +1000
+Date:   Fri Nov 18 12:32:33 2016 +0000
+    modes
-    new summary, only 51 verb _forms_ missing :)
+commit 686db20d4fcb5156e73f4f2c94acdf8ee2a6371a
-commit e82e7ee247cf0b6c814899c34f0369dc02a65f50
-Author: Kevin Brubeck Unhammer <unhammer@fsfe.org>
+Author: Francis M. Tyers <ftyers@prompsit.com>
-Date:   Sat Aug 20 23:57:00 2011 +1000
+Date:   Fri Nov 18 12:31:35 2016 +0000
-    minor typo
+    move
+commit 8b3078c3d83ed2f51f7e782304851ddb56e03a46
-commit 230f3e08bffce6298cb4b0f2522acd8198b24489
+Author: Francis M. Tyers <ftyers@prompsit.com>
-Author: n0nick
-Date:   Sat Aug 20 19:13:12 2011 +1000
+Date:   Fri Nov 18 12:30:38 2016 +0000
+    move to mlt-heb
-    adding more missing verbs to he.dix
+commit 386091477c4f1640ad11b70db3ff998234e002e5
-commit e07a02a4bbdd7b3333e6422aca9ca8c16b3350b2
+Author: Kevin Brubeck Unhammer <unhammer@fsfe.org>
-Author: n0nick
-Date:   Sat Aug 20 19:04:58 2011 +1000
+Date:   Mon Jan 4 10:34:00 2016 +0000
-    adding some missing verbs
+    tolbox→toolbox
-commit aa1069ca6741f42f7faf0c2cd68af2fe8e6c68c8
-Author: n0nick
-Date:   Sat Aug 20 18:43:58 2011 +1000
 </pre>
 Let's start by checking out an earlier revision and work our way forward:
 <pre>
+git checkout a3f1b90732c3d155b2db0ae1211288d51d998c77
-git pull -r 33402
 </pre>

@@ Line 49: / Line 49: @@
 <pre>
 $ git log
+commit d52aaf072e6d76f951e9ed7a46ec10335aaf11c6
-------------------------------------------------------------------------
+Author: Kevin Brubeck Unhammer <unhammer@fsfe.org>
-r33535 | unhammer | 2011-08-21 00:03:20 +1000 (Sun, 21 Aug 2011) | 1 line
+Date:   Sun Aug 21 00:03:20 2011 +1000
-new summary, only 51 verb _forms_ missing :)
+    new summary, only 51 verb _forms_ missing :)
-------------------------------------------------------------------------
-r33534 | unhammer | 2011-08-20 23:57:00 +1000 (Sat, 20 Aug 2011) | 1 line
+commit e82e7ee247cf0b6c814899c34f0369dc02a65f50
-minor typo
+Author: Kevin Brubeck Unhammer <unhammer@fsfe.org>
-------------------------------------------------------------------------
+Date:   Sat Aug 20 23:57:00 2011 +1000
-r33525 | n0nick | 2011-08-20 19:13:12 +1000 (Sat, 20 Aug 2011) | 1 line
+    minor typo
-adding more missing verbs to he.dix
-------------------------------------------------------------------------
-r33524 | n0nick | 2011-08-20 19:04:58 +1000 (Sat, 20 Aug 2011) | 1 line
+commit 230f3e08bffce6298cb4b0f2522acd8198b24489
-adding some missing verbs
+Author: n0nick
-------------------------------------------------------------------------
+Date:   Sat Aug 20 19:13:12 2011 +1000
-r33523 | n0nick | 2011-08-20 18:43:58 +1000 (Sat, 20 Aug 2011) | 1 line
+    adding more missing verbs to he.dix
+commit e07a02a4bbdd7b3333e6422aca9ca8c16b3350b2
+Author: n0nick
+Date:   Sat Aug 20 19:04:58 2011 +1000
+    adding some missing verbs
+commit aa1069ca6741f42f7faf0c2cd68af2fe8e6c68c8
+Author: n0nick
+Date:   Sat Aug 20 18:43:58 2011 +1000
 </pre>

@@ Line 7: / Line 7: @@
 * Autotools
 * Python &gt;= 3.1
+* Git
-* Subversion
-* Apertium &gt;= 3.2 (or SVN version)
+* Apertium &gt;= 3.2
 Assuming Debian or Ubuntu, you can most likely get these by running:
 <pre>
-apt-get install automake autoconf subversion apertium python3 python3-lxml
+apt-get install automake autoconf git apertium python3 python3-lxml
 </pre>
@@ Line 19: / Line 19: @@
 ==== Installing apertium-quality ====
 First of all, we need to actually install `apertium-quality`, so in the terminal, enter:
-<pre>svn co https://apertium.svn.sourceforge.net/svnroot/apertium/trunk/apertium-tools/apertium-quality</pre>
+<pre>git clone https://github.com/apertium/apertium-quality</pre>
 It may ask you to accept a security certificate. Feel free to accept permanently. Once it has downloaded, <code>cd</code> into the directory and run the usual commands:
@@ Line 31: / Line 31: @@
 Add `--prefix` wherever you please, and if Python 3 isn't detected for some reason, prefix the `./autogen.sh` command with `PYTHON=/path/to/python3`.
-Before running any of the tools you must export the line specified by the Makefile after you build the module if you installed to a prefix, otherwise Python wont be able to find the module.
+Before running any of the tools you must export the line specified by the Makefile after you build the module if you installed to a prefix, otherwise Python won't be able to find the module.
 ''PLEASE NOTE: if you cannot install lxml, you may come across a few bugs, as lxml is preferred over the built-in ElementTree library for compatibility and speed reasons. If you do come across any bugs however, please report them!''
@@ Line 40: / Line 40: @@
 Let's download it:
 <pre>
-svn co https://apertium.svn.sourceforge.net/svnroot/apertium/incubator/apertium-mt-he
+git clone https://github.com/apertium/apertium-mlt-heb
-cd apertium-mt-he
+cd apertium-mlt-heb
 </pre>
@@ Line 48: / Line 48: @@
 First of all, we should pick a few revisions to test so we can get out some pretty stats, so the logs today show this for me:
 <pre>
-$ svn log
+$ git log
 ------------------------------------------------------------------------
 r33535 | unhammer | 2011-08-21 00:03:20 +1000 (Sun, 21 Aug 2011) | 1 line
@@ Line 71: / Line 71: @@
 Let's start by checking out an earlier revision and work our way forward:
 <pre>
-svn up -r 33402
+git pull -r 33402
 </pre>
@@ Line 113: / Line 113: @@
 * Python 3.2: `wget http://apertium.bbqsrc.net/static/maltese-3.2.pickle`
-''For information on how to generate your own tokenisers, check out dev/punktgen.py in the apertium-quality svn repo.''
+''For information on how to generate your own tokenisers, check out dev/punktgen.py in the apertium-quality git repo.''
 And now, assuming Python 3.2, we run the following:
@@ Line 195: / Line 195: @@
 Now to make this rather short, I have made a simple script so that you can generate the remaining revisions in a timely fashion for the other revisions:
 <pre>
-for i in 33402 33403 33404 33405 33406 33421 33423 33424 33425 33426 33427 33471 33523 33524 33525 33534 33535 33681 33682 33683; do svn up -r $i; aq-autotest quality.aqx -X; done
+for i in 33402 33403 33404 33405 33406 33421 33423 33424 33425 33426 33427 33471 33523 33524 33525 33534 33535 33681 33682 33683; do git pull -r $i; aq-autotest quality.aqx -X; done
 </pre>

@@ Line 24: / Line 24: @@
 <pre>
 cd apertium-quality
-./autogen.sh && make && make install && make install-nltk
+./autogen.sh && make && sudo make install && sudo make install-nltk
 </pre>

@@ Line 144: / Line 144: @@
 ==== Vocabulary testing ====
+{{see-also|Testvoc}}
 This test allows you to see the manner that your dictionary's transfer rules are working, and get some useful output. By default it will save the output in voctest.txt, so it's worth having a look at this output to find transfer bugs and what not.
 <pre>

@@ Line 136: / Line 136: @@
 ==== Dictionary testing ====
-There are a few other tests that can be done on the dictionaries that don't really need their own command, so these functions are in `aq-dixtest`. They include counting entries and counting rules, but will soon be extended to count CG rules and transfer rules. It can be run:
+There are a few other tests that can be done on the dictionaries that don't really need their own command, so these functions are in `aq-dixtest`. Currently they include counting entries and counting transfer rules, but might be extended to count CG rules as well. It can be run:
 <pre>
 aq-dixtest mt-he -X

@@ Line 98: / Line 98: @@
 ==== Regression testing ====
-Dictionaries are complex beings. Sometimes you want to write some tests to prove you haven't broken something, or you've met a milestone. Luckily, this dictionary has quite a few regression tests already written for our testing pleasure, so let's have at it then!
+Dictionaries are complex beings. Sometimes you want to write some tests to prove you haven't broken something, or you've met a milestone. Luckily, this dictionary has quite a few regression tests already written for our testing pleasure, so let's have a look at it then!
 <pre>

@@ Line 33: / Line 33: @@
 Before running any of the tools you must export the line specified by the Makefile after you build the module if you installed to a prefix, otherwise Python wont be able to find the module.
-''PLEASE NOTE: if you cannot install lxml, you may come across a few bugs, as lxml is preferred ove the built-in ElementTree library for compatibility and speed reasons. If you do come across any bugs however, please report them!''
+''PLEASE NOTE: if you cannot install lxml, you may come across a few bugs, as lxml is preferred over the built-in ElementTree library for compatibility and speed reasons. If you do come across any bugs however, please report them!''
 ==== Getting apertium-mt-he ====