User:Ilnar.salimzyan/On testing

Test-driven language pair development or on testing strategy in Apertium
========================================================================

Some terminology
----------------

# Acceptance tests define when a language pair or a particular sub-module (like
# morphological transducer or CG) are done. That is, they define the
# requirements for what you are going to develop and are written in the process
# of communicating with "stakeholders"/mentors or anyone funding the
# development.

# Unit tests are written by programmers for programmers. They describe how the
# system works and what the structure and behavior of the code is.

# Integration tests, as the name suggests, test whether components (in our case,
# these are modules like morphological transducers, disambiguators, lexical
# selection rules and transfer rules) are successfully integrated into a system
# (=language pair). In case of a language pair, you can think of the acceptance
# tests for that language pair as integration tests, since they test how modules
# of the language pair integrate into a complete machine translation system.

Overview
--------

                    Testing an Apertium MT system
                            /         \
                           /           \
                          /             \
                     Acceptance       Unit tests
                       tests               |
                       /             testing the output
                      /              of each module
                     /                     |
      * Regression-Pending tests      Morphological--Acceptance: * recall or coverage            
              on the wiki              transducers               * precision
      * Corpus test                        |      \              * # of stems
         * upper bound for WER             |       --Unit: * morphophonology
         * upper bounds for [*@#]          |               * morphotactics
                 errors              ConstraintGr----Acceptance: * ambig. rate before&after 
      * Testvoc (has to be clean)          |   \                 * precision
                                           |    \                * # of rules
                  +                        |     ----Unit: * INPUT/OUTPUT comments for each
                                           |                      rule
               numbers:                  Lexical-----Acceptance: * ambig. rate before&after 
      * of stems in the bidix           selection                * precision
      * of lrx rules <-> ambiguity         |    \               * # of rules
         rate before and after             |     ----Unit: * INPUT/OUTPUT comments for each
      * transfer rules                     |                      rule
                                       Transfer------Acceptance: * wiki tests for phrases and
                 +                            \                       sentences
                                               \                  * "testvoc-lite" tests for           
        (gisting evaluation)                    \                      single-words
                                                 ----Unit: * INPUT/OUTPUT comments in the headers
                                                                   of rules