Dictionary coverage

From Apertium
Revision as of 11:07, 6 December 2019 by Intello (talk | contribs)
Jump to navigation Jump to search

En français

An experimental tool “Apertium-dixtools” exists to do frequency statistics on a dictionary. It finds out which entries have been used and which entries have not been used. It works on both, normal entries as well as entries in paradigms.

The tool can be used, for example, if you have a pair of languages that you want to work in the other direction. The tool has been successfully used on English and Esperanto to make the eo-en direction. Contact me (--Jacob Nordfalk, 18: 07, November 3, 2009 (UTC)) for help using it.

Here is the method how it could be applied on sv-da to analyse how much of the Danish dictionary (which contains many entries not used from sv to da) is useful. The steps would be:

  1. Remove duplicates in dixes
  2. Make a copy of the pair in the profiler/ subdirectory
  3. Create a "profiler" version of your dictionaries.
  4. Edit the modes.xml and add a mode.
  5. Use the mode.


The steps are explained with suitable examples as below:

  • To remove duplicates in dictionary, do:
 apertium-dixtools fix -alignBidix apertium-sv-da.sv-da.dix apertium-sv-da.sv-da.dix
 apertium-dixtools fix -alignMonodix apertium-sv-da.da.dix apertium-sv-da.da.dix
 apertium-dixtools fix -alignMonodix apertium-sv-da.sv.dix apertium-sv-da.sv.dix
  • To make a copy of the pair in the profiler/ subdirectory, do:
 $ mkdir profiler/
 $ cp * profiler/*


  • To create a "profiler" version of your dictionaries(A profiler version of dictionary is a subdirectory of a dictionary), do:
 $ apertium-dixtools profilecreate . sv-da 

This will create a file dixtools-profilekeys.txt and overwrite the profiler/*.dix dictionaries.

  • Now do
 $ cd profiler
 $ make
  • To edit the modes.xml and add a mode where you replace with usage of the profiling version of the dictionary, after which the task apertium-dixtools profilecollect must be inserted
to collect the data (saving it to dixtools-profiledata.txt) and filter the output for the following stages in the mode file, do:
     <program name="lt-proc $1">
       <file name="sv-da.autogen.bin"/>
     </program>
     <program name="lt-proc $1">
       <file name="profiler/sv-da.autogen.bin"/>
     </program>
     <program name="apertium-dixtools profilecollect">
       <file name="dixtools-profiledata.txt"/>
     </program>
  • To use the mode, for example on a corpus and/or with your favorite test script, do:
 $ apertium-dixtools profileresult
 Reading dixtools-profilekeys.txt
 Reading dixtools-profiledata.txt
 Writing dixtools-profileresult.txt


How it works

To add keys to the dictionaries, do( See also Monodix basics) :

<pardef n="b/urde__vbmod">
  <e>       <p><l>urde</l>      <r>urde<s n="vbmod"/><s n="inf"/><s n="actv"/></r></p></e>
  <e>       <p><l>ør</l>        <r>urde<s n="vbmod"/><s n="pres"/><s n="actv"/></r></p></e>
  <e>       <p><l>urde</l>      <r>urde<s n="vbmod"/><s n="past"/><s n="actv"/></r></p></e>
  <e>       <p><l>urdet</l>     <r>urde<s n="vbmod"/><s n="pp"/></r></p></e>
</pardef>

<pardef n="må/tte__vaux">
  <e>       <p><l>tte</l>       <r>tte<s n="vaux"/><s n="inf"/><s n="actv"/></r></p></e>
  <e>       <p><l></l>          <r>tte<s n="vaux"/><s n="pres"/><s n="actv"/></r></p></e>
  <e>       <p><l>tte</l>       <r>tte<s n="vaux"/><s n="past"/><s n="actv"/></r></p></e>
  <e>       <p><l>ttet</l>      <r>tte<s n="vaux"/><s n="pp"/></r></p></e>
</pardef>

becomes

<pardef n="b/urde__vbmod">
  <e>       <p><l>urde</l>      <r>urde<s n="vbmod"/><s n="inf"/><s n="actv"/></r></p><p><l>%bct%</l><r/></p></e>
  <e>       <p><l>ør</l>        <r>urde<s n="vbmod"/><s n="pres"/><s n="actv"/></r></p><p><l>%bcu%</l><r/></p></e>
  <e>       <p><l>urde</l>      <r>urde<s n="vbmod"/><s n="past"/><s n="actv"/></r></p><p><l>%bcv%</l><r/></p></e>
  <e>       <p><l>urdet</l>     <r>urde<s n="vbmod"/><s n="pp"/></r></p><p><l>%bcw%</l><r/></p></e>
</pardef>

<pardef n="må/tte__vaux">
  <e>       <p><l>tte</l>       <r>tte<s n="vaux"/><s n="inf"/><s n="actv"/></r></p><p><l>%bcx%</l><r/></p></e>
  <e>       <p><l></l>          <r>tte<s n="vaux"/><s n="pres"/><s n="actv"/></r></p><p><l>%bcy%</l><r/></p></e>
  <e>       <p><l>tte</l>       <r>tte<s n="vaux"/><s n="past"/><s n="actv"/></r></p><p><l>%bcz%</l><r/></p></e>
  <e>       <p><l>ttet</l>      <r>tte<s n="vaux"/><s n="pp"/></r></p><p><l>%bd0%</l><r/></p></e>
</pardef>

The dixtools-profile result.txt contains-

0 bct <e><l>urde</l><r>urde<vbmod><inf><actv></r></e>
0 bcu <e><l>ør</l><r>urde<vbmod><pres><actv></r></e>
0 bcv <e><l>urde</l><r>urde<vbmod><past><actv></r></e>
0 bcw <e><l>urdet</l><r>urde<vbmod><pp></r></e>
1 bcx <e><l>tte</l><r>tte<vaux><inf><actv></r></e>
41 bcy <e><l></l><r>tte<vaux><pres><actv></r></e>
1 bcz <e><l>tte</l><r>tte<vaux><past><actv></r></e>
1 bd0 <e><l>ttet</l><r>tte<vaux><pp></r></e>

So you can see that the first paradigm isn't used at all. While, in the 2nd paradigm the %bcy% was used 41 times and the other entries 1 time each.