Difference between revisions of "Constraint-based lexical selection module"

From Apertium
Jump to navigation Jump to search
Line 30: Line 30:
The module requires [[VM for transfer]], or using <code>apertium-transfer -b</code> in order to work.
The module requires [[VM for transfer]], or using <code>apertium-transfer -b</code> in order to work.
The same as lttoolbox and apertium (see [[Installation]]) as well as (on Debian/Ubuntu) zlib1g-dev

Revision as of 09:12, 21 June 2013

Lexical transfer

This is the output of lt-proc -b on an ambiguous bilingual dictionary.

[74306] ^El<det><def><f><sg>/The<det><def><f><sg>$ 
^estació<n><f><sg>/season<n><sg>/station<n><sg>$ ^més<preadv>/more<preadv>$ 
^i<cnjcoo>/and<cnjcoo>$ ^el<det><def><f><sg>/the<det><def><f><sg>$ 
^més<preadv>/more<preadv>$ ^sec<adj><f><sg>/dry<adj><sint><f><sg>$ 


L'estació més plujós és el tardor, i la més sec l'estiu

Goes to:

The season/station more rainy is the autumn/fall, and the more dry the summer.

The module requires VM for transfer, or using apertium-transfer -b in order to work.


The same as lttoolbox and apertium (see Installation) as well as (on Debian/Ubuntu) zlib1g-dev


Check out the code from

$ svn co https://apertium.svn.sourceforge.net/svnroot/apertium/trunk/apertium-lex-tools

you can make it using:

$ ./autogen.sh
$ make
$ sudo make install


If you get the message lrx-comp: error while loading shared libraries: libapertium3-3.2.so.0: cannot open shared object file: No such file or directory you may need to put this in your ~/.bashrc


Then open a new terminal before using lrx-comp/lrx-proc.


Make a simple rule file,

    <match lemma="criminal" tags="adj"/>
    <match lemma="court" tags="n.*"><select lemma="juzgado" tags="n.*"/></match>

Then compile it:

$ lrx-comp rules.xml rules.fst
1: 32@32

The input is the output of lt-proc -b,

$ echo "^There<adv>/Allí<adv>$ ^be<vbser><pri><p3><sg>/ser<vbser><pri><p3><sg>$ ^a<det><ind><sg>/uno<det><ind><GD><sg>$ 
^court<n><sg>/corte<n><f><sg>/cancha<n><f><sg>/juzgado<n><m><sg>/tribunal<n><m><sg>$^.<sent>/.<sent>$" | ./lrx-proc -t rules.fst 
^There<adv>/Allí<adv>$ ^be<vbser><pri><p3><sg>/ser<vbser><pri><p3><sg>$ ^a<det><ind><sg>/uno<det><ind><GD><sg>$ 
^criminal<adj>/criminal<adj><mf>/delictivo<adj>$ ^court<n><sg>/juzgado<n><m><sg>$^.<sent>/.<sent>$

Rule format

A rule is made up of an ordered list of:

  • Matches
  • Operations (select, remove)
  <match lemma="el"/>  
  <match lemma="dona" tags="n.*">    
    <select lemma="wife"/> 
  <match lemma="de"/>

  <match lemma="estació" tags="n.*">    
    <select lemma="season"/> 
  <match lemma="més"/>
  <match lemma="plujós"/>

  <match lemma="guanyador"/>
  <match lemma="de"/>
  <match lemma="prova" tags="n.*">    
    <select lemma="event"/> 

Writing and generating rules


Main article: How to get started with lexical selection rules

A good way to start writing lexical selection rules is to take a corpus, and search for the problem word, you can then look at how the word should be translated, and the contexts it appears in.


Parallel corpus
Main article: Generating lexical-selection rules from a parallel corpus
Monolingual corpora

Todo and bugs

  • xml compiler
  • compile rule operation patterns, as well as matching patterns
  • make rules with gaps work
  • optimal coverage
  • fix bug with processing multiple sentences
  • instead of having regex OR, insert separate paths/states.
  • optimise the bestPath function (don't use strings to store the paths)
  • autotoolsise build
  • add option to compiler to spit out ATT transducers
  • fix bug with outputting an extra '\n' at the end
  • edit transfer.cc to allow input from lt-proc -b
  • profiling and speed up
    • why do the regex transducers have to be minimised ?
    • retrieve vector of strings corresponding to paths, instead of a single string corresponding to all of the paths
    • stop using string processing to retrieve rule numbers
    • retrieve vector of vectors of words, not string of words from lttoolbox
    • why does the performance drop substantially with more rules ?
    • add a pattern -> first letter map so we don't have to call recognise() with every transition (didn't work so well)
  • there is a problem with the regex recognition code: see bug1 in testing.
  • there is a problem with two defaults next to each other; bug2 in testing.
  • default to case insensitive ? (perhaps case insensitive for lower case, case sensitive for uppercase) -- see bug4 in testing/.
  • make sure that -b works with -n too.
  • testing
  • null flush
  • add option to processor to spit out ATT transducers
  • 2011-12-12: 10,000 words / 97 seconds = 103 words/sec (71290 words, 14.84 sec = 4803 words/sec)
  • 2011-12-19: 10,000 words / 4 seconds = 2,035 words/sec (71290 words, 8 secs = 8911 words/sec)

Preparedness of language pairs

Pair LR (L) LR (L→R) Fertility Rules
apertium-is-en 18,563 22,220 1.19 115
apertium-eu-es 16,946 18,550 1.09 250
apertium-br-fr 20,489 20,770 1.01 256
apertium-mk-en 8,568 10,624 1.24 81
apertium-en-es 267,469 268,522 1.003 334

See also