Difference between revisions of "Lttoolbox-java"

From Apertium
Jump to navigation Jump to search
Line 88: Line 88:
 
== State of Java port ==
 
== State of Java port ==
   
  +
Compatibility can be checked by invoking test_java_and_c.sh in testdata/regression.
<pre>
 
  +
j@j-laptop-nova:~/esperanto/apertium/lttoolbox-java/testdata/regression$ ./compare_java_and_c.sh
 
 
C analysis is... 0.39sec
 
C analysis is... 0.39sec
 
OK
 
OK
Java analysis is... 1.91sec
+
Java analysis is... 1.85sec
 
OK
 
OK
C generator -g is ... 0.33sec
+
C generator -g is ... 0.32sec
 
OK
 
OK
Java generator -g is ... 1.26sec
+
Java generator -g is ... 2.01sec
 
OK
 
OK
C generator -d is ... 0.33sec
+
C generator -d is ... 0.85sec
 
OK
 
OK
Java generator -d is ... 1.27sec
+
Java generator -d is ... 2.14sec
 
OK
 
OK
C generator -n is ... 0.33sec
+
C generator -n is ... 0.92sec
 
OK
 
OK
Java generator -n is ... 1.25sec
+
Java generator -n is ... 1.77sec
 
OK
 
OK
 
C postgenerator -p is ... 0.04sec
 
C postgenerator -p is ... 0.04sec
 
OK
 
OK
Java postgenerator -p is ... 0.72sec
+
Java postgenerator -p is ... 0.81sec
 
OK
 
OK
 
All tests passed
 
All tests passed
</pre>
 
--[[User:Jacob Nordfalk|Jacob Nordfalk]] 08:52, 30 November 2009 (UTC)
 
   
  +
As you see Java version is currently (jan 2010) a factor 2-8 slower than the C version.
  +
It still gives great performance, however, and Apertium running on Java is very fast, compared to other MT systems.
   
  +
There are currently (jan 2010) problems compiling some very seldom strange constructs (testdata/strange.dix).
  +
You can use the C version to compile these, and the binary files will work fine when used from lttoolbox-java.
  +
  +
 
--[[User:Jacob Nordfalk|Jacob Nordfalk]] 08:52, 30 November 2009 (UTC)
   
 
==Features==
 
==Features==

Revision as of 14:03, 26 January 2010

What is lttoolbox

lttoolbox are 1) making binary files out of the .dix files (lt-comp), 2) analysing or generating text (lt-proc) and 3) expanding a .dix file (lt-expand).

Usage

$ java -jar dist/lttoolbox.jar
lttoolbox: is a toolbox for lexical processing, morphological analysis and generation of words
USAGE: java -jar dist/lttoolbox.jar [task]
Examples:
 java -jar dist/lttoolbox.jar lt-expand dictionary.dix     expands a dictionary
 java -jar dist/lttoolbox.jar lt-comp lr dic.dix dic.bin   compiles a dictionary
 java -jar dist/lttoolbox.jar lt-proc dic.bin              morphological analysis

or, using the a shell scripts:

$ lt-comp-j 
 v3.2j: build a letter transducer from a dictionary
USAGE: LTComp lr | rl dictionary_file output_file [acx_file]
Modes:
  lr:     left-to-right compilation
  rl:     right-to-left compilation


$ lt-proc-j
LTProc: process a stream with a letter transducer
USAGE: LTProc [-c] [-a|-g|-n|-d|-b|-p|-s|-t] fst_file [input_file [output_file]]
Options:
  -a:   morphological analysis (default behavior)
  -c:   use the literal case of the incoming characters
  -e:   morphological analysis, with compound analysis on unknown words
  -f:   match flags (experimental)
  -g:   morphological generation
  -n:   morph. generation without unknown word marks
  -d:   morph. generation with all the stuff
  -t:   morph. generation, but retaining part-of-speech
  -p:   post-generation
  -s:   SAO annotation system input processing
  -t:   apply transliteration dictionary
  -z:   flush output on the null character 
  -v:   version
  -D:   debug; print diagnostics to stderr
  -h:   show this help


$ lt-expand-j 
 v3.2j: expand the contents of a dictionary fileUSAGE: LTExpand dictionary_file [output_file]


$ lt-validate-j 
 v3.2j: validate an XML file according to a schema
USAGE : LTValidate -dix dictionary.xml
        LTValidate -acx dictionary.acx

Examples

Use the new compounding feature:

echo "lambakjöti" | java -jar dist/lttoolbox.jar lt-proc -e /home/j/esperanto/apertium/apertium-is-en/is.bin

^lambakjöti/lamb<n><nt><pl><gen><ind>+kjöt<n><nt><sg><dat><ind>$

Encoding problems

Try -Dfile.encoding=UTF-8, like

echo "lambakjöti" | java -Dfile.encoding=UTF-8 -jar dist/lttoolbox.jar lt-proc -e /home/j/esperanto/apertium/apertium-is-en/is.bin

^lambakjöti/lamb<n><nt><pl><gen><ind>+kjöt<n><nt><sg><dat><ind>$

Mac users

You need JDK1.6. Try

/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Commands/java -jar dist/lttoolbox.jar

Reasons for a Java port

  • There are several devices (mobile phones, for example) which can run quite complicated software, but only if written in Java. lttoolbox is the first step to having Apertium run on these devices.
  • Windows port. It won't be as powerfull as Unix based system, but it will be there
  • Apertium will be the first MT system *ever* that can be demonstradet within a Java applets
  • Transfer in bytecode has a promise of speedup factor 4 - compared to what we use now (interpreted XML). And transfer CPU usage is dominating when processing large amounts of text

State of Java port

Compatibility can be checked by invoking test_java_and_c.sh in testdata/regression.

C analysis is... 0.39sec OK Java analysis is... 1.85sec OK C generator -g is ... 0.32sec OK Java generator -g is ... 2.01sec OK C generator -d is ... 0.85sec OK Java generator -d is ... 2.14sec OK C generator -n is ... 0.92sec OK Java generator -n is ... 1.77sec OK C postgenerator -p is ... 0.04sec OK Java postgenerator -p is ... 0.81sec OK All tests passed

As you see Java version is currently (jan 2010) a factor 2-8 slower than the C version. It still gives great performance, however, and Apertium running on Java is very fast, compared to other MT systems.

There are currently (jan 2010) problems compiling some very seldom strange constructs (testdata/strange.dix). You can use the C version to compile these, and the binary files will work fine when used from lttoolbox-java.


--Jacob Nordfalk 08:52, 30 November 2009 (UTC)

Features

  • Binary compatibility with lttoolbox. lttoolbox-java is able _read_ and _write_ the binary files lttoolbox and generates exactly the same output
  • There is a comprehensive test suite that tests both lttoolbox (C++) and lttoolbox-java.


Other notes

<Drew_> jacobEo: I can't find a main class in the source code, am I looking in the wrong place? :S
<jacobEo> Drew_: LTComp.java, LTExpand.java, LTProc.java

Thanks

  • Nic Cottrell contributed an initial version of a Java port of lttoolbox.
  • During GSOC2009 Raphaël and Sergio worked on it, but processing still didnt work (compilation and expansion worked)
  • November 2009 Jacob Nordfalk finished it up and optimized it