Difference between revisions of "IRSTLM"

From Apertium
Jump to navigation Jump to search
m (exporting IRSTLM)
 
(9 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
''IRSTLM'' is a free and open source exact statistical language model using memory-mapping. The language models are compatible with those created with the closed-source SRILM Tooolkit.
 
''IRSTLM'' is a free and open source exact statistical language model using memory-mapping. The language models are compatible with those created with the closed-source SRILM Tooolkit.
   
See the homepage at http://hlt.fbk.eu/en/irstlm
+
See the homepage at https://hlt-mt.fbk.eu/technologies/irstlm
   
   
 
==Installation==
 
==Installation==
  +
see https://github.com/irstlm-team/irstlm
 
  +
or
 
<pre>
 
<pre>
 
svn checkout svn://svn.code.sf.net/p/irstlm/code/trunk irstlm
 
svn checkout svn://svn.code.sf.net/p/irstlm/code/trunk irstlm
Line 12: Line 13:
 
make -j4
 
make -j4
 
make install
 
make install
  +
</pre>
  +
  +
if you get <code>stdlib.h</code> error, see https://github.com/irstlm-team/irstlm/issues/22
  +
  +
== Make a language model ==
  +
<pre>
  +
# if you specified /path/prefix previously
  +
export IRSTLM=/path/prefix
  +
# else
  +
export IRSTLM=/usr/local
  +
  +
$IRSTLM/bin/build-lm.sh -i incorpus.txt -o out.lm.gz -t tmp/
 
</pre>
 
</pre>
   
 
==See also==
 
==See also==
* [[Moses]]
+
* [[Moses]] (includes alternative LM system KenLM)
 
* [[Using GIZA++]]
 
* [[Using GIZA++]]
 
* [[RandLM]] - a randomised LM, based on [http://en.wikipedia.org/wiki/Bloom_filter Bloom Filters]
 
* [[RandLM]] - a randomised LM, based on [http://en.wikipedia.org/wiki/Bloom_filter Bloom Filters]

Latest revision as of 15:15, 5 August 2021

IRSTLM is a free and open source exact statistical language model using memory-mapping. The language models are compatible with those created with the closed-source SRILM Tooolkit.

See the homepage at https://hlt-mt.fbk.eu/technologies/irstlm


Installation[edit]

see https://github.com/irstlm-team/irstlm or

svn checkout svn://svn.code.sf.net/p/irstlm/code/trunk irstlm
cd irstlm
cmake -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX=/path/prefix
make -j4
make install

if you get stdlib.h error, see https://github.com/irstlm-team/irstlm/issues/22

Make a language model[edit]

# if you specified /path/prefix previously
export IRSTLM=/path/prefix
# else
export IRSTLM=/usr/local

$IRSTLM/bin/build-lm.sh -i incorpus.txt -o out.lm.gz -t tmp/

See also[edit]