Compilation Speed

Compiling dictionaries takes time. There are some tricks for speeding it up.

Each language pair is different so you may want to time the effect yourself, but the following tricks have all had real speedups on apertium-nno-nob at least. All timings are real/wall clock:

1) PARALLELLISE: Use make -j instead of just make, in order to use several processors for make goals that can be parallellised. (Note that if you're low on memory and your dictionaries are very large, you may want to cap it to make -j2 or similar.)

• make clean && make in nno-nob takes 1m55s
• make clean && make -j in nno-nob takes 1m02s

2) SPLIT SECTIONS: Put export LT_JOBS=yes in your ~/.bashrc (or ~/.bash_profile) – this will let lttoolbox split dictionaries into sections of at most 50k entries and minimise them in parallel (and large sections get exponentially slower to minimise). You can also tweak the threshold with export LT_MAX_SECTION_ENTRIES=50000.

• make clean && export LT_JOBS=no && make -j nob.automorf.bin in nob takes 16s (typical situation where only monodix has been changed)
• make clean && export LT_JOBS=yes && make -j nob.automorf.bin in nob takes 8s

3) ALLOCATE FASTER: Try another malloc, e.g. tcmalloc or jemalloc. On Debian/Ubuntu, you would sudo apt install libtcmalloc-minimal4 and then put function m () { export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4; make "$@"; } into your ~/.bashrc (or ~/.bash_profile). Now you can use m instead of make to compile with tcmalloc.

• make clean && export LT_JOBS=yes && make -j in nno-nob takes 57s with regular malloc (glibc)
• make clean && export LT_JOBS=yes && m -j in nno-nob takes 45s with tcmalloc

4) BUILD LESS: If you work mostly on one direction, you may want to make a debug goal in your Makefile.am for just that direction (example, used as m -j e)

• make clean in all three dirs, then export LT_JOBS=yes && m -j langs in nno-nob takes 1m20s
• make clean in all three dirs, then export LT_JOBS=yes && m -j e in nno-nob takes 42s

5) TRIM SMARTER: If you have sections that you already know contain matching elements in the analyser and bidix, you can ensure they're only trimmed with each other (ignoring other sections). Large regular expressions in particular can slow down trimming if you have to check every single bidix entry for a match – by putting <re> entries in a <section id="regex" type="standard" in both files and changing Makefile.am to lt-trim --match-section=regex@standard $^ $@, lt-trim will ignore entries from sections other than regex@standard in bidix when trimming regex@standard from the analyser. (This will of course work with any kind of entry, not just regexes.) Time how long the lt-trim step takes first – for most pairs it's fast enough already.

and for comparison, with none of the above tricks:

• make clean in all three dirs, then export LT_JOBS=no && make langs in nno-nob takes 3m14s