Difference between revisions of "Compilation Speed"
(timings after dangswan's latest improvements :)) |
|||
Line 22: | Line 22: | ||
* make clean in all three dirs, then <code>export LT_JOBS=yes && m -j langs</code> in nno-nob takes 1m20s |
* make clean in all three dirs, then <code>export LT_JOBS=yes && m -j langs</code> in nno-nob takes 1m20s |
||
* make clean in all three dirs, then <code>export LT_JOBS=yes && m -j e </code> in nno-nob takes 42s |
* make clean in all three dirs, then <code>export LT_JOBS=yes && m -j e </code> in nno-nob takes 42s |
||
and for comparison, with ''none'' of the above tricks: |
|||
* make clean in all three dirs, then <code>export LT_JOBS=no && make langs</code> in nno-nob takes 3m14s |
|||
[[Category:Development]] |
[[Category:Development]] |
Revision as of 08:50, 5 August 2022
Compiling dictionaries takes time. There are some tricks for speeding it up.
Each language pair is different so you may want to time the effect yourself, but the following tricks have all had real speedups on apertium-nno-nob at least. All timings are real/wall clock:
1) PARALLELLISE: Use make -j
instead of just make
, in order to use several processors for make goals that can be parallellised. (Note that if you're low on memory and your dictionaries are very large, you may want to cap it to make -j2
or similar.)
make clean && make
in nno-nob takes 1m55smake clean && make -j
in nno-nob takes 1m02s
2) SPLIT SECTIONS: Put export LT_JOBS=yes
in your ~/.bashrc (or ~/.bash_profile) – this will let lttoolbox split dictionaries into sections of at most 50k entries and minimise them in parallel (and large sections get exponentially slower to minimise). You can also tweak the threshold with export LT_MAX_SECTION_ENTRIES=50000
.
make clean && export LT_JOBS=no && make -j nob.automorf.bin
in nob takes 16s (typical situation where only monodix has been changed)make clean && export LT_JOBS=yes && make -j nob.automorf.bin
in nob takes 8s
3) ALLOCATE FASTER: Try another malloc, e.g. tcmalloc or jemalloc. On Debian/Ubuntu, you would sudo apt install libtcmalloc-minimal4
and then put function m () { export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4; make "$@"; }
into your ~/.bashrc (or ~/.bash_profile). Now you can use m
instead of make
to compile with tcmalloc.
make clean && export LT_JOBS=yes && make -j
in nno-nob takes 57s with regular malloc (glibc)make clean && export LT_JOBS=yes && m -j
in nno-nob takes 45s with tcmalloc
4) BUILD LESS: If you work mostly on one direction, you may want to make a debug goal in your Makefile.am for just that direction (example, used as m -j e
)
- make clean in all three dirs, then
export LT_JOBS=yes && m -j langs
in nno-nob takes 1m20s - make clean in all three dirs, then
export LT_JOBS=yes && m -j e
in nno-nob takes 42s
and for comparison, with none of the above tricks:
- make clean in all three dirs, then
export LT_JOBS=no && make langs
in nno-nob takes 3m14s