Compilation Speed - Revision history

Unhammer at 11:47, 4 October 2022

2022-10-04T11:47:01Z

Unhammer at 11:45, 4 October 2022

2022-10-04T11:45:58Z

Unhammer at 11:44, 4 October 2022

2022-10-04T11:44:47Z

Unhammer at 09:05, 11 August 2022

2022-08-11T09:05:47Z

Unhammer at 08:50, 5 August 2022

2022-08-05T08:50:07Z

Unhammer: timings after dangswan's latest improvements :)

2022-08-05T08:42:59Z

timings after dangswan's latest improvements :)

Unhammer at 10:59, 27 June 2022

2022-06-27T10:59:09Z

Unhammer at 10:58, 27 June 2022

2022-06-27T10:58:44Z

Unhammer at 10:58, 27 June 2022

2022-06-27T10:58:18Z

Unhammer: formatting

2022-06-27T10:57:39Z

formatting

← Older revision		Revision as of 11:47, 4 October 2022
Line 24:		Line 24:


	5) '''TRIM SMARTER''': If you have sections that you already know contain matching elements in the analyser and bidix, you can ensure they're only trimmed with each other (ignoring other sections). Large regular expressions in particular can slow down trimming if you have to check every single bidix entry for a match – by putting <code><re></code> entries in a <code><section id="regex" type="standard"</code> in both files and changing Makefile.am to <code>lt-trim --match-section=regex@standard $^ $@</code>, lt-trim will ignore entries from sections other than regex@standard in bidix when trimming regex@standard from the analyser. (This will of course work with any kind of entry, not just regexes.)		5) '''TRIM SMARTER''': If you have sections that you already know contain matching elements in the analyser and bidix, you can ensure they're only trimmed with each other (ignoring other sections). Large regular expressions in particular can slow down trimming if you have to check every single bidix entry for a match – by putting <code><re></code> entries in a <code><section id="regex" type="standard"</code> in both files and changing Makefile.am to <code>lt-trim --match-section=regex@standard $^ $@</code>, lt-trim will ignore entries from sections other than regex@standard in bidix when trimming regex@standard from the analyser. (This will of course work with any kind of entry, not just regexes.) Time how long the lt-trim step takes first – for most pairs it's fast enough already.

← Older revision		Revision as of 11:45, 4 October 2022
Line 24:		Line 24:


	5) '''TRIM SMARTER''': If you have sections that you already know contain matching elements in the analyser and bidix, you can ensure they're only trimmed with each other (ignoring other sections). Large regular expressions in particular can slow down trimming if you have to check every single bidix entry for a match – by putting <code><re></code> entries in a <code><section id="regex" type="standard"</code> in both files and changing Makefile.am to <code>lt-trim --match-section=regex@standard $^ $@</code>, lt-trim will ignore entries from sections other than regex@standard in bidix when trimming regex@standard from the analyser. (This will of course work with any kind of entry, not just regexes.)		5) '''TRIM SMARTER''': If you have sections that you already know contain matching elements in the analyser and bidix, you can ensure they're only trimmed with each other (ignoring other sections). Large regular expressions in particular can slow down trimming if you have to check every single bidix entry for a match – by putting <code><re></code> entries in a <code><section id="regex" type="standard"</code> in both files and changing Makefile.am to <code>lt-trim --match-section=regex@standard $^ $@</code>, lt-trim will ignore entries from sections other than regex@standard in bidix when trimming regex@standard from the analyser. (This will of course work with any kind of entry, not just regexes.)


	and for comparison, with ''none'' of the above tricks:		and for comparison, with ''none'' of the above tricks:

@@ Line 23: / Line 23: @@
 * make clean in all three dirs, then <code>export LT_JOBS=yes && m -j e    </code> in nno-nob takes 42s
+) '''TRIM SMARTER''': If you have sections that you already know contain matching elements in the analyser and bidix, you can ensure they're only trimmed with each other (ignoring other sections). Large regular expressions in particular can slow down trimming if you have to check every single bidix entry for a match – by putting <code>&lt;re&gt;</code> entries in a <code>&lt;section id="regex" type="standard"</code> in both files and changing Makefile.am to <code>lt-trim --match-section=regex@standard $^ $@</code>, lt-trim will ignore entries from sections other than regex@standard in bidix when trimming regex@standard from the analyser. (This will of course work with any kind of entry, not just regexes.)
 and for comparison, with ''none'' of the above tricks:

@@ Line 22: / Line 22: @@
 * make clean in all three dirs, then <code>export LT_JOBS=yes && m -j langs</code> in nno-nob takes 1m20s
 * make clean in all three dirs, then <code>export LT_JOBS=yes && m -j e    </code> in nno-nob takes 42s
 and for comparison, with ''none'' of the above tricks:

@@ Line 22: / Line 22: @@
 * make clean in all three dirs, then <code>export LT_JOBS=yes && m -j langs</code> in nno-nob takes 1m20s
 * make clean in all three dirs, then <code>export LT_JOBS=yes && m -j e    </code> in nno-nob takes 42s
+and for comparison, with ''none'' of the above tricks:
+* make clean in all three dirs, then <code>export LT_JOBS=no && make langs</code> in nno-nob takes 3m14s
 [[Category:Development]]

@@ Line 5: / Line 5: @@
 ) '''PARALLELLISE''': Use <code>make -j</code> instead of just <code>make</code>, in order to use several processors for make goals that can be parallellised. (Note that if you're low on memory and your dictionaries are very large, you may want to cap it to <code>make -j2</code> or similar.)
-* <code>make clean && make   </code> in nno-nob takes 2m33s
+* <code>make clean && make   </code> in nno-nob takes 1m55s
-* <code>make clean && make -j</code> in nno-nob takes 1m24s
+* <code>make clean && make -j</code> in nno-nob takes 1m02s
 ) '''SPLIT SECTIONS''': Put <code><nowiki>export LT_JOBS=yes</nowiki></code> in your ~/.bashrc (or ~/.bash_profile) – this will let lttoolbox split dictionaries into sections of at most 50k entries and minimise them in parallel (and large sections get exponentially slower to minimise). You can also tweak the threshold with <code><nowiki>export LT_MAX_SECTION_ENTRIES=50000</nowiki></code>.
-* <code>make clean && export LT_JOBS=no  && make -j nob.automorf.bin</code> in nob takes 41s (typical situation where only monodix has been changed)
+* <code>make clean && export LT_JOBS=no  && make -j nob.automorf.bin</code> in nob takes 16s (typical situation where only monodix has been changed)
-* <code>make clean && export LT_JOBS=yes && make -j nob.automorf.bin</code> in nob takes 17s
+* <code>make clean && export LT_JOBS=yes && make -j nob.automorf.bin</code> in nob takes 8s
 ) '''ALLOCATE FASTER''': Try another malloc, e.g. tcmalloc or jemalloc. On Debian/Ubuntu, you would <code>sudo apt install libtcmalloc-minimal4</code> and then put <code><nowiki>function m () { export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4; make "$@"; }</nowiki></code> into your ~/.bashrc (or ~/.bash_profile). Now you can use <code>m</code> instead of <code>make</code> to compile with tcmalloc.
-* <code>make clean && export LT_JOBS=yes && make -j</code> in nno-nob takes 1m24s with regular malloc (glibc)
+* <code>make clean && export LT_JOBS=yes && make -j</code> in nno-nob takes 57s with regular malloc (glibc)
-* <code>make clean && export LT_JOBS=yes && m    -j</code> in nno-nob takes 1m03s with tcmalloc
+* <code>make clean && export LT_JOBS=yes && m    -j</code> in nno-nob takes 45s with tcmalloc
 ) '''BUILD LESS''': If you work mostly on one direction, you may want to make a debug goal in your Makefile.am for just that direction ([https://github.com/apertium/apertium-nno-nob/blob/d9c07db56a8ca3748039ff48db331082a0a33c3a/Makefile.am#L52..L56 example], used as <code>m -j e</code>)
+* make clean in all three dirs, then <code>export LT_JOBS=yes && m -j langs</code> in nno-nob takes 1m20s
+* make clean in all three dirs, then <code>export LT_JOBS=yes && m -j e    </code> in nno-nob takes 42s
 [[Category:Development]]

@@ Line 9: / Line 9: @@
-) '''SPLIT SECTIONS''': Put <nowiki><code>export LT_JOBS=yes</code></nowiki> in your ~/.bashrc (or ~/.bash_profile) – this will let lttoolbox split dictionaries into sections of at most 50k entries and minimise them in parallel (and large sections get exponentially slower to minimise). You can also tweak the threshold with <nowiki><code>export LT_MAX_SECTION_ENTRIES=50000</code></nowiki>.
+) '''SPLIT SECTIONS''': Put <code><nowiki>export LT_JOBS=yes</nowiki></code> in your ~/.bashrc (or ~/.bash_profile) – this will let lttoolbox split dictionaries into sections of at most 50k entries and minimise them in parallel (and large sections get exponentially slower to minimise). You can also tweak the threshold with <nowiki><code>export LT_MAX_SECTION_ENTRIES=50000</code></nowiki>.
 * <code>make clean && export LT_JOBS=no  && make -j nob.automorf.bin</code> in nob takes 41s (typical situation where only monodix has been changed)
 * <code>make clean && export LT_JOBS=yes && make -j nob.automorf.bin</code> in nob takes 17s

@@ Line 14: / Line 14: @@
-) '''ALLOCATE FASTER''': PutTry another malloc, e.g. tcmalloc or jemalloc. On Debian/Ubuntu, you would <code>sudo apt install libtcmalloc-minimal4</code> and then put <nowiki><code>function m () { export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4; make "$@"; }</code></nowiki> into your ~/.bashrc (or ~/.bash_profile). Now you can use <code>m</code> instead of <code>make</code> to compile with tcmalloc.
+) '''ALLOCATE FASTER''': PutTry another malloc, e.g. tcmalloc or jemalloc. On Debian/Ubuntu, you would <code>sudo apt install libtcmalloc-minimal4</code> and then put <code><nowiki>function m () { export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4; make "$@"; }</nowiki></code> into your ~/.bashrc (or ~/.bash_profile). Now you can use <code>m</code> instead of <code>make</code> to compile with tcmalloc.
 * <code>make clean && export LT_JOBS=yes && make -j</code> in nno-nob takes 1m24s with regular malloc (glibc)
 * <code>make clean && export LT_JOBS=yes && m    -j</code> in nno-nob takes 1m03s with tcmalloc