https://wiki.apertium.org/w/index.php?action=history&feed=atom&title=Extracting_bilingual_dictionaries_with_Giza%2B%2B 2026-05-18T08:56:41Z Revision history for this page on the wiki MediaWiki 1.34.1 https://wiki.apertium.org/w/index.php?title=Extracting_bilingual_dictionaries_with_Giza%2B%2B&diff=60136&oldid=prev 2016-09-26T11:53:33Z

<p></p> <table class="diff diff-contentalign-left" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">Revision as of 11:53, 26 September 2016</td> </tr><tr> <td colspan="2" class="diff-lineno">Line 126:</td> <td colspan="2" class="diff-lineno">Line 126:</td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><br /></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><br /></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>[[Category:Documentation]]</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>[[Category:Documentation]]</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>[[Category:Documentation in English]]</div></td> </tr> </table>

Rcrowther https://wiki.apertium.org/w/index.php?title=Extracting_bilingual_dictionaries_with_Giza%2B%2B&diff=33411&oldid=prev 2012-04-07T19:07:46Z

<p></p> <table class="diff diff-contentalign-left" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">Revision as of 19:07, 7 April 2012</td> </tr><tr> <td colspan="2" class="diff-lineno">Line 1:</td> <td colspan="2" class="diff-lineno">Line 1:</td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>{{TOCD}}</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>{{TOCD}}</div></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><br /></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><br /></td> </tr> <tr> <td class="diff-marker" data-marker="−"></td> <td class="diff-deletedline diff-side-deleted"><div>This <del class="diffchange diffchange-inline">is</del> <del class="diffchange diffchange-inline">a placeholder for documentation on</del> how to use a parallel corpus and Giza++ to extract bilingual dictionaries for Apertium.</div></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>This <ins class="diffchange diffchange-inline">page</ins> <ins class="diffchange diffchange-inline">explains</ins> how to use a parallel corpus and Giza++ to extract bilingual dictionaries for Apertium.</div></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><br /></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><br /></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>==You need==</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>==You need==</div></td> </tr> </table>

Francis Tyers https://wiki.apertium.org/w/index.php?title=Extracting_bilingual_dictionaries_with_Giza%2B%2B&diff=32065&oldid=prev 2012-02-07T14:08:18Z

<p></p> <table class="diff diff-contentalign-left" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">Revision as of 14:08, 7 February 2012</td> </tr><tr> <td colspan="2" class="diff-lineno">Line 2:</td> <td colspan="2" class="diff-lineno">Line 2:</td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><br /></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><br /></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>This is a placeholder for documentation on how to use a parallel corpus and Giza++ to extract bilingual dictionaries for Apertium.</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>This is a placeholder for documentation on how to use a parallel corpus and Giza++ to extract bilingual dictionaries for Apertium.</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><br /></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>==You need==</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><br /></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>* Giza++</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>* Moses</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>* Morphological analyser/disambiguator for each language</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>* Some scripts</div></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><br /></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><br /></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>==Get your corpus==</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>==Get your corpus==</div></td> </tr> </table>

Francis Tyers https://wiki.apertium.org/w/index.php?title=Extracting_bilingual_dictionaries_with_Giza%2B%2B&diff=32064&oldid=prev 2012-02-07T14:06:09Z

<p><span dir="auto"><span class="autocomment">Process corpus</span></span></p> <table class="diff diff-contentalign-left" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">Revision as of 14:06, 7 February 2012</td> </tr><tr> <td colspan="2" class="diff-lineno">Line 37:</td> <td colspan="2" class="diff-lineno">Line 37:</td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><br /></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><br /></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>&lt;pre&gt;</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>&lt;pre&gt;</div></td> </tr> <tr> <td class="diff-marker" data-marker="−"></td> <td class="diff-deletedline diff-side-deleted"><div>$ head forvaltningsordbok.tagged<del class="diffchange diffchange-inline">.raw</del>.nob | python process-tags.py nob.process-relabel &gt; forvaltningsordbok.tagged.clean.nob</div></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>$ head forvaltningsordbok.tagged.nob | python process-tags.py nob.process-relabel &gt; forvaltningsordbok.tagged.clean.nob</div></td> </tr> <tr> <td class="diff-marker" data-marker="−"></td> <td class="diff-deletedline diff-side-deleted"><div>$ head forvaltningsordbok.tagged<del class="diffchange diffchange-inline">.raw</del>.sme | python process-tags.py sme.process-relabel &gt; forvaltningsordbok.tagged.clean.sme</div></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>$ head forvaltningsordbok.tagged.sme | python process-tags.py sme.process-relabel &gt; forvaltningsordbok.tagged.clean.sme</div></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>&lt;/pre&gt;</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>&lt;/pre&gt;</div></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><br /></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><br /></td> </tr> </table>

Francis Tyers https://wiki.apertium.org/w/index.php?title=Extracting_bilingual_dictionaries_with_Giza%2B%2B&diff=32063&oldid=prev 2012-02-07T14:03:56Z

<p><span dir="auto"><span class="autocomment">Prune with relative frequency</span></span></p> <table class="diff diff-contentalign-left" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">Revision as of 14:03, 7 February 2012</td> </tr><tr> <td colspan="2" class="diff-lineno">Line 105:</td> <td colspan="2" class="diff-lineno">Line 105:</td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>==Prune with relative frequency==</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>==Prune with relative frequency==</div></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><br /></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><br /></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>Use the &lt;code&gt;relative-freq.py&lt;/code&gt; script to generate relative frequency lists of in-domain and out-domain corpora.</div></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><br /></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><br /></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>&lt;pre&gt;</div></td> </tr> <tr> <td class="diff-marker" data-marker="−"></td> <td class="diff-deletedline diff-side-deleted"><div>While waiting: [[Generating_lexical-selection_rules_from_a_parallel_corpus|This page will shorten the waiting time]]</div></td> <td colspan="2" class="diff-empty diff-side-added"></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><br /></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>&lt;/pre&gt;</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><br /></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>Use the &lt;code&gt;extract-candidate-terms.py&lt;/code&gt; script to give a confidence to each alignment based on in-domain and out-domain relative frequency.</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><br /></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>&lt;pre&gt;</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>$ python extract-candidate-terms.py lex.e2f.sorted forvaltningsordbok.relfreq.nob no.crp.txt.relfreq forvaltningsordbok.freq.nob no.crp.txt.freq | sort -gr</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>&lt;/pre&gt;</div></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><br /></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><br /></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><br /></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><br /></td> </tr> </table>

Francis Tyers https://wiki.apertium.org/w/index.php?title=Extracting_bilingual_dictionaries_with_Giza%2B%2B&diff=32062&oldid=prev 2012-02-07T14:00:25Z

<p><span dir="auto"><span class="autocomment">Extract lexicon</span></span></p> <table class="diff diff-contentalign-left" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">Revision as of 14:00, 7 February 2012</td> </tr><tr> <td colspan="2" class="diff-lineno">Line 84:</td> <td colspan="2" class="diff-lineno">Line 84:</td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><br /></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><br /></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>==Extract lexicon==</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>==Extract lexicon==</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><br /></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>The interesting files are in &lt;code&gt;model/&lt;/code&gt;. Particularly the &lt;code&gt;lex.e2f&lt;/code&gt; and &lt;code&gt;lex.f2e&lt;/code&gt; files.</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><br /></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>&lt;pre&gt;</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>$ head model/lex.e2f</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>*planlovens plánalága&lt;N&gt; 0.1666667</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>*mantallsorganiseringa *SVLa 0.0666667</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>*mantallsorganiseringa *MagnhildMathisen 0.1428571</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>*mantallsorganiseringa sámediggejoavku&lt;N&gt; 0.0020040</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>*mantallsorganiseringa áirras&lt;N&gt; 0.0005531</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>*mantallsorganiseringa guolástus+ráddi+addit&lt;V&gt;&lt;TV&gt; 0.2500000</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>&lt;/pre&gt;</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><br /></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>Sort the &lt;code&gt;e2f&lt;/code&gt; file:</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><br /></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>&lt;pre&gt;</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>$ cat model/lex.e2f | sort &gt; model/lex.e2f.sorted</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>&lt;/pre&gt;</div></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><br /></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><br /></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>==Prune with relative frequency==</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>==Prune with relative frequency==</div></td> </tr> </table>

Francis Tyers https://wiki.apertium.org/w/index.php?title=Extracting_bilingual_dictionaries_with_Giza%2B%2B&diff=32061&oldid=prev 2012-02-07T13:57:10Z

<p><span dir="auto"><span class="autocomment">Align corpus</span></span></p> <table class="diff diff-contentalign-left" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">Revision as of 13:57, 7 February 2012</td> </tr><tr> <td colspan="2" class="diff-lineno">Line 72:</td> <td colspan="2" class="diff-lineno">Line 72:</td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>&lt;pre&gt;</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>&lt;pre&gt;</div></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>$ nohup perl ~/local/bin/scripts-20120109-1229/training/train-model.perl -scripts-root-dir \</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>$ nohup perl ~/local/bin/scripts-20120109-1229/training/train-model.perl -scripts-root-dir \</div></td> </tr> <tr> <td class="diff-marker" data-marker="−"></td> <td class="diff-deletedline diff-side-deleted"><div> /home/fran/local/bin/scripts-20120109-1229/ -root-dir . -corpus <del class="diffchange diffchange-inline">europarl</del>.<del class="diffchange diffchange-inline">tag-tok</del> \</div></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div> /home/fran/local/bin/scripts-20120109-1229/ -root-dir . -corpus <ins class="diffchange diffchange-inline">forvaltningsordbok</ins>.<ins class="diffchange diffchange-inline">tagged.clean</ins> \</div></td> </tr> <tr> <td class="diff-marker" data-marker="−"></td> <td class="diff-deletedline diff-side-deleted"><div> -f <del class="diffchange diffchange-inline">en</del> -e <del class="diffchange diffchange-inline">es</del> -alignment grow-diag-final-and -reordering msd-bidirectional-fe \</div></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div> -f <ins class="diffchange diffchange-inline">sme</ins> -e <ins class="diffchange diffchange-inline">nob</ins> -alignment grow-diag-final-and -reordering msd-bidirectional-fe \</div></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div> -lm 0:5:/home/fran/corpora/europarl/europarl.lm:0 &gt;log 2&gt;&amp;1 &amp;</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div> -lm 0:5:/home/fran/corpora/europarl/europarl.lm:0 &gt;log 2&gt;&amp;1 &amp;</div></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>&lt;/pre&gt;</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>&lt;/pre&gt;</div></td> </tr> </table>

Francis Tyers https://wiki.apertium.org/w/index.php?title=Extracting_bilingual_dictionaries_with_Giza%2B%2B&diff=32060&oldid=prev 2012-02-07T13:56:13Z

<p><span dir="auto"><span class="autocomment">Align corpus</span></span></p> <table class="diff diff-contentalign-left" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">Revision as of 13:56, 7 February 2012</td> </tr><tr> <td colspan="2" class="diff-lineno">Line 67:</td> <td colspan="2" class="diff-lineno">Line 67:</td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><br /></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><br /></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>==Align corpus==</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>==Align corpus==</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><br /></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>Now use Moses to align your corpus in order to get the probabilistic dictionaries:</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><br /></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>&lt;pre&gt;</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>$ nohup perl ~/local/bin/scripts-20120109-1229/training/train-model.perl -scripts-root-dir \</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div> /home/fran/local/bin/scripts-20120109-1229/ -root-dir . -corpus europarl.tag-tok \</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div> -f en -e es -alignment grow-diag-final-and -reordering msd-bidirectional-fe \</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div> -lm 0:5:/home/fran/corpora/europarl/europarl.lm:0 &gt;log 2&gt;&amp;1 &amp;</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>&lt;/pre&gt;</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><br /></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>Note: Remember to change all the paths in the above command!</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><br /></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>You'll need an LM file, but you can copy it from a previous Moses installation. If you don't have one, make an empty file and put a few words in it. We won't be using the LM anyway.</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><br /></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>This takes a while, probably about a day. So leave it running and go and make a soufflé, or chop some wood or something.</div></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><br /></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><br /></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>==Extract lexicon==</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>==Extract lexicon==</div></td> </tr> </table>

Francis Tyers https://wiki.apertium.org/w/index.php?title=Extracting_bilingual_dictionaries_with_Giza%2B%2B&diff=32059&oldid=prev 2012-02-07T13:55:01Z

<p><span dir="auto"><span class="autocomment">Process corpus</span></span></p> <table class="diff diff-contentalign-left" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">Revision as of 13:55, 7 February 2012</td> </tr><tr> <td colspan="2" class="diff-lineno">Line 37:</td> <td colspan="2" class="diff-lineno">Line 37:</td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><br /></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><br /></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>&lt;pre&gt;</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>&lt;pre&gt;</div></td> </tr> <tr> <td class="diff-marker" data-marker="−"></td> <td class="diff-deletedline diff-side-deleted"><div>$ head forvaltningsordbok.tagged.raw.nob | python process-tags.py nob.relabel </div></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>$ head forvaltningsordbok.tagged.raw.nob | python process-tags.py nob.<ins class="diffchange diffchange-inline">process-</ins>relabel <ins class="diffchange diffchange-inline">&gt; forvaltningsordbok.tagged.clean.nob</ins></div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>$ head forvaltningsordbok.tagged.raw.sme | python process-tags.py sme.process-relabel &gt; forvaltningsordbok.tagged.clean.sme</div></td> </tr> <tr> <td class="diff-marker" data-marker="−"></td> <td class="diff-deletedline diff-side-deleted"><div>1&lt;det&gt;&lt;qnt&gt; 558,4&lt;det&gt;&lt;qnt&gt; million&lt;n&gt;&lt;m&gt; krone&lt;n&gt;&lt;m&gt; i&lt;pr&gt; *spillemidler til&lt;pr&gt; idrett+formål&lt;n&gt;&lt;nt&gt; for&lt;pr&gt; 2010&lt;det&gt;&lt;qnt&gt; </div></td> <td colspan="2" class="diff-empty diff-side-added"></td> </tr> <tr> <td class="diff-marker" data-marker="−"></td> <td class="diff-deletedline diff-side-deleted"><div>søk&lt;n&gt;&lt;nt&gt; hos&lt;pr&gt; kulturdepartement&lt;n&gt;&lt;nt&gt; </div></td> <td colspan="2" class="diff-empty diff-side-added"></td> </tr> <tr> <td class="diff-marker" data-marker="−"></td> <td class="diff-deletedline diff-side-deleted"><div>søk&lt;n&gt;&lt;nt&gt; på&lt;pr&gt; hel&lt;adj&gt; regjering&lt;n&gt;&lt;m&gt; *no </div></td> <td colspan="2" class="diff-empty diff-side-added"></td> </tr> <tr> <td class="diff-marker" data-marker="−"></td> <td class="diff-deletedline diff-side-deleted"><div>tips&lt;n&gt;&lt;nt&gt; en&lt;det&gt;&lt;qnt&gt; venn&lt;n&gt;&lt;m&gt; </div></td> <td colspan="2" class="diff-empty diff-side-added"></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>&lt;/pre&gt;</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>&lt;/pre&gt;</div></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><br /></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><br /></td> </tr> </table>

Francis Tyers https://wiki.apertium.org/w/index.php?title=Extracting_bilingual_dictionaries_with_Giza%2B%2B&diff=32058&oldid=prev 2012-02-07T13:51:21Z

<p><span dir="auto"><span class="autocomment">Process corpus</span></span></p> <table class="diff diff-contentalign-left" data-mw="interface"> <col class="diff-marker" /> <col class="diff-content" /> <col class="diff-marker" /> <col class="diff-content" /> <tr class="diff-title" lang="en"> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">← Older revision</td> <td colspan="2" style="background-color: #fff; color: #222; text-align: center;">Revision as of 13:51, 7 February 2012</td> </tr><tr> <td colspan="2" class="diff-lineno">Line 34:</td> <td colspan="2" class="diff-lineno">Line 34:</td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>&lt;/pre&gt;</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>&lt;/pre&gt;</div></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><br /></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><br /></td> </tr> <tr> <td class="diff-marker" data-marker="−"></td> <td class="diff-deletedline diff-side-deleted"><div>Then remove the tags which are unnecessary (for example number, and tense)<del class="diffchange diffchange-inline">:</del></div></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>Then remove the tags which are unnecessary<ins class="diffchange diffchange-inline"> using the process tags script (&lt;code&gt;process-tags.py&lt;/code&gt;)</ins> (for example number, and tense)<ins class="diffchange diffchange-inline">, e.g.</ins></div></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><br /></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><br /></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>&lt;pre&gt;</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>&lt;pre&gt;</div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>$ head forvaltningsordbok.tagged.raw.nob | python process-tags.py nob.relabel </div></td> </tr> <tr> <td class="diff-marker" data-marker="−"></td> <td class="diff-deletedline diff-side-deleted"><br /></td> <td colspan="2" class="diff-empty diff-side-added"></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>1&lt;det&gt;&lt;qnt&gt; 558,4&lt;det&gt;&lt;qnt&gt; million&lt;n&gt;&lt;m&gt; krone&lt;n&gt;&lt;m&gt; i&lt;pr&gt; *spillemidler til&lt;pr&gt; idrett+formål&lt;n&gt;&lt;nt&gt; for&lt;pr&gt; 2010&lt;det&gt;&lt;qnt&gt; </div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>søk&lt;n&gt;&lt;nt&gt; hos&lt;pr&gt; kulturdepartement&lt;n&gt;&lt;nt&gt; </div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>søk&lt;n&gt;&lt;nt&gt; på&lt;pr&gt; hel&lt;adj&gt; regjering&lt;n&gt;&lt;m&gt; *no </div></td> </tr> <tr> <td colspan="2" class="diff-empty diff-side-deleted"></td> <td class="diff-marker" data-marker="+"></td> <td class="diff-addedline diff-side-added"><div>tips&lt;n&gt;&lt;nt&gt; en&lt;det&gt;&lt;qnt&gt; venn&lt;n&gt;&lt;m&gt; </div></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><div>&lt;/pre&gt;</div></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><div>&lt;/pre&gt;</div></td> </tr> <tr> <td class="diff-marker"></td> <td class="diff-context diff-side-deleted"><br /></td> <td class="diff-marker"></td> <td class="diff-context diff-side-added"><br /></td> </tr> </table>

Francis Tyers