Wikipedia dumps - Revision history

Ilnar.salimzyan: add an internal link to Content Translation tool

2019-04-10T05:40:01Z

add an internal link to Content Translation tool

2018-11-15T14:16:49Z

Content Translation dumps

2018-11-15T14:16:31Z

Content Translation dumps

2018-11-15T14:16:18Z

Content Translation dumps

2018-11-15T14:12:41Z

Content Translation dumps

2018-11-15T14:05:40Z

Content Translation dumps

2018-11-15T13:59:09Z

Content Translation dumps

2018-11-15T13:58:43Z

Content Translation dumps

2018-11-15T13:58:11Z

Content Translation dumps

2018-11-15T13:56:53Z

Content Translation dumps

← Older revision		Revision as of 05:40, 10 April 2019
Line 1:		Line 1:
	Wikipedia dumps are useful for quickly getting a corpus. They are also the best corpora for making your language pair are useful for Wikipedia's Content Translation tool :-)		Wikipedia dumps are useful for quickly getting a corpus. They are also the best corpora for making your language pair are useful for Wikipedia's [[Content Translation]] tool :-)

	You download them from		You download them from

@@ Line 34: / Line 34: @@
 </pre>
-[[Image:Dwdiff-content-translation.png|500px]]
+[[Image:Dwdiff-content-translation.png|600px]]
 and find the original if you need it in <code>cut -f1 nb2nn.tsv</code>.

@@ Line 34: / Line 34: @@
 </pre>
-[[Image:Dwdiff-content-translation.png|300px]]
+[[Image:Dwdiff-content-translation.png|500px]]
 and find the original if you need it in <code>cut -f1 nb2nn.tsv</code>.

@@ Line 33: / Line 33: @@
 $ diff -U0 <(cut -f2 nb2nn.tsv) <(cut -f3 nb2nn.tsv) | dwdiff --diff-input -c | less
 </pre>
+[[Image:Dwdiff-content-translation.png|300px]]
 and find the original if you need it in <code>cut -f1 nb2nn.tsv</code>.

@@ Line 33: / Line 33: @@
 $ diff -U0 <(cut -f2 nb2nn.tsv) <(cut -f3 nb2nn.tsv) | dwdiff --diff-input -c | less
 </pre>
-and find the original if you need it in <code>cut -f1 nb2nn.tsv</code>
+and find the original if you need it in <code>cut -f1 nb2nn.tsv</code>.
+For some languages, you have to get the _2CODE files or _2_ files, e.g. sv2da is in https://dumps.wikimedia.org/other/contenttranslation/20180810/cx-corpora._2da.text.tmx.gz and da2sv is in https://dumps.wikimedia.org/other/contenttranslation/20180810/cx-corpora._2_.text.tmx.gz – so let's filter it to the languages we want:
+<pre>
+$ zcat ~/Nedlastingar/cx-corpora._2_.text.tmx.gz                             \
+ | xmlstarlet sel -t                                                         \
+    -m '//tu[@srclang="da" and tuv/prop/text()="mt" and tuv/@xml:lang="sv"]' \
+    -c 'tuv[./prop/text()="source"]/seg/text()' -o $'\t'                     \
+    -c 'tuv[./prop/text()="mt"]/seg/text()' -o $'\t'                         \
+    -c 'tuv[./prop/text()="user"]/seg/text()' -n                             \
+    > da2sv.tsv
+</pre>
 [[Category:Resources]]

@@ Line 31: / Line 31: @@
 Now view the word diff between MT and GOLD with:
 <pre>
-$ diff -U0 <(cut -f2 nb2nn.tsv) <(cut -f3 nb2nn.tsv) | dwdiff -c | less
+$ diff -U0 <(cut -f2 nb2nn.tsv) <(cut -f3 nb2nn.tsv) | dwdiff --diff-input -c | less
 </pre>
 and find the original if you need it in <code>cut -f1 nb2nn.tsv</code>

@@ Line 20: / Line 20: @@
-To turn a tmx into a <code>SOURCE\tMT\tGOLD</code> tsv, <code>sudo apt install xmlstarlet</code> and do:
+To turn a tmx into a <code>SOURCE\tMT\tGOLD</code> tab-separated text file, install xmlstarlet (<code>sudo apt install xmlstarlet</code>) and do:
 <pre>
 $ zcat ~/Nedlastingar/cx-corpora.nb2nn.text.tmx.gz          \

@@ Line 19: / Line 19: @@
 * https://www.mediawiki.org/wiki/Content_translation/Published_translations
-Turn tmx into a "SOURCE\tMT\tGOLD" tsv:
+To turn a tmx into a "SOURCE\tMT\tGOLD" tsv, `sudo apt install xmlstarlet` and do:
 <pre>
 $ zcat ~/Nedlastingar/cx-corpora.nb2nn.text.tmx.gz          \
@@ Line 27: / Line 28: @@
        -c 'tuv[./prop/text()="user"]/seg/text()'   -n       \
   > nb2nn.tsv
+</pre>
-# Now view the word diff between MT and GOLD with:
+Now view the word diff between MT and GOLD with:
+<pre>
 $ diff -U0 <(cut -f2 nb2nn.tsv) <(cut -f3 nb2nn.tsv) | dwdiff -c | less
-# and find the original if you need it in `cut -f1 nb2nn.tsv`
 </pre>
+and find the original if you need it in <code>cut -f1 nb2nn.tsv</code>
 [[Category:Resources]]