View source for WikiExtractor

'''WikiExtractor''' is a script for extracting a text corpus from Wikipedia dumps.

You can find the script here:

https://svn.code.sf.net/p/apertium/svn/trunk/apertium-tools/WikiExtractor.py

You find a Wikipedia dump here:

https://dumps.wikimedia.org/backup-index.html

Navigate to the page of the language you are interested in, there will be a link called <code>&lt;language code&gt;</code>wiki.

You want the file that ends in: <code>-pages-articles.xml.bz2</code>, e.g. <code>euwiki-20160701-pages-articles.xml.bz2</code>

And you run it like this:

<pre>
$ python3 WikiExtractor.py --infn euwiki-20160701-pages-articles.xml.bz2
</pre>

It will spit a lot of output (the article titles) and output a file called <code>wiki.txt</code>. This is your corpus.

[[Category:Tools]]