WikiExtractor - Revision history

Shardulc at 02:47, 10 March 2018

2018-03-10T02:47:57Z

Francis Tyers at 11:41, 18 August 2016

2016-08-18T11:41:13Z

Francis Tyers: Francis Tyers moved page Wiki Extractor to WikiExtractor

2016-07-09T13:37:12Z

Francis Tyers moved page Wiki Extractor to WikiExtractor

Francis Tyers: Created page with "'''WikiExtractor''' is a script for extracting a text corpus from Wikipedia dumps. You can find the script here: https://svn.code.sf.net/p/apertium/svn/trunk/apertium-tools/..."

2016-07-09T13:36:21Z

Created page with "'''WikiExtractor''' is a script for extracting a text corpus from Wikipedia dumps. You can find the script here: https://svn.code.sf.net/p/apertium/svn/trunk/apertium-tools/..."

New page

'''WikiExtractor''' is a script for extracting a text corpus from Wikipedia dumps.

You can find the script here:

https://svn.code.sf.net/p/apertium/svn/trunk/apertium-tools/WikiExtractor.py

You find a Wikipedia dump here:

https://dumps.wikimedia.org/backup-index.html

Navigate to the page of the language you are interested in, there will be a link called <code><language code></code>wiki.

You want the file that ends in: <code>-pages-articles.xml.bz2</code>, e.g. <code>euwiki-20160701-pages-articles.xml.bz2</code>

And you run it like this:

<pre>
$ python3 WikiExtractor.py --infn euwiki-20160701-pages-articles.xml.bz2
</pre>

It will spit a lot of output (the article titles) and output a file called <code>wiki.txt</code>. This is your corpus.

[[Category:Tools]]

← Older revision		Revision as of 02:47, 10 March 2018
Line 1:		Line 1:
			{{Github-unmigrated-tool}}
	'''WikiExtractor''' is a script for extracting a text corpus from Wikipedia dumps.		'''WikiExtractor''' is a script for extracting a text corpus from Wikipedia dumps.

@@ Line 12: / Line 12: @@
 You want the file that ends in: <code>-pages-articles.xml.bz2</code>, e.g. <code>euwiki-20160701-pages-articles.xml.bz2</code>
+You'll need to download the file, you can use <code>wget</code> or <code>curl</code> or something...
+<pre>
+$ wget <url>
+</pre>
 And you run it like this:

← Older revision	Revision as of 13:37, 9 July 2016
(No difference)