Difference between revisions of "Wikipedia dumps"
Jump to navigation
Jump to search
Hectoralos (talk | contribs) (+ wikiextractor) |
|||
| Line 8: | Line 8: | ||
* [[Wikipedia Extractor]] – a python script that tries to remove all formatting |
* [[Wikipedia Extractor]] – a python script that tries to remove all formatting |
||
* [https://github.com/attardi/wikiextractor wikiextractor] – another python script that removes all formatting (with different options), putting XML marks just to know when begins and ends everty single article |
|||
* [https://gist.github.com/unhammer/3372222878580d1e4c6f mwdump-to-pandoc] – shell wrapper around [http://pandoc.org/ pandoc] (see the usage.sh below the script for how to use) |
* [https://gist.github.com/unhammer/3372222878580d1e4c6f mwdump-to-pandoc] – shell wrapper around [http://pandoc.org/ pandoc] (see the usage.sh below the script for how to use) |
||
* [[Calculating_coverage#More_involved_scripts]] – an ugly shell script that does the job |
* [[Calculating_coverage#More_involved_scripts]] – an ugly shell script that does the job |
||
Revision as of 07:41, 15 May 2018
Wikipedia dumps are useful for quickly getting a corpus. They are also the best corpora for making your language pair are useful for Wikipedia's Content Translation tool :-)
You download them from
There are several tools for turning dumps into useful plaintext, e.g.
- Wikipedia Extractor – a python script that tries to remove all formatting
- wikiextractor – another python script that removes all formatting (with different options), putting XML marks just to know when begins and ends everty single article
- mwdump-to-pandoc – shell wrapper around pandoc (see the usage.sh below the script for how to use)
- Calculating_coverage#More_involved_scripts – an ugly shell script that does the job
- wp2txt – some ruby thing (does this work?)
