Wikipedia Extractor - Revision history

Firespeaker at 18:55, 30 January 2023

2023-01-30T18:55:09Z

2023-01-30T18:53:22Z

Steps

2023-01-30T18:53:06Z

2023-01-30T18:52:42Z

2021-08-25T15:51:47Z

Steps

2021-06-30T19:00:26Z

Steps

2021-06-30T19:00:09Z

2020-06-04T21:37:28Z

2017-09-13T08:06:12Z

2017-04-27T03:03:11Z

@@ Line 2: / Line 2: @@
 == Goal ==
-This tool extracts main text from xml [[Wikipedia dump]] files (at http://dumps.wikimedia.org/backup-index.html, ideally the "pages-articles" file), producing a text corpus, which is useful for training unsupervised part-of-speech taggers, n-gram language models, etc.
+This tool extracts main text from xml [[Wikipedia dump]] files (at https://dumps.wikimedia.org/backup-index.html, ideally the "'''pages-articles.xml.bz2'''" file), producing a text corpus, which is useful for training unsupervised part-of-speech taggers, n-gram language models, etc.
 It was modified by a number of people, including by BenStobaugh during Google Code-In 2013, and can be cloned from GitHub at [https://github.com/apertium/WikiExtractor https://github.com/apertium/WikiExtractor].

@@ Line 27: / Line 27: @@
 #: <code>$ python3 WikiExtractor.py --infn xyzwiki-20230120-pages-articles.xml.bz2 --compress</code>
-This will output a file called <code>wiki.txt.bz2</code>.  You will probably want to rename it to something like <code>xyz.wikipedia.20210620.txt.bz2</code>.
+This will output a file called <code>wiki.txt.bz2</code>.  You will probably want to rename it to something like <code>xyz.wikipedia.20230120.txt.bz2</code>.
 ==See also==

@@ Line 23: / Line 23: @@
 #: <code>$ wget https://raw.githubusercontent.com/apertium/WikiExtractor/master/WikiExtractor.py</code>
 # Download the Wikipedia dump for the language in question from http://dumps.wikimedia.org/backup-index.html.  The below uses a hypothetical file name for a language with the <tt>xyz</tt> code:
-#: <code>$ wget https://dumps.wikimedia.org/xyzwiki/20210620/xyzwiki-20230120-pages-articles.xml.bz2</code>
+#: <code>$ wget https://dumps.wikimedia.org/xyzwiki/20230120/xyzwiki-20230120-pages-articles.xml.bz2</code>
 # Run the script on the Wikipedia dump file:
-#: <code>$ python3 WikiExtractor.py --infn xyzwiki-20210620-pages-articles.xml.bz2 --compress</code>
+#: <code>$ python3 WikiExtractor.py --infn xyzwiki-20230120-pages-articles.xml.bz2 --compress</code>
 This will output a file called <code>wiki.txt.bz2</code>.  You will probably want to rename it to something like <code>xyz.wikipedia.20210620.txt.bz2</code>.

@@ Line 23: / Line 23: @@
 #: <code>$ wget https://raw.githubusercontent.com/apertium/WikiExtractor/master/WikiExtractor.py</code>
 # Download the Wikipedia dump for the language in question from http://dumps.wikimedia.org/backup-index.html.  The below uses a hypothetical file name for a language with the <tt>xyz</tt> code:
-#: <code>$ wget https://dumps.wikimedia.org/xyzwiki/20210620/xyzwiki-20210620-pages-articles.xml.bz2</code>
+#: <code>$ wget https://dumps.wikimedia.org/xyzwiki/20210620/xyzwiki-20230120-pages-articles.xml.bz2</code>
 # Run the script on the Wikipedia dump file:
 #: <code>$ python3 WikiExtractor.py --infn xyzwiki-20210620-pages-articles.xml.bz2 --compress</code>

@@ Line 22: / Line 22: @@
 # Get the <code>WikiExtractor.py</code> script from https://github.com/apertium/WikiExtractor:
 #: <code>$ wget https://raw.githubusercontent.com/apertium/WikiExtractor/master/WikiExtractor.py</code>
-# Download the Wikipedia dump for the language in question from http://dumps.wikimedia.org/backup-index.html.  The below uses a hypothetical file name:
+# Download the Wikipedia dump for the language in question from http://dumps.wikimedia.org/backup-index.html.  The below uses a hypothetical file name for a language with the <tt>xyz</tt> code:
 #: <code>$ wget https://dumps.wikimedia.org/xyzwiki/20210620/xyzwiki-20210620-pages-articles.xml.bz2</code>
 # Run the script on the Wikipedia dump file:

@@ Line 17: / Line 17: @@
 == Steps ==
+Here's a simple step-by-step guide to the above.
 # Get the <code>WikiExtractor.py</code> script from https://github.com/apertium/WikiExtractor:

@@ Line 4: / Line 4: @@
 This tool extracts main text from xml [[Wikipedia dump]] files (at http://dumps.wikimedia.org/backup-index.html, ideally the "pages-articles" file), producing a text corpus, which is useful for training unsupervised part-of-speech taggers, n-gram language models, etc.
-It was written by BenStobaugh during GCI-2013 and can be downloaded from SVN at [https://svn.code.sf.net/p/apertium/svn/trunk/apertium-tools/WikiExtractor.py https://svn.code.sf.net/p/apertium/svn/trunk/apertium-tools/WikiExtractor.py]
+It was modified by a number of people, including by BenStobaugh during Google Code-In 2013, and can be cloned from GitHub at [https://github.com/apertium/WikiExtractor https://github.com/apertium/WikiExtractor].
 This version is much simpler than the old version. This version auto-removes any formatting and only outputs the text to one file. To use it, simply use the following command in your terminal, where dump.xml is the Wikipedia dump

@@ Line 9: / Line 9: @@
  $ python3 WikiExtractor.py --infn dump.xml.bz2
+(Note: If you are on a Mac, make sure that <tt>--</tt> is really two hyphens and not an em-dash like this: &mdash;).
 This will run through all of the articles, get all of the text and put it in wiki.txt. This version also supports compression (BZip2 and Gzip), so you can use <code>dump.xml.bz2</code> or <code>dump.xml.gz</code> instead of <code>dump.xml.</code> You can also compress (Bzip2) the output file by adding <code>--compress</code> to the command.