Difference between revisions of "User:Gang Chen"

Revision as of 08:52, 12 July 2013

Name: Gang Chen

Email: pkuchengang@gmail.com

IRC: Gang

SourceForge: elephantgcc

GitHub Repo: https://github.com/elephantgcc

I'm working with Apertium for the GSoC 2013, on the project "Sliding Window Part of Speech Tagger for Apertium".

my proposal is here: Proposal

This tool extracts main text from Wikipedia, producing a text corpus, which is useful for part of speech tagging, language model training, etc.

1. Get the script.

  Let's Take Chinese as an example, download the file zhwiki-20130625-pages-articles.xml.bz2 on this page http://dumps.wikimedia.org/zhwiki/20130625/

  Alternatively, we can download the latest version on this page, http://dumps.wikimedia.org/zhwiki/lastest/

3. Use the script.

4. clean up "<>" tags

@@ Line 21: / Line 21: @@
 https://svn.code.sf.net/p/apertium/svn/branches/apertium-swpost/apertium
-=== Current Progress ===
+=== LSW tagger: Current Progress ===
 http://wiki.apertium.org/w/index.php?title=User:Gang_Chen/GSoC_2013_Progress
+== Useful tools ==
+=== Wikipedia extractor ===
+==== tool ====
+http://code.google.com/p/natural-language-qa/source/browse/MakeCorpus/WikiExtractor.py
+==== target ====
+This tool extracts main text from Wikipedia, producing a text corpus, which is useful for part of speech tagging, language model training, etc.
+==== usage ====
+. Get the script.
+http://code.google.com/p/natural-language-qa/source/browse/MakeCorpus/WikiExtractor.py
+. Download the Wikipedia dump file from http://dumps.wikimedia.org/backup-index.html
+   Let's Take Chinese as an example, download the file '''zhwiki-20130625-pages-articles.xml.bz2''' on this page http://dumps.wikimedia.org/zhwiki/20130625/
+   Alternatively, we can download the latest version on this page, http://dumps.wikimedia.org/zhwiki/lastest/
+. Use the script.
+<pre>
+</pre>
+. clean up "<>" tags