Aravot.am - Revision history

Dtvrij74: /* Sentence Aligning */

2013-01-09T00:17:42Z

Sentence Aligning

Dtvrij74: /* Scraping */

2013-01-08T23:45:17Z

Scraping

Dtvrij74: /* Scraping */

2013-01-08T23:44:26Z

Scraping

Dtvrij74: /* Sentence Aligning */

2013-01-08T01:05:27Z

Sentence Aligning

Dtvrij74: /* Sentence Aligning */

2013-01-08T01:02:15Z

Sentence Aligning

Dtvrij74: /* Sentence Segmentation */

2013-01-08T00:35:16Z

Sentence Segmentation

Dtvrij74: /* Sentence Aligning */

2013-01-08T00:31:41Z

Sentence Aligning

Dtvrij74: /* Scraping and aligning */

2013-01-08T00:22:23Z

Scraping and aligning

Dtvrij74: /* Scraping and aligning */

2013-01-08T00:13:40Z

Scraping and aligning

Dtvrij74: /* Scraping and aligning */

2013-01-08T00:00:17Z

Scraping and aligning

@@ Line 259: / Line 259: @@
 (The above quote is from [http://mokk.bme.hu/resources/hunalign/ here].)
-A sample Armenian to English dictionary  based on Apertium's hye-eng bidix (converted to a Hunalign-compatible dictionary format via a simple Python script) can be found [http://sebsauvage.net/paste/?ea47c257cf471482#7RcEqbvdH1NwNHAlPrwUDPAVkgfxrkwbzpQkdPoI0Go= here].
+A sample Armenian to English dictionary  based on Apertium's hye-eng bidix (converted to a Hunalign-compatible dictionary format via [http://danielhonline.com/2013/01/apertium-bidix-to-hunalign-dictionary-converter-script/ a simple Python script]) can be found [http://sebsauvage.net/paste/?ea47c257cf471482#7RcEqbvdH1NwNHAlPrwUDPAVkgfxrkwbzpQkdPoI0Go= here].
 Information about all Hunalign commands can be found [http://mokk.bme.hu/resources/hunalign/ here].

@@ Line 225: / Line 225: @@
 Loop through all the article URLs for each day and language and get the HTML tags that store article contents (e.g. <nowiki><pre>, <p>, etc.</nowiki>) by using Beautiful Soup. Append the contents to a string.
-The Python 3 scraper for Aravot.am can be found [http://wiki.apertium.org/wiki/Aravot.am here].
+The Python 3 scraper for Aravot.am can be found [http://danielhonline.com/2013/01/scrape-armenian-english-and-russian-articles-from-aravot-am/  here].
 ===Sentence Segmentation===

@@ Line 224: / Line 224: @@
 Loop through all the article URLs for each day and language and get the HTML tags that store article contents (e.g. <nowiki><pre>, <p>, etc.</nowiki>) by using Beautiful Soup. Append the contents to a string.
+The Python 3 scraper for Aravot.am can be found [http://wiki.apertium.org/wiki/Aravot.am here].
 ===Sentence Segmentation===

@@ Line 289: / Line 289: @@
 You need to loop through all the files that you generated after the sentence segmentation step. You can create a simple Python script to do so:
 <pre>
-os.popen('~/hun3/hunalign-1.1/src/hunalign/hunalign -utf -bisent -text dict.txt '+en+' '+hy+' > ~/hun3/hunalign-1.1/align/'+year+'/'+month+'/hy-'+lang+'/'+num).read()
+os.popen('~/hunalign-1.1/src/hunalign/hunalign -utf -bisent -text dict.txt '+en+' '+hy+' > ~/hunalign-1.1/align/'+year+'/'+month+'/hy-'+lang+'/'+num).read()
 </pre>
-* Hunalign is under this file path: <code>~/hun3/hunalign-1.1/src/hunalign/hunalign</code>.
+* Hunalign is under this file path: <code>~/hunalign-1.1/src/hunalign/hunalign</code>.
 * <code>en</code> is the variable with the file path to the source language article
 * <code>hy</code> is the variable with the file path to the target language article

@@ Line 272: / Line 272: @@
 Use the <code>-utf</code> argument to input and output in <code>UTF-8</code>.
-Use the <code>-bisent<code> argument to only print bisentences (one-to-one alignment segments).
+Use the <code>-bisent</code> argument to only print bisentences (one-to-one alignment segments).
 An example command would be:

@@ Line 239: / Line 239: @@
 After tokenisation, Article contents can be saved to a file. The file structure can be: year/month (the month folder contains the articles, in all three languages, for that month).
 You will have to loop through this process for all files for sentence segmentation.
 ===Sentence Aligning===
 Hunalign will be used to align sentences from two languages. For example, hy-en and hy-ru pairs articles can be ran through.

@@ Line 255: / Line 255: @@
 </blockquote>
 (The above quote is from [http://mokk.bme.hu/resources/hunalign/ here].)
 A sample Armenian to English dictionary  based on Apertium's hye-eng bidix (converted to a Hunalign-compatible dictionary format via a simple Python script) can be found [http://sebsauvage.net/paste/?ea47c257cf471482#7RcEqbvdH1NwNHAlPrwUDPAVkgfxrkwbzpQkdPoI0Go= here].
 Information about all Hunalign commands can be found [http://mokk.bme.hu/resources/hunalign/ here].
-If you do not have a bilingual dictionary, you can use the -realign argument.
+If you do not have a bilingual dictionary, you can use the <code>-realign</code> argument.
 Hunalign will build its own dictionary based on the two text files (in two languages) that you input. You can input a 0 byte file as a placeholder for the dictionary argument.
@@ Line 268: / Line 269: @@
 </pre>
-Use the -utf argument to input and output in UTF-8.
+Use the <code>-utf</code> argument to input and output in <code>UTF-8</code>.
-Use the -bisent argument to only print bisentences (one-to-one alignment segments).
+Use the <code>-bisent<code> argument to only print bisentences (one-to-one alignment segments).
 An example command would be:
@@ Line 279: / Line 280: @@
 In this command:
 * The command is executed in Hunspell's root directory.
-* The -utf argument (explained above) is used.
+* The <code>-utf</code> argument (explained above) is used.
-* The -bisent command (explained above) is used.
+* The <code>-bisent</code> command (explained above) is used.
 * The bilingual dictionary used is called <code>dict.txt</code>
 *<code>en_2012_06_23_84517.txt</code> is the file in the source language.
@@ Line 304: / Line 305: @@
 * The second column contains the (supposedly) corresponding sentence from the target language text.
 * The third column is a confidence score for the alignment. It averages all the scores to determine the quality of the alignment.
-* At the bottom, there will be the quality score. I found that the quality score increases with the amount of content in the input articles.
+* At the bottom, there will be the quality score. It seems like the quality score increases with the amount of content in the input articles.

@@ Line 282: / Line 282: @@
 * The -bisent command (explained above) is used.
 * The bilingual dictionary used is called <code>dict.txt</code>
-*<code>en_2012_06_23_84517.txt </code> is the file in the source language.
+*<code>en_2012_06_23_84517.txt</code> is the file in the source language.
-* <code>hy_2012_06_23_84517.txt</code> is the file in the target language.
+*<code>hy_2012_06_23_84517.txt</code> is the file in the target language.
 You need to loop through all the files that you generated after the sentence segmentation step. You can create a simple Python script to do so:
@@ Line 290: / Line 290: @@
 </pre>
-* Hunalign is under this filepath: <code>~/hun3/hunalign-1.1/src/hunalign/hunalign</code>.
+* Hunalign is under this file path: <code>~/hun3/hunalign-1.1/src/hunalign/hunalign</code>.
-* en is the variable with the filepath to the source language article
+* <code>en</code> is the variable with the file path to the source language article
-* hy is the variable with the filepath to the target language article
+* <code>hy</code> is the variable with the file path to the target language article
-* year is the variable with the year (i.e. "2012")
+* <code>year</code> is the variable with the year (i.e. "2012")
-* lang is the variable with the source language ISO code (i.e. "en")
+* <code>lang</code> is the variable with the source language ISO code (i.e. "en")
-* num is the varaible with the article number (i.e. "1234")
+* <code>num</code> is the variable with the article number (i.e. "1234")
 * The output from Hunalign (for each article) is stored inside a directory with the target language and the source language (separated by a hyphen), which is inside a month directory inside a year directory. Their names are their article numbers. (i.e. align/2012/10/hy-en/12345)

@@ Line 240: / Line 240: @@
 You will have to loop through this process for all files for sentence segmentation.
 ===Sentence Aligning===
-Hunalign will be used to align sentences from two languages. In my case, I ran it on hy-en and hy-ru articles. You can download Hunalign [http://mokk.bme.hu/resources/hunalign/ here].
+Hunalign will be used to align sentences from two languages. For example, hy-en and hy-ru pairs articles can be ran through.
-To build Hunalign on Linux/Unix/Mac OS X, run these commands in the directory where you downloaded Hunalign:
+Hunalign can be downloaded from [http://mokk.bme.hu/resources/hunalign/ here].
+To build Hunalign on Linux/Unix/Mac OS X, run these commands in the directory where Hunalign was downloaded:
 <pre>
 tar zxvf hunalign-1.1.tgz
@@ Line 252: / Line 254: @@
 The dictionary consists of newline-separated dictionary items. An item consists of a target languge phrase and a source language phrase, separated by the ” @ ” sequence. Multiword phrases are allowed. The words of a phrase are space-separated as usual. IMPORTANT NOTE: In the current version, for historical reasons, the target language phrases come first. Therefore the ordering is the opposite of the ordering of the command-line arguments or the results.
 </blockquote>
+(The above quote is from [http://mokk.bme.hu/resources/hunalign/ here].)
 A sample Armenian to English dictionary  based on Apertium's hye-eng bidix (converted to a Hunalign-compatible dictionary format via a simple Python script) can be found [http://sebsauvage.net/paste/?ea47c257cf471482#7RcEqbvdH1NwNHAlPrwUDPAVkgfxrkwbzpQkdPoI0Go= here].
-If you do not have a bilingual dictionary, you can use the -realign argument. Hunalign will build its own dictionary based on the two text files (in two languages) that you input. You can input a 0 byte file as a placeholder for the dictionary argument.
+Information about all Hunalign commands can be found [http://mokk.bme.hu/resources/hunalign/ here].
+If you do not have a bilingual dictionary, you can use the -realign argument.
+Hunalign will build its own dictionary based on the two text files (in two languages) that you input. You can input a 0 byte file as a placeholder for the dictionary argument.
 Hunalign is used like this:

@@ Line 218: / Line 218: @@
 ===Scraping===
+English, Russian, and Armenian articles' content will be scraped from aravot.am.
-Write the Python script with Beautiful Soup, which aids in screen scraping.
+Write the Python script with Beautiful Soup, which aids in screen scraping.
-English, Russian, and Armenian will be scraped from aravot.am. Loop through the archive pages for each day (and month and year) of the given range and for each language to get all the article URLs.
+Loop through the archive pages for each day (and month and year) of the given range and for each language to get all the article URLs.
-Loop through all the article URLs for each day and language and get the HTML tags that store content (e.g. <nowiki><pre>, <p>, etc.</nowiki>) by using Beautiful Soup. Append the contents to a string.
+Loop through all the article URLs for each day and language and get the HTML tags that store article contents (e.g. <nowiki><pre>, <p>, etc.</nowiki>) by using Beautiful Soup. Append the contents to a string.
 ===Sentence Segmentation===
-Some things need to be done with the text before tokenising it. For English, smart quotes should be replaced:
+Some things need to be done with the content string before tokenising it. For English, smart quotes should be replaced. This is an example code snippet:
 <pre>
 output_en=output_en.replace('“','"')
@@ Line 234: / Line 235: @@
-Please refer to the [http://wiki.apertium.org/wiki/Sentence_segmenting sentence segmentation] page for sentence tokenisation. I used NLTK Punkt.
+Please refer to the [http://wiki.apertium.org/wiki/Sentence_segmenting sentence segmentation] page for sentence tokenisation. NLTK Punkt can be used.
-After tokenisation, you can save the article text to a file. The file structure should be: year/month.
+After tokenisation, Article contents can be saved to a file. The file structure can be: year/month (the month folder contains the articles, in all three languages, for that month).
-You will have to loop through this process for all files.
+You will have to loop through this process for all files for sentence segmentation.
 ===Sentence Aligning===
 Hunalign will be used to align sentences from two languages. In my case, I ran it on hy-en and hy-ru articles. You can download Hunalign [http://mokk.bme.hu/resources/hunalign/ here].