Difference between revisions of "Writing a scraper"

From Apertium
Jump to navigation Jump to search
(add outline)
(add to outline)
Line 4: Line 4:
   
 
# Visit the website which you plan to scrape and locate the archive section which usually offers an interface to select a given day and see a list of links to articles published on that day.
 
# Visit the website which you plan to scrape and locate the archive section which usually offers an interface to select a given day and see a list of links to articles published on that day.
  +
#* If you can't understand the language the website is written in, ask for help in IRC or use a translator and look for a section marked "Archive". If you're unable to locate an archive, find the sitemap and use it as a starting point.
 
# Familiarize yourself with the structure of the URL and how manipulating it will yield a different set of articles to scrape.
 
# Familiarize yourself with the structure of the URL and how manipulating it will yield a different set of articles to scrape.
  +
#* The URL will sometimes contain a date which can be manipulated to yield all the articles published on a certain day.
# Write a driver script named <code>scrp-*.py</code> which will given a certain range of dates (or other parameters depending on the site's structure), be able to generate, for example, a list of tuples containing the article's link, its title and its publication date. LXML and BeautifulSoup are two useful tools for scraping HTML.
 
  +
#* Other common configurations include having a sequential number which marks pages of articles chronologically. For example, the latest articles have a URL containing "1" and older ones "2", etc.
 
# Write a driver script named <code>scrp-*.py</code> which will given a certain range of dates (or other parameters depending on the site's structure), be able to generate, for example, a list of tuples containing the article's link, its title and its publication date.
  +
#* [http://lxml.de/ LXML] and [http://www.crummy.com/software/BeautifulSoup/ BeautifulSoup] are two useful tools for scraping HTML.
  +
#* Use Chrome/Firefox's Developer Console with Inspect Element to find distinguishing characteristics for each article link element. For example, each article link could be wrapped in a <code>div</code> with <code>.articleLink</code> (it's not always that obvious).
  +
#* Using LXML offers many choices to extracting the article info from the page, from picking specific CSS classes to arbitrary XPATH expressions.
  +
#* If you find that selecting all the article info requires a more complex CSS selector, use the [http://css2xpath.appspot.com/ CSS to XPATH converter].
  +
#* As you populate the article list, writing the list to a file is useful. Outputting it to the console could fail due to character encoding issues. Look in the other <code>scrp-*.py</code> scripts to find a useful helper method.
 
# Add an entry to the <code>feed_sites</code> dictionary in <code>scraper_classes.py</code> which maps from the name of the website to a unique Scraper class.
 
# Add an entry to the <code>feed_sites</code> dictionary in <code>scraper_classes.py</code> which maps from the name of the website to a unique Scraper class.
# Define a new class in <code>scrapers.py</code> that inherits the Scraper class with two functions: <code>url_to_aid()</code> and <code>scraped()</code>. Details are described below in the <code>scrapers.py</code> section. Again, LXML/BeautifulSoup will be very useful for scraping the article content.
+
# Define a new class in <code>scrapers.py</code> that inherits the Scraper class with two functions: <code>url_to_aid()</code> and <code>scraped()</code> with very important specifications.
  +
## <code>url_to_aid()</code>: This function will take as an input and convert it to a unique "article id" (aid).
# Finally, in the driver script loop through the list of articles and send each article to the Scraper class you created to fill the corpus with articles. Have a look at the various <code>scrp-*.py</code> scripts currently available to get a feel for how to use the Scraper class.
 
  +
##* Many sites will use a unique ID inside their article URLs (e.g., http://example.com/news?id=3141592653 or http://example.com/news/3141592653.html), these are fairly simple to extract using a regex or string splitting.
 
  +
##* However, if this is for some reason not unique, or the site doesn't use unique ids, or if it's difficult to extract for some reason, it's okay to make a hash of the full url (which should be unique...).
== scrapers.py ==
 
  +
##* There are examples of both of these methods implemented in other scrapers in <code>scrapers.py</code>. Take a look if you get stuck.
You need to define a new class in scrapers.py that inherits the Scraper class.
 
  +
## <code>scraped()</code>: This function will take as "input" the HTML contents of the article and output a cleaned version of the article's text.
 
 
##* First, fill <code>self.doc</code> with the contents of the page, by calling <code>self.get_content()</code>. This is all written for you already, so just call the function once and you're ready for the hard stuff.
Your new class will have two new functions:
 
  +
##* Now, LXML/BeautifulSoup will be very useful for scraping the actual article content from the HTML of the entire page.
 
  +
##* Most likely, the article text will be wrapped in some sort of an identifiable container, so follow a similar procedure to that which proved useful when populating the list of articles, and identify this element.
* <code>url_to_aid()</code>:
 
  +
##* Take the element which contains the article content, extract it from the HTML, and then clean it with LXML (to remove scripts, etc. which shouldn't be in the corpus).
:: This takes a url and converts it to a unique "article id". For sites that use some form of unique id for their articles (e.g., http://example.com/news?id=3141592653 or http://example.com/news/3141592653.html), you'll want to extract the id, probably with a simple regex. However, if this is for some reason not unique, or the site doesn't use unique ids, or if it's difficult to extract for some reason, it's okay to make a hash of the full url (which should be unique...). There are examples of both of these implemented in other scrapers in scrapers.py
 
  +
##* The cleaning procedure below often suffices to remove all the HTML tags, changing break tags and paragraph tags into line breaks as necessary.
 
* <code>scraped()</code>:
+
<code>
:: The first thing this function does is to fill <code>self.doc</code> with the contents of the page, by calling <code>self.get_content()</code>. This is all written for you already, so just call the function once and you're ready for the hard stuff.
 
:: The hard stuff consists of getting a cleaned, text-only version of just the article content from the page. You'll have to first make sure you know which element in the page is going to consistently contain just the article content, and then extract that out with lxml. You'll then want to take that element and clean it with lxml (since there are scripts and stuff that can be in there too that could get in the output), and then get the <code>.text_content()</code> of the element. An example of all this follows:
 
<pre>
 
 
self.get_content()
 
self.get_content()
 
cleaned = lxml.html.document_fromstring(lxml.html.clean.clean_html(lxml.html.tostring(self.doc.xpath("//div[@align='justify']")[0]).decode('utf-8')))
 
cleaned = lxml.html.document_fromstring(lxml.html.clean.clean_html(lxml.html.tostring(self.doc.xpath("//div[@align='justify']")[0]).decode('utf-8')))
 
cleaned = cleaned.text_content()
 
cleaned = cleaned.text_content()
 
return cleaned.strip()
 
return cleaned.strip()
</pre>
+
</code>
  +
##* Sometimes, this won't suffice and you'll have to be able to identify the offending elements and remove them manually from the HTML before invoking LXML's <code>clean</code>.
 
# Finally, in the driver script loop through the list of articles and send each article to the Scraper class you created to fill the corpus with articles. Have a look at the various <code>scrp-*.py</code> scripts currently available to get a feel for how to use the <code>Scraper</code> class.
  +
#* Make sure to set the correct language code when setting up the <code>Source</code> class.
   
 
=== RFERL ===
 
=== RFERL ===

Revision as of 00:28, 20 November 2013

This page outlines how to develop a scraper for apertium using our RFERL scraper. The code can be found in our subversion repository at https://svn.code.sf.net/p/apertium/svn/trunk/apertium-tools/scraper.

Outline

  1. Visit the website which you plan to scrape and locate the archive section which usually offers an interface to select a given day and see a list of links to articles published on that day.
    • If you can't understand the language the website is written in, ask for help in IRC or use a translator and look for a section marked "Archive". If you're unable to locate an archive, find the sitemap and use it as a starting point.
  2. Familiarize yourself with the structure of the URL and how manipulating it will yield a different set of articles to scrape.
    • The URL will sometimes contain a date which can be manipulated to yield all the articles published on a certain day.
    • Other common configurations include having a sequential number which marks pages of articles chronologically. For example, the latest articles have a URL containing "1" and older ones "2", etc.
  3. Write a driver script named scrp-*.py which will given a certain range of dates (or other parameters depending on the site's structure), be able to generate, for example, a list of tuples containing the article's link, its title and its publication date.
    • LXML and BeautifulSoup are two useful tools for scraping HTML.
    • Use Chrome/Firefox's Developer Console with Inspect Element to find distinguishing characteristics for each article link element. For example, each article link could be wrapped in a div with .articleLink (it's not always that obvious).
    • Using LXML offers many choices to extracting the article info from the page, from picking specific CSS classes to arbitrary XPATH expressions.
    • If you find that selecting all the article info requires a more complex CSS selector, use the CSS to XPATH converter.
    • As you populate the article list, writing the list to a file is useful. Outputting it to the console could fail due to character encoding issues. Look in the other scrp-*.py scripts to find a useful helper method.
  4. Add an entry to the feed_sites dictionary in scraper_classes.py which maps from the name of the website to a unique Scraper class.
  5. Define a new class in scrapers.py that inherits the Scraper class with two functions: url_to_aid() and scraped() with very important specifications.
    1. url_to_aid(): This function will take as an input and convert it to a unique "article id" (aid).
      • Many sites will use a unique ID inside their article URLs (e.g., http://example.com/news?id=3141592653 or http://example.com/news/3141592653.html), these are fairly simple to extract using a regex or string splitting.
      • However, if this is for some reason not unique, or the site doesn't use unique ids, or if it's difficult to extract for some reason, it's okay to make a hash of the full url (which should be unique...).
      • There are examples of both of these methods implemented in other scrapers in scrapers.py. Take a look if you get stuck.
    2. scraped(): This function will take as "input" the HTML contents of the article and output a cleaned version of the article's text.
      • First, fill self.doc with the contents of the page, by calling self.get_content(). This is all written for you already, so just call the function once and you're ready for the hard stuff.
      • Now, LXML/BeautifulSoup will be very useful for scraping the actual article content from the HTML of the entire page.
      • Most likely, the article text will be wrapped in some sort of an identifiable container, so follow a similar procedure to that which proved useful when populating the list of articles, and identify this element.
      • Take the element which contains the article content, extract it from the HTML, and then clean it with LXML (to remove scripts, etc. which shouldn't be in the corpus).
      • The cleaning procedure below often suffices to remove all the HTML tags, changing break tags and paragraph tags into line breaks as necessary.

       self.get_content()
       cleaned = lxml.html.document_fromstring(lxml.html.clean.clean_html(lxml.html.tostring(self.doc.xpath("//div[@align='justify']")[0]).decode('utf-8')))
       cleaned = cleaned.text_content()
       return cleaned.strip()

      • Sometimes, this won't suffice and you'll have to be able to identify the offending elements and remove them manually from the HTML before invoking LXML's clean.
  1. Finally, in the driver script loop through the list of articles and send each article to the Scraper class you created to fill the corpus with articles. Have a look at the various scrp-*.py scripts currently available to get a feel for how to use the Scraper class.
    • Make sure to set the correct language code when setting up the Source class.

RFERL

If you are scraping RFERL content, you will need category names and numbers of only the real content categories.

Issues with newlines

Problem: The characters "& # 1 3 ;" (spaced apart intentionally) appear throughout after scraped content is written to .xml file.
Research: Retrieving the page html through using either curl or wget results in the problematic characters not appearing in final .xml output, however they reappear when the html is downloaded through a Python HTTPConnection. Since furthermore the characters are not present in other preceding output of the page html, it can be intelligently assumed that the error occurs with lxml: lxml.html.document_fromstring(lxml.html.clean.clean_html(lxml.html.tostring(doc.find_class('zoomMe')[1]).decode('utf-8'))). Directly following this step, the characters appear in the xml output. However, that still leaves uncertain the discrepancy between manually downloaded code and python downloaded code. This difference is likely due to curl and wget treating the code differently than python does. This can be painlessly confirmed with a diff command which confirms that most (i.e. 95%) of the discrepancies are whitespace. The characters represent "\r", the carriage return. Online research shows that these problems can be attributed to Windows being incompatible with Linux\Unix standards: "When you code in windows, and use "DOS/Windows" line endings, the your lines will end like this "\r\n". In some xhtml editors, that "\r" is illegal so the editor coverts it to "& # 1 3"." Accordingly, running scrp-azzatyk.py shows that the offending characters unilaterally appear following the end of lines in the HTML.
Suggested Solution: The simplest solution is to manually remove the "\r" from raw html after download, like so: res.read().decode('utf-8').replace('\r',' '). This should have no side effects for two reasons. One, HTML generally ignores conventional whitespace. Two, each "\r" is likely followed by a "\n", so replacing "\r" with nothing will only remove extraneous characters while otherwise preserving whitespace. This will solve the problem because the problematic characters represent "\r". This type of a solution to this seemingly not uncommon problem has been utilized by others and will ensure compatibility with Windows style "\r\n".This "solution" has been implemented.

Problem: The character "x" appears throughout after scraped content is written to .xml file.
Research & Solution: The problem was a small error due to not filtering out a bad class in ScraperAzattyk, the problem has been fixed and will be committed. This solution has been committed.

Problem: Paragraphs are not always being created correctly in scraped content, i.e. breaks tags are occasionally ignored
Research: Testing shows that the problem is occurring when two break tags are present on two separate lines and they are directly followed by another tag, generally an em or a strong, however the same problem has been observed with other tags. In the case that the break tags are seperated by text, lxml properly handles them. However, in the case that they are not, lxml fails to properly recognize the break tags. Test script, Test HTML
Suggested Solution: Submit a bug report to lxml. We could create custom Element classes? I'm fairly sure that even if we managed to do that, it would be fairly inelegant. A bug report has been filed.