Writing a scraper - Revision history

Firespeaker at 05:20, 29 March 2019

2019-03-29T05:20:20Z

Rcrowther at 16:57, 26 September 2016

2016-09-26T16:57:39Z

Putti: /* Use Scraper class and test */

2014-12-28T22:43:40Z

Use Scraper class and test

Putti: /* Use Scraper class and test */

2014-12-26T21:16:48Z

Use Scraper class and test

Putti: /* Use Scraper class and test */

2014-12-26T21:16:08Z

Use Scraper class and test

Putti: /* Use Scraper class and test */

2014-12-26T21:15:34Z

Use Scraper class and test

Putti: /* Use Scraper class and test */

2014-12-08T10:09:05Z

Use Scraper class and test

Sushain at 21:53, 20 November 2013

2013-11-20T21:53:01Z

Sushain: /* RFERL */

2013-11-20T18:18:18Z

RFERL

Sushain: /* RFERL */

2013-11-20T18:12:05Z

RFERL

← Older revision		Revision as of 05:20, 29 March 2019
Line 1:		Line 1:
	This page outlines how to develop a scraper for ~~apertium~~ using our ~~RFERL~~ scraper. The code can be found in ~~our subversion repository~~ at [https://~~svn~~.~~code.sf.net/p~~/apertium~~/svn/trunk~~/apertium-~~tools/scraper~~ https://~~svn~~.~~code.sf.net/p~~/apertium~~/svn/trunk~~/apertium-~~tools/scraper~~].		This page outlines how to develop a scraper of news websites for Apertium using our '''news scraper'''. The code can be found on GitHub at [https://github.com/apertium/apertium-news-scrapers https://github.com/apertium/apertium-news-scrapers].

	== How-To ==		== How-To ==

@@ Line 137: / Line 137: @@
 '''Research:''' Testing shows that the problem is occurring when two break tags are present on two separate lines and they are directly followed by another tag, generally an <code>em</code> or a <code>strong</code>, however the same problem has been observed with other tags. In the case that the break tags are seperated by text, lxml properly handles them. However, in the case that they are not, lxml fails to properly recognize the break tags. [https://pastee.org/ev5dc Test script], [https://pastee.org/pp6h4 Test HTML] <br />
 '''Suggested Solution:''' Submit a bug report to lxml. We could create [http://lxml.de/element_classes.html custom Element classes]? I'm fairly sure that even if we managed to do that, it would be fairly inelegant. A [https://bugs.launchpad.net/lxml/+bug/1095945 bug report] has been filed. Turns out that the bug was in libxml2 rather than lxml and was addressed in a newer version of libxml2 (check the bug report)<br />
+[[Category:Documentation in English]]

@@ Line 87: / Line 87: @@
 <code><pre>
 w = Writer()
+# Create a SIGTERM signal handler in order to save all the article data if kill command is called.
+def term_handler(sigNum, frame):
+	w.close()
+	sys.exit(0)
+signal.signal(signal.SIGTERM, term_handler)
 try:
 	for (title, url, date) in articles:

@@ Line 83: / Line 83: @@
 <li>Catch exceptions that occur during scraping but don't fail silently. You don't want a single badly formatted article to stop the entire process.</li>
 <li>Remeber to call <code>Writer</code> object's <code>close()</code> function before closing the scraper. It will make sure all the data has been saved correctly.</li>
-<li>Put the main code inside a <code>try</code> structure and <code>expect</code> a <code>KeyboardInterrup</code>, in order to save all scraped data when user does a keyboard interrupt (^C).</li>
+<li>Put the main code inside a <code>try</code> structure and <code>expect</code> a <code>KeyboardInterrupt</code>, in order to save all the scraped data when user does a keyboard interrupt (^C).</li>
 </ul>
 <code><pre>

@@ Line 83: / Line 83: @@
 <li>Catch exceptions that occur during scraping but don't fail silently. You don't want a single badly formatted article to stop the entire process.</li>
 <li>Remeber to call <code>Writer</code> object's <code>close()</code> function before closing the scraper. It will make sure all the data has been saved correctly.</li>
-<li>Put the main code in side a <code>try</code> structure and <code>expect</code> a <code>KeyboardInterrup</code>, in order to save all scraped data when user does a keyboard interrupt (^C).</li>
+<li>Put the main code inside a <code>try</code> structure and <code>expect</code> a <code>KeyboardInterrup</code>, in order to save all scraped data when user does a keyboard interrupt (^C).</li>
 </ul>
 <code><pre>

@@ Line 83: / Line 83: @@
 <li>Catch exceptions that occur during scraping but don't fail silently. You don't want a single badly formatted article to stop the entire process.</li>
 <li>Remeber to call <code>Writer</code> object's <code>close()</code> function before closing the scraper. It will make sure all the data has been saved correctly.</li>
+<li>Put the main code in side a <code>try</code> structure and <code>expect</code> a <code>KeyboardInterrup</code>, in order to save all scraped data when user does a keyboard interrupt (^C).</li>
 </ul>
 <code><pre>
 w = Writer()
+try:
-for (title, url, date) in articles:
+	for (title, url, date) in articles:
-	try:
+		try:
-		source = Source(url, title=title, date=date, scraper=ScraperAzadliq, conn=conn) #replace scraper with the one you created earlier
-		source.makeRoot("./", ids=ids, root=root, lang="aze") #replace language with the appropriate one
+			source = Source(url, title=title, date=date, scraper=ScraperAzadliq, conn=conn) #replace scraper with the one you created earlier
+			source.makeRoot("./", ids=ids, root=root, lang="aze") #replace language with the appropriate one
-		source.add_to_archive()
+			source.add_to_archive()
-		if ids is None:
-			ids = source.ids
+			if ids is None:
+				ids = source.ids
-		if root is None:
-			root = source.root
+			if root is None:
+				root = source.root
-	except Exception as e:
+		except Exception as e:
-		print(url + " " + str(e))
+			print(url + " " + str(e))
+except KeyboardInterrupt:
+	print("\nReceived a keyboard interrupt. Closing the program.")
 w.close()
 </pre></code>

@@ Line 1: / Line 1: @@
 This page outlines how to develop a scraper for apertium using our RFERL scraper.  The code can be found in our subversion repository at [https://svn.code.sf.net/p/apertium/svn/trunk/apertium-tools/scraper https://svn.code.sf.net/p/apertium/svn/trunk/apertium-tools/scraper].
-== Outline ==
+== How-To ==
 ===Get to know the website===

@@ Line 111: / Line 111: @@
 * "20131120" indicates that the publication date of the articles displayed will be "11/20/2013". Varying this part of the URL in your scraping program will allow you to scrape a given date range.
 * "330" is the identifier this website uses for the "news" category. In this case it's duplicated; however, this might not always be the case.
-* "25174069" is
 == Issues with newlines ==