Corpora - Revision history

ScoopGracie: /* Corpora */ Broken link... :(

2020-01-25T20:36:08Z

Corpora: Broken link... :(

Memduh at 10:27, 1 September 2017

2017-09-01T10:27:38Z

Unhammer at 13:26, 30 January 2016

2016-01-30T13:26:12Z

Ilnar.salimzyan at 18:46, 11 July 2015

2015-07-11T18:46:12Z

Ilnar.salimzyan: /* Corpora */

2015-02-25T03:24:28Z

Corpora

Sushain: /* DTD */

2015-01-11T00:05:43Z

DTD

Sushain: /* XSD */

2015-01-11T00:05:19Z

XSD

Sushain: /* Format */

2015-01-09T23:55:08Z

Format

Sushain at 07:40, 9 January 2015

2015-01-09T07:40:27Z

Sushain at 07:31, 9 January 2015

2015-01-09T07:31:24Z

@@ Line 8: / Line 8: @@
 * EuroParl &mdash; http://www.statmt.org/europarl/ &mdash; EU12 languages up to 44 million words per language
 ::Use this if you want to do English--<something> (funny alignments for non-English pairs)
-* JRC-Acquis &mdash; http://langtech.jrc.it/JRC-Acquis.html &mdash; EU22 languages
-::Use this if you want to do <anything>--<anything> and there is ''nothing better available''.
 * Southeast European Times &mdash; http://opus.lingfil.uu.se/SETIMES.php &mdash; English,Turkish,Bulgarian,Macedonian,Serbo-Croatian,Albanian,Greek,Romanian &mdash; 9,000 approx. paragraph aligned, 90,000&mdash;120,000 words.
 * South African Government Services &mdash; http://xixona.dlsi.ua.es/~fran/services-gov-za-en_ZA-af_ZA.txt &mdash; English&mdash;Afrikaans &mdash; 2,500 approx. sentence aligned, 49,375 words.

@@ Line 10: / Line 10: @@
 * JRC-Acquis &mdash; http://langtech.jrc.it/JRC-Acquis.html &mdash; EU22 languages
 ::Use this if you want to do <anything>--<anything> and there is ''nothing better available''.
-* Southeast European Times &mdash; http://xixona.dlsi.ua.es/~fran/setimes/ &mdash; English,Turkish,Bulgarian,Macedonian,Serbo-Croatian,Albanian,Greek,Romanian &mdash; 9,000 approx. paragraph aligned, 90,000&mdash;120,000 words.
+* Southeast European Times &mdash; http://opus.lingfil.uu.se/SETIMES.php &mdash; English,Turkish,Bulgarian,Macedonian,Serbo-Croatian,Albanian,Greek,Romanian &mdash; 9,000 approx. paragraph aligned, 90,000&mdash;120,000 words.
 * South African Government Services &mdash; http://xixona.dlsi.ua.es/~fran/services-gov-za-en_ZA-af_ZA.txt &mdash; English&mdash;Afrikaans &mdash; 2,500 approx. sentence aligned, 49,375 words.
 * IJS-ELAN &mdash; http://nl.ijs.si/elan/ &mdash; English-Slovenian

@@ Line 2: / Line 2: @@
 Lists of corpora under free licences (public domain, CC-BY-SA, GPL, etc.).
-You might also want to use Wikipedia as a corpus, see [[Tagger_training#Creating_a_corpus]] or [[Building_dictionaries#Wikipedia_dumps]] and the cleanup script at [[Calculating_coverage]].
+You might also want to use Wikipedia as a corpus, see [[Tagger_training#Creating_a_corpus]] or [[Building_dictionaries#Wikipedia_dumps]] and the cleanup scripts at [[Wikipedia dumps]].
 ==Corpora==

@@ Line 16: / Line 16: @@
 * Open-Tran &mdash; http://www.open-tran.eu &mdash; single point of access to translations of open-source software in many languages (downloadable as SQLite databases)
 * Tatoeba Project &mdash; http://tatoeba.org/ &mdash; Database of example sentences translated into several languages.
-* http://translatedby.com/
+* Translated.by &mdash; http://translatedby.com/ (Various licenses)
+* Heidelberg Named Entity Resource &mdash; http://heiner.cl.uni-heidelberg.de
 == Corpus tools ==

@@ Line 16: / Line 16: @@
 * Open-Tran &mdash; http://www.open-tran.eu &mdash; single point of access to translations of open-source software in many languages (downloadable as SQLite databases)
 * Tatoeba Project &mdash; http://tatoeba.org/ &mdash; Database of example sentences translated into several languages.
+* http://translatedby.com/
 == Corpus tools ==

@@ Line 93: / Line 93: @@
 <!ATTLIST entry timestamp CDATA #IMPLIED >
 <!ATTLIST entry title CDATA #IMPLIED >
+<!ATTLIST entry author CDATA #IMPLIED >
 </pre>
 </code>

@@ Line 63: / Line 63: @@
                 <xs:attribute type="xs:dateTime" name="timestamp" use="optional"/>
                 <xs:attribute type="xs:string" name="title" use="optional"/>
+                <xs:attribute type="xs:string" name="author" use="optional"/>
                 <xs:attribute type="xs:string" name="id" use="required"/>
                 <xs:attribute type="xs:anyURI" name="source" use="required"/>

@@ Line 45: / Line 45: @@
 </pre>
 </code>
+On the server hosting the XSD, <code>corpus.xsd</code> should be in <code>/xml/corpus/0.9/</code>.
 === XSD ===

@@ Line 25: / Line 25: @@
 == Format ==
-We use a standard format for corpora, defined by either an XSD (XML Schema Definition) or DTD (Document Type Definition).
+We use a standard format for corpora, defined by either an XSD (XML Schema Definition) or DTD (Document Type Definition). The [[Writing a scraper|RFERL scraper]] generates corpora that follow this format.
 In order to use the XSD (recommended), the root <code><corpus></code> must have these attributes set:
+<code>
-<code>xmlns="http://apertium.org/xml/corpus/0.9" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://apertium.org/xml/corpus/0.9 http://apertium.org/xml/corpus/0.9/corpus.xsd"</code>.
+xmlns="http://apertium.org/xml/corpus/0.9" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://apertium.org/xml/corpus/0.9 http://apertium.org/xml/corpus/0.9/corpus.xsd"
+</code>
+A complete example follows,
+<code>
+<pre>
+<?xml version="1.0"?>
+<corpus xmlns="http://apertium.org/xml/corpus/0.9" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://apertium.org/xml/corpus/0.9 http://apertium.org/xml/corpus/0.9/corpus.xsd" name="rferl" language="tuk">
+  <entry date="2011-12-29" timestamp="2012-04-22T01:56:42.813713" title="Rus hem türkmen saýlawlary: meňzeşlikler we tapawutlar " id="rferl.24437444" source="http://www.azathabar.com/content/article/24437444.html">Prezident saýlawlary Türkmenistanda 12-nji fewralda, Orsýet Federasiýasynda 4-nji martda geçirilýär. Türkmenistanda şu günler prezidentlige kandidatlary hödürlemek kampaniýasy dowam edýär. etc.</entry>
+  <entry date="2011-12-19" timestamp="2012-04-22T01:56:45.102281" title="Aşgabadyň ýaşaýjylary Täze ýyla taýýarlanýar" id="rferl.24426930" source="http://www.azathabar.com/content/article/24426930.html">Aşgabat täze 2012-nji ýyly garşylamaga bir aýa golaý wagt öňünden taýýarlyk görüp başlady. Şäheriň köçelerinde, seýilgählerde, edara-kärhanalaryň öňünde gögerip oturan arça agaçlary bilen birlikde, boýy 12-15 metre ýetýän emeli arçalar hem dürli oýnawaçlara bürendiler. etc.</entry>
+</corpus>
+</pre>
+</code>
 === XSD ===

@@ Line 1: / Line 1: @@
+{{TOCD}}
 Lists of corpora under free licences (public domain, CC-BY-SA, GPL, etc.).
@@ Line 21: / Line 22: @@
 * BootCaT &mdash; http://sslmit.unibo.it/~baroni/bootcat.html - Simple Utilities to Bootstrap Corpora and Terms from the Web
 * Bitextor &mdash; http://sourceforge.net/projects/bitextor/ - Bootstrap bilingual corpora from the web
+== Format ==
+We use a standard format for corpora, defined by either an XSD (XML Schema Definition) or DTD (Document Type Definition).
+In order to use the XSD (recommended), the root <code><corpus></code> must have these attributes set:
+<code>xmlns="http://apertium.org/xml/corpus/0.9" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://apertium.org/xml/corpus/0.9 http://apertium.org/xml/corpus/0.9/corpus.xsd"</code>.
+=== XSD ===
+<code>
+<pre>
+<?xml version="1.0" encoding="UTF-8"?>
+<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" targetNamespace="http://apertium.org/xml/corpus/0.9" xmlns:xs="http://www.w3.org/2001/XMLSchema">
+  <xs:element name="corpus">
+    <xs:complexType>
+      <xs:sequence>
+        <xs:element name="entry" maxOccurs="unbounded" minOccurs="0">
+          <xs:complexType>
+            <xs:simpleContent>
+              <xs:extension base="xs:string">
+                <xs:attribute type="xs:date" name="date" use="optional"/>
+                <xs:attribute type="xs:dateTime" name="timestamp" use="optional"/>
+                <xs:attribute type="xs:string" name="title" use="optional"/>
+                <xs:attribute type="xs:string" name="id" use="required"/>
+                <xs:attribute type="xs:anyURI" name="source" use="required"/>
+              </xs:extension>
+            </xs:simpleContent>
+          </xs:complexType>
+        </xs:element>
+      </xs:sequence>
+      <xs:attribute type="xs:string" name="name" use="required"/>
+      <xs:attribute type="xs:string" name="language" use="required"/>
+    </xs:complexType>
+  </xs:element>
+</xs:schema>
+</pre>
+</code>
+=== DTD ===
+<code>
+<pre>
+<!ELEMENT corpus ( entry+ ) >
+<!ATTLIST corpus language CDATA #REQUIRED >
+<!ATTLIST corpus name NMTOKEN #REQUIRED >
+<!ATTLIST corpus xmlns CDATA #REQUIRED >
+<!ELEMENT entry ( #PCDATA ) >
+<!ATTLIST entry date CDATA #IMPLIED >
+<!ATTLIST entry id ID #REQUIRED >
+<!ATTLIST entry source CDATA #REQUIRED >
+<!ATTLIST entry timestamp CDATA #IMPLIED >
+<!ATTLIST entry title CDATA #IMPLIED >
+</pre>
+</code>
 [[Category:Resources]]