Difference between revisions of "Corpora"

From Apertium
Jump to navigation Jump to search
Line 16: Line 16:
* Open-Tran — http://www.open-tran.eu — single point of access to translations of open-source software in many languages (downloadable as SQLite databases)
* Open-Tran — http://www.open-tran.eu — single point of access to translations of open-source software in many languages (downloadable as SQLite databases)
* Tatoeba Project — http://tatoeba.org/ — Database of example sentences translated into several languages.
* Tatoeba Project — http://tatoeba.org/ — Database of example sentences translated into several languages.
* http://translatedby.com/


== Corpus tools ==
== Corpus tools ==

Revision as of 03:24, 25 February 2015

Lists of corpora under free licences (public domain, CC-BY-SA, GPL, etc.).

You might also want to use Wikipedia as a corpus, see Tagger_training#Creating_a_corpus or Building_dictionaries#Wikipedia_dumps and the cleanup script at Calculating_coverage.

Corpora

Use this if you want to do English--<something> (funny alignments for non-English pairs)
Use this if you want to do <anything>--<anything> and there is nothing better available.

Corpus tools

Format

We use a standard format for corpora, defined by either an XSD (XML Schema Definition) or DTD (Document Type Definition). The RFERL scraper generates corpora that follow this format.

In order to use the XSD (recommended), the root <corpus> must have these attributes set:

xmlns="http://apertium.org/xml/corpus/0.9" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://apertium.org/xml/corpus/0.9 http://apertium.org/xml/corpus/0.9/corpus.xsd"

A complete example follows,

<?xml version="1.0"?>
 
<corpus xmlns="http://apertium.org/xml/corpus/0.9" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://apertium.org/xml/corpus/0.9 http://apertium.org/xml/corpus/0.9/corpus.xsd" name="rferl" language="tuk">
  <entry date="2011-12-29" timestamp="2012-04-22T01:56:42.813713" title="Rus hem türkmen saýlawlary: meňzeşlikler we tapawutlar " id="rferl.24437444" source="http://www.azathabar.com/content/article/24437444.html">Prezident saýlawlary Türkmenistanda 12-nji fewralda, Orsýet Federasiýasynda 4-nji martda geçirilýär. Türkmenistanda şu günler prezidentlige kandidatlary hödürlemek kampaniýasy dowam edýär. etc.</entry>
  <entry date="2011-12-19" timestamp="2012-04-22T01:56:45.102281" title="Aşgabadyň ýaşaýjylary Täze ýyla taýýarlanýar" id="rferl.24426930" source="http://www.azathabar.com/content/article/24426930.html">Aşgabat täze 2012-nji ýyly garşylamaga bir aýa golaý wagt öňünden taýýarlyk görüp başlady. Şäheriň köçelerinde, seýilgählerde, edara-kärhanalaryň öňünde gögerip oturan arça agaçlary bilen birlikde, boýy 12-15 metre ýetýän emeli arçalar hem dürli oýnawaçlara bürendiler. etc.</entry>
</corpus>

On the server hosting the XSD, corpus.xsd should be in /xml/corpus/0.9/.

XSD

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" targetNamespace="http://apertium.org/xml/corpus/0.9" xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="corpus">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="entry" maxOccurs="unbounded" minOccurs="0">
          <xs:complexType>
            <xs:simpleContent>
              <xs:extension base="xs:string">
                <xs:attribute type="xs:date" name="date" use="optional"/>
                <xs:attribute type="xs:dateTime" name="timestamp" use="optional"/>
                <xs:attribute type="xs:string" name="title" use="optional"/>
                <xs:attribute type="xs:string" name="author" use="optional"/>
                <xs:attribute type="xs:string" name="id" use="required"/>
                <xs:attribute type="xs:anyURI" name="source" use="required"/>
              </xs:extension>
            </xs:simpleContent>
          </xs:complexType>
        </xs:element>
      </xs:sequence>
      <xs:attribute type="xs:string" name="name" use="required"/>
      <xs:attribute type="xs:string" name="language" use="required"/>
    </xs:complexType>
  </xs:element>
</xs:schema>

DTD

<!ELEMENT corpus ( entry+ ) >
<!ATTLIST corpus language CDATA #REQUIRED >
<!ATTLIST corpus name NMTOKEN #REQUIRED >
<!ATTLIST corpus xmlns CDATA #REQUIRED >
 
<!ELEMENT entry ( #PCDATA ) >
<!ATTLIST entry date CDATA #IMPLIED >
<!ATTLIST entry id ID #REQUIRED >
<!ATTLIST entry source CDATA #REQUIRED >
<!ATTLIST entry timestamp CDATA #IMPLIED >
<!ATTLIST entry title CDATA #IMPLIED >
<!ATTLIST entry author CDATA #IMPLIED >