Using linguistic resources - Revision history

Unhammer: /* Wiktionary */

2023-05-26T12:11:14Z

Wiktionary

Bech: Link to French page

2014-10-06T12:04:34Z

Link to French page

Wei2912: /* Wiktionary */ Add my user talk page for people to contact me regarding crawling of Wikrionary

2014-04-26T17:36:16Z

Wiktionary: Add my user talk page for people to contact me regarding crawling of Wikrionary

Bech: /* Wiktionary */ discussion link

2011-10-03T19:31:14Z

Wiktionary: discussion link

Francis Tyers: /* Some final notes */

2011-09-03T10:24:55Z

Some final notes

Unhammer: /* Wiktionary */

2010-07-26T07:50:02Z

Wiktionary

Unhammer: /* Collecting linguistic data */

2010-07-26T07:35:57Z

Collecting linguistic data

Francis Tyers: /* Ways of storing data */

2008-02-27T23:41:36Z

Ways of storing data

Francis Tyers: /* Ways of storing data */

2007-12-11T10:34:48Z

Ways of storing data

Francis Tyers: /* Collecting linguistic data */ typo

2007-11-10T17:43:40Z

Collecting linguistic data: typo

@@ Line 34: / Line 34: @@
 For Wiktionary, contact [[User_talk:Francis Tyers|Francis Tyers]] who has some pre-rolled scripts for retrieving morphological information and [[User_talk:wei2912|Wei En]] who has written a crawler for obtaining such information.
+See also [[Task_ideas_for_Google_Code-in/Scrape_inflection_information_from_Wiktionary#Resources]]
 ===More sources of linguistic data===

← Older revision		Revision as of 12:04, 6 October 2014
Line 1:		Line 1:
			[[Utilisation de ressources linguistiques\|En français]]

	{{TOCD}}		{{TOCD}}

@@ Line 31: / Line 31: @@
 For some languages, the free dictionary project, [http://en.wiktionary.org Wiktionary] has reasonable data. For example, for Faroese, nouns come with declination, which can be automatically extracted by means of scripts. Some of this can be quite complicated, but if there is a site which has this information freely available in a standardised format, contact someone on the Apertium team, and they'd be glad to help you write a script, or to write the script for you.
-For Wiktionary, contact [[User_talk:Francis Tyers|Francis Tyers]] who has some pre-rolled scripts for retrieving morphological information.
+For Wiktionary, contact [[User_talk:Francis Tyers|Francis Tyers]] who has some pre-rolled scripts for retrieving morphological information and [[User_talk:wei2912|Wei En]] who has written a crawler for obtaining such information.
 ===More sources of linguistic data===

@@ Line 31: / Line 31: @@
 For some languages, the free dictionary project, [http://en.wiktionary.org Wiktionary] has reasonable data. For example, for Faroese, nouns come with declination, which can be automatically extracted by means of scripts. Some of this can be quite complicated, but if there is a site which has this information freely available in a standardised format, contact someone on the Apertium team, and they'd be glad to help you write a script, or to write the script for you.
-For Wiktionary, contact [[User:Francis Tyers]] who has some pre-rolled scripts for retrieving morphological information.
+For Wiktionary, contact [[User_talk:Francis Tyers|Francis Tyers]] who has some pre-rolled scripts for retrieving morphological information.
 ===More sources of linguistic data===

← Older revision		Revision as of 10:24, 3 September 2011
Line 118:		Line 118:
	Without language data, it is impossible to build language tools. So by putting together your datastore, you have already taken an enormous step towards making the riches of your language available to others.		Without language data, it is impossible to build language tools. So by putting together your datastore, you have already taken an enormous step towards making the riches of your language available to others.

	[[Category:Documentation]]		[[Category:Documentation in English]]

@@ Line 32: / Line 32: @@
 For Wiktionary, contact [[User:Francis Tyers]] who has some pre-rolled scripts for retrieving morphological information.
+===More sources of linguistic data===
+The Wiki of the Association for Computational Linguistics has a [http://www.aclweb.org/aclwiki/index.php?title=List_of_resources_by_language List of resources by language] which often contains useful links (often clearly marked for whether the resources are Free Software or Proprietary).
 ==Ways of storing data==

@@ Line 15: / Line 15: @@
 ==Collecting linguistic data==
-Before these files can be produced, you need a collection of linguistic data which can be inserted into them.  This data might consist of wordlists, word-corpora derived from web-crawlers such as Crubadán, grammar notes, existing translation of open-source software such as KDE or GNOME, etc.  Some practical suggestions on how to build some starter wordlists can be found at [[Building dictionaries]], but if you feel that this is too technical, just ask one of the Apertium team to put together something like this for you.
+Before these files can be produced, you need a collection of linguistic data which can be inserted into them.  This data might consist of wordlists, word-corpora derived from web-crawlers such as [http://borel.slu.edu/crubadan/ Crubadán], grammar notes, existing translation of open-source software such as KDE or GNOME, etc.  Some practical suggestions on how to build some starter wordlists can be found at [[Building dictionaries]], but if you feel that this is too technical, just ask one of the Apertium team to put together something like this for you.
 A crucial point here is that the data must either have been gathered from scratch, or must be available under a license which is compatible with Apertium's [http://www.fsf.org/licensing/licenses/gpl.html GPL].  In other words, you ''definitely'' cannot just start copying published dictionaries or other material wholesale into your data store.

@@ Line 34: / Line 34: @@
 ==Ways of storing data==
+{{see-also|Speling format}}
 It is easiest to start storing the words in a spreadsheet or database. Gnumeric, KSpread or OOCalc are examples of the former.  Once complete, your data can be exported into a format (e.g. CSV, comma-separated values) where it can be used by other software to build the Apertium dictionaries. Databases such as PostgreSQL, MySQL and SQLite are even more attractive, provided you are familiar with them, since the data can be manipulated in various ways before exporting.  Further information on the software mentioned here is [at this other page].
@@ Line 68: / Line 68: @@
 The "part of speech" entry will allow you to combine various wordlists whenever necessary without losing information about the contents - you will be able to separate them again.  Typical parts of speech in European languages might be: noun, proper noun, adjective, verb, adverb, preposition, pronoun, conjunction, interjection, interrogative, demonstrative, numeral.
-For example, a spreadsheet might look like:
+For example, a spreadsheet (which closely mirrors the [[Speling format]]) might look like:
 <pre>
@@ Line 75: / Line 75: @@
 beer  ; beers        ; noun           ; pl
 red   ; red          ; adj            ;
-goes  ; go           ; verb           ; pres.p3.sg
+go    ; goes         ; verb           ; pres.p3.sg
 go    ; go           ; verb           ; pres.p2.sg
 go    ; go           ; verb           ; pres.p1.sg
-went  ; go           ; verb           ; past
+go    ; went         ; verb           ; past
 </pre>