Language identification - Revision history

Unhammer: /* See also */

2016-03-01T20:57:43Z

Unhammer: /* See also */

2016-03-01T16:54:26Z

Unhammer: /* See also */

2015-03-09T11:22:48Z

Unhammer at 09:00, 13 February 2015

2015-02-13T09:00:03Z

Unhammer: Created page with "'''Language identification''' or '''language recognition''' is the process of identifying what language a text (document/paragraph/sentence/word/…) is in. Apertium-apy uses..."

2015-02-13T08:55:44Z

Created page with "'''Language identification''' or '''language recognition''' is the process of identifying what language a text (document/paragraph/sentence/word/…) is in. Apertium-apy uses..."

New page

'''Language identification''' or '''language recognition''' is the process of identifying what language a text (document/paragraph/sentence/word/…) is in.

Apertium-apy uses the CLD2 library for language identification (optionally it can use coverage of analysers, but this is really slow)

See [[Apertium-apy/Language identification]] for some accuracy experiments of CLD2

==See also==
* http://odur.let.rug.nl/~vannoord/TextCat/ the original TextCat library (perl, there's also a C port)
** https://github.com/unhammer/gt-CorpusTools/blob/master/corpustools/text_cat.py python2 reimplementation of textcat
* http://www.ling.upenn.edu/Events/PLC/plc37/abstracts/Szymanski_PLC37_CT.pdf for identifying parallel text in another language within the same document.

@@ Line 11: / Line 11: @@
 * http://www.ling.upenn.edu/Events/PLC/plc37/abstracts/Szymanski_PLC37_CT.pdf for identifying parallel text in another language within the same document.
 * https://github.com/saffsd/langid.py Naïve Bayes method implemented in Python, comes with lots of pre-trained models
-* https://www.mediawiki.org/wiki/User:TJones_%28WMF%29/Notes/Balanced_Language_Identification_Evaluation_Set_for_Queries
+* https://www.mediawiki.org/wiki/User:TJones_%28WMF%29/Notes/Balanced_Language_Identification_Evaluation_Set_for_Queries / https://www.mediawiki.org/wiki/User:TJones_%28WMF%29/Notes/Language_Detection_with_TextCat#Query_Data_Collection – modifies the textcat method for working on very short strings

@@ Line 11: / Line 11: @@
 * http://www.ling.upenn.edu/Events/PLC/plc37/abstracts/Szymanski_PLC37_CT.pdf for identifying parallel text in another language within the same document.
 * https://github.com/saffsd/langid.py Naïve Bayes method implemented in Python, comes with lots of pre-trained models
+* https://www.mediawiki.org/wiki/User:TJones_%28WMF%29/Notes/Balanced_Language_Identification_Evaluation_Set_for_Queries

@@ Line 1: / Line 1: @@
 '''Language identification''' or '''language recognition''' is the process of identifying what language a text (document/paragraph/sentence/word/…) is in.
-Apertium-apy uses the CLD2 library for language identification (optionally it can use coverage of analysers, but this is really slow)
+Apertium-apy uses the [[Apertium-apy#Language_identification|CLD2]] library for language identification (optionally it can use coverage of analysers, but this is really slow)
-See [[Apertium-apy/Language identification]] for some accuracy experiments of CLD2
 ==See also==
+* [[Apertium-apy/Language identification]] for some accuracy experiments of CLD2
 * http://odur.let.rug.nl/~vannoord/TextCat/ the original TextCat library (perl, there's also a C port)
 ** https://github.com/unhammer/gt-CorpusTools/blob/master/corpustools/text_cat.py python2 reimplementation of textcat