Orthographic normalisation - Revision history

Bech: Category:Documentation in English

Bech — Sat, 24 Mar 2012 11:43:43 GMT

Category:Documentation in English

Francis Tyers — Fri, 18 Apr 2008 14:24:05 GMT

Francis Tyers — Sat, 20 Oct 2007 23:14:31 GMT

Francis Tyers — Sat, 20 Oct 2007 23:11:42 GMT

Francis Tyers — Sat, 20 Oct 2007 21:29:52 GMT

Francis Tyers — Sat, 20 Oct 2007 21:25:43 GMT

Francis Tyers — Fri, 05 Oct 2007 13:29:59 GMT

Francis Tyers — Tue, 25 Sep 2007 13:25:42 GMT

Francis Tyers — Mon, 24 Sep 2007 21:02:00 GMT

Francis Tyers — Sun, 23 Sep 2007 18:35:05 GMT

New page: A module to do orthographic normalisation on input streams would be nice. For example ;Romanian * ţ → ț * ş → ș ;Lingala

New page

A module to do orthographic normalisation on input streams would be nice.

For example

;Romanian

* ţ → ț
* ş → ș

;Lingala

← Older revision		Revision as of 11:43, 24 March 2012
Line 36:		Line 36:

	[[Category:Development]]		[[Category:Development]]
			[[Category:Documentation in English]]

@@ Line 1: / Line 1: @@
-A module to do orthographic normalisation on input streams would be nice. Some are more complicated than others.
+A module to do orthographic normalisation on input streams would be nice. Some are more complicated than others. Certain forms of orthographic normalisation (for example for Romanian ș and ş) can be done with the [[ACX format]] which allows the definition of sets of equivalent characters.
-;Romanian
-Romanian has two characters that should be written with ''commas'' but are often (probably over 90% of text "in the wild") written with ''cedillas''. Apertium has standardised on the forms with commas. An orthographic normalisation module would convert the legacy version into the new version.
-* ţ → ț
-* ş → ș
 ;Serbo-Croatian

@@ Line 3: / Line 3: @@
 ;Romanian
-Romanian has two characters that should be written with ''commas'' but are often (probably over 90% of text "in the wild") written with ''cedillas''. An orthographic normalisation module would convert the legacy version into the new version.
+Romanian has two characters that should be written with ''commas'' but are often (probably over 90% of text "in the wild") written with ''cedillas''. Apertium has standardised on the forms with commas. An orthographic normalisation module would convert the legacy version into the new version.
 * ţ → ț

@@ Line 12: / Line 12: @@
 There are a couple of special characters in Serbo-Croatian which can be written with two characters or one character, a decision must be made which version to use in the dictionaries, and then forms not like this need to be converted:
-* dž
+* dž ←→ ǆ
-* lj
+* lj ←→ ǉ
-* nj
+* nj ←→ ǌ
 ;Afrikaans

@@ Line 9: / Line 9: @@
 ;Serbo-Croatian
+There are a couple of special characters in Serbo-Croatian which can be written with two characters or one character, a decision must be made which version to use in the dictionaries, and then forms not like this need to be converted:
+* dž
+* lj
+* nj
 ;Afrikaans
@@ Line 14: / Line 20: @@
 The indefinite article in Afrikaans is "'n". This can be written a number of different ways:
+*'n  U+0027 U+006E
-*'n
-*‘n
+*‘n  U+2018 U+006E
-*ŉ
+*ŉ   U+0149
-*’n
+*’n  U+2019 U+006E
 This ideally needs to be merged into one form.

@@ Line 10: / Line 10: @@
 ;Lingala
+{{see-also|Unicode issues}}
-Pre-composed vs. composed unicode characters.
+When a character has an accent, sometimes there is more than one way of representing it, using either pre-combined (sometimes referred to as ''pre-composed'') or combining characters. These look different when encoded in UTF-8, but the same to the user.
+<pre>
+UTF-8 0xC3 0xA0 vs. 0x61 0xCC 0x81
+      á         vs.      á
+      U+00E1    vs. U+0061 U+0301
+</pre>
+The best thing to do is probably standardise on one variant for analysis/generation, and then normalise all input coming into the analyser using a transliterator or something similar.
 [[Category:Development]]

← Older revision		Revision as of 13:25, 25 September 2007
Line 11:		Line 11:

	Pre-composed vs. composed unicode characters.		Pre-composed vs. composed unicode characters.

			[[Category:Development]]