Unicode issues - Revision history

Bech: Category:Documentation in English

2012-03-24T11:46:17Z

Category:Documentation in English

Francis Tyers at 13:34, 5 October 2007

2007-10-05T13:34:06Z

Francis Tyers at 13:33, 5 October 2007

2007-10-05T13:33:47Z

Francis Tyers at 13:28, 5 October 2007

2007-10-05T13:28:14Z

Francis Tyers: /* Unicode normalization */

2007-06-13T16:40:01Z

Unicode normalization

Moyogo: /* Combining vs. pre-combined characters */

2007-06-13T16:18:45Z

Combining vs. pre-combined characters

Francis Tyers at 15:43, 13 June 2007

2007-06-13T15:43:49Z

Francis Tyers: New page: Some issues (potential and otherwise) with Unicode support. ==Combining vs. pre-combined characters== When a character has an accent, sometimes there is more than one way of representing...

2007-06-13T15:43:11Z

New page: Some issues (potential and otherwise) with Unicode support. ==Combining vs. pre-combined characters== When a character has an accent, sometimes there is more than one way of representing...

New page

Some issues (potential and otherwise) with Unicode support.

==Combining vs. pre-combined characters==

When a character has an accent, sometimes there is more than one way of representing it, using either pre-combined or combining characters. These look different in UTF-8, but the same to the user.

<pre>
UTF-8 0xC3 0xA0 vs. 0x61 0xCC 0x81
á vs. á
</pre>

The best thing to do is probably standardise on one variant for analysis/generation, and then normalise all input coming into the analyser using a transliterator or something similar.

← Older revision		Revision as of 11:46, 24 March 2012
Line 59:		Line 59:

	[[Category:Development]]		[[Category:Development]]
			[[Category:Documentation in English]]

← Older revision		Revision as of 13:34, 5 October 2007
Line 1:		Line 1:
			{{TOCD}}
	Some issues (potential and otherwise) with Unicode support.		Some issues (potential and otherwise) with Unicode support.

@@ Line 52: / Line 52: @@
 ==Zero-width non-joiner (ZWNJ)==
+==See also==
+* [[Orthographic normalisation]]
+[[Category:Development]]

@@ Line 14: / Line 14: @@
 The best thing to do is probably standardise on one variant for analysis/generation, and then normalise all input coming into the analyser using a transliterator or something similar.
-=== Unicode normalization ===
+=== Unicode normalisation ===
-Unicode defines normalization algorithms to transform various semantically equivalent strings to a single form.
+Unicode defines normalisation algorithms to transform various semantically equivalent strings to a single form.
-; NFD (Normalization Form Canonical Decomposition)
+; NFD (Normalisation Form Canonical Decomposition)
 The strings are decomposed and characters are reordered.
 Here's an example with 4 strings that are equivalent 'e&#769;&#803;', 'é&#803;', 'ẹ&#769;' and 'e&#803;&#769;'. If they are not the same at your end you are missing a good font or the software you use does not support that part of Unicode.
-When normalized to NFD all those 4 strings have the same form and can be used as they are meant to, with the same meaning.
+When normalised to NFD all those 4 strings have the same form and can be used as they are meant to, with the same meaning.
 <pre>
 UTF-8 0x65 0xCC 0x81 0xCC 0xA3 -> 0x65 0xCC 0xA3 0xCC 0x81
@@ Line 35: / Line 35: @@
 </pre>
-; NFC (Normalization Form Canonical Composition)
+; NFC (Normalisation Form Canonical Composition)
 The strings are decomposed, characters are reordered and are composed
-With the same example, all strings are normalized to a different form.
+With the same example, all strings are normalised to a different form.
 <pre>
 UTF-8 0x65 0xCC 0xA3 0xCC 0x81 -> 0xE1 0xBA 0xB9 0xCC 0x81

@@ Line 14: / Line 14: @@
 The best thing to do is probably standardise on one variant for analysis/generation, and then normalise all input coming into the analyser using a transliterator or something similar.
-== Unicode normalization ==
+=== Unicode normalization ===
 Unicode defines normalization algorithms to transform various semantically equivalent strings to a single form.

@@ Line 6: / Line 6: @@
 <pre>
 UTF-8 0xC3 0xA0 vs. 0x61 0xCC 0x81
       á         vs.      á
+      U+00E1    vs. U+0061 U+0301
 </pre>
 The best thing to do is probably standardise on one variant for analysis/generation, and then normalise all input coming into the analyser using a transliterator or something similar.
+== Unicode normalization ==
+Unicode defines normalization algorithms to transform various semantically equivalent strings to a single form.
+; NFD (Normalization Form Canonical Decomposition)
+The strings are decomposed and characters are reordered.
+Here's an example with 4 strings that are equivalent 'e&#769;&#803;', 'é&#803;', 'ẹ&#769;' and 'e&#803;&#769;'. If they are not the same at your end you are missing a good font or the software you use does not support that part of Unicode.
+When normalized to NFD all those 4 strings have the same form and can be used as they are meant to, with the same meaning.
+<pre>
+UTF-8 0x65 0xCC 0x81 0xCC 0xA3 -> 0x65 0xCC 0xA3 0xCC 0x81
+      e&#769;&#803;                        -> e&#803;&#769;
+      U+0065 U+0301 U+0323     -> U+0065 U+0323 U+0301
+UTF-8 0xC3 0xA9 0xCC 0xA3      -> 0x65 0xCC 0xA3 0xCC 0x81
+      é&#803;                        -> e&#803;&#769;
+      U+00E9 U+0323            -> U+0065 U+0323 U+0301
+UTF-8 0xE1 0xBA 0xB9 0xCC 0x81 -> 0x65 0xCC 0xA3 0xCC 0x81
+      ẹ&#769;                        -> e&#803;&#769;
+      U+1EB9 U+0301            -> U+0065 U+0323 U+0301
+</pre>
+; NFC (Normalization Form Canonical Composition)
+The strings are decomposed, characters are reordered and are composed
+With the same example, all strings are normalized to a different form.
+<pre>
+UTF-8 0x65 0xCC 0xA3 0xCC 0x81 -> 0xE1 0xBA 0xB9 0xCC 0x81
+      e&#803;&#769;                        -> ẹ&#769;
+      U+0065 U+0323 U+0301     -> U+1EB9 U+0301
+UTF-8 0x65 0xCC 0x81 0xCC 0xA3 -> 0xE1 0xBA 0xB9 0xCC 0x81
+      e&#769;&#803;                        -> ẹ&#769;
+      U+0065 U+0301 U+0323     -> U+1EB9 U+0301
+UTF-8 0xC3 0xA9 0xCC 0xA3      -> 0xE1 0xBA 0xB9 0xCC 0x81
+      é&#803;                        -> ẹ&#769;
+      U+00E9 U+0323            -> U+1EB9 U+0301
+</pre>
 ==Zero-width non-joiner (ZWNJ)==

← Older revision		Revision as of 15:43, 13 June 2007
Line 11:		Line 11:

	The best thing to do is probably standardise on one variant for analysis/generation, and then normalise all input coming into the analyser using a transliterator or something similar.		The best thing to do is probably standardise on one variant for analysis/generation, and then normalise all input coming into the analyser using a transliterator or something similar.

			==Zero-width non-joiner (ZWNJ)==