Difference between revisions of "Orthographic normalisation"
Jump to navigation
Jump to search
(New page: A module to do orthographic normalisation on input streams would be nice. For example ;Romanian * ţ → ț * ş → ș ;Lingala) |
|||
| Line 1: | Line 1: | ||
A module to do orthographic normalisation on input streams would be nice. |
A module to do orthographic normalisation on input streams would be nice. Some are more complicated than others. |
||
For example |
|||
;Romanian |
;Romanian |
||
Romanian has two characters that should be written with ''commas'' but are often (probably over 90% of text "in the wild") written with ''cedillas''. An orthographic normalisation module would convert the legacy version into the new version. |
|||
* ţ → ț |
* ţ → ț |
||
| Line 9: | Line 9: | ||
;Lingala |
;Lingala |
||
Pre-composed vs. composed unicode characters. |
|||
Revision as of 21:02, 24 September 2007
A module to do orthographic normalisation on input streams would be nice. Some are more complicated than others.
- Romanian
Romanian has two characters that should be written with commas but are often (probably over 90% of text "in the wild") written with cedillas. An orthographic normalisation module would convert the legacy version into the new version.
- ţ → ț
- ş → ș
- Lingala
Pre-composed vs. composed unicode characters.
