User:Rcrowther - Revision history

Rcrowther at 12:55, 24 April 2017

2017-04-24T12:55:08Z

Rcrowther at 09:27, 24 April 2017

2017-04-24T09:27:08Z

Rcrowther at 08:27, 24 April 2017

2017-04-24T08:27:22Z

Show changes

Rcrowther at 15:37, 29 December 2016

2016-12-29T15:37:09Z

Rcrowther at 10:04, 23 December 2016

2016-12-23T10:04:07Z

Rcrowther at 10:00, 23 December 2016

2016-12-23T10:00:53Z

Rcrowther at 09:56, 23 December 2016

2016-12-23T09:56:29Z

Show changes

Rcrowther at 09:49, 23 December 2016

2016-12-23T09:49:44Z

Show changes

Rcrowther at 09:40, 23 December 2016

2016-12-23T09:40:24Z

Show changes

Hectoralos: /* Configuration Files */

2016-12-22T19:55:57Z

Configuration Files

@@ Line 30: / Line 30: @@
 ====  For translators: Install language data/dictionaries/pairs from repositories ====
-[[Install language data using packaging]], including hints about the private repository.
+[[Install language data using packaging]], including hints about the Apertium package repository.
 ====  For language developers: Install language data/dictionaries/pairs by compiling ====
@@ Line 42: / Line 42: @@
 ====Installing Apertium core by compiling====
-Apertium maintains a private repository that is up-to-date and reliable. If you do not want to work in core, or develop languages, please use either packaging or a virtual environment. The packages stay up-to-date and are stable. A compile will waste your time.
+Apertium maintains a package repository that is up-to-date and reliable. If you do not want to work in core, or develop languages, please use either packaging or a virtual environment. The packages stay up-to-date and are stable. A compile will waste your time.
 However, if you are planning to work on Apertium core, or have an operating system not covered above, go right ahead, [[Install Apertium core by compiling]]<ref name="about installing">Most people know the word 'install'. It means 'put code in my operating system'. When developing, it is not usual to fully 'install'. You get the code working enough to get results.
@@ Line 63: / Line 63: @@
 [[Category:Installation]]
 [[Category:Documentation in English]]
 = Minimal installation from SVN=

@@ Line 63: / Line 63: @@
 [[Category:Installation]]
 [[Category:Documentation in English]]
+= Minimal installation from SVN=
+This page is deprecated, and the information split across other pages.
+It used to contain instructions on how to compile Apertium core. For this, please see [[Install Apertium core by compiling]]
+How to create language builds with new and exisiting repository information. Please see [[Install language data by compiling]]
+And details about the HFST and CG modules. Please see [[Installation of grammar libraries]]
+Or start from the information root at [[Installation]]?

@@ Line 531: / Line 531: @@
 * [[Post-generator]]
+==Older/unusual modules==
+These modules may only be used for special cases, or have been deprecated, but are present in some existing pairs. They are included in the binary .deb distribution.
+===Constraint grammar===
+Also called 'Visl cg3' and 'CG3'.
+The 'Constraint Grammar' module is not used often. In Apertium, it is preferred, if possible, to use the 'Lexical Selection' module. The module is placed after the morphological analysis (the first monodix) and attempts to disambiguate text.
+A Constraint Grammar module is a very powerful way of disambiguating input text. It uses rules (compare to the 'POS tagger' module below, which uses statistics). The Constraint Grammar module can tear text apart and restructure, like the 'chunker' modules. Also, it is a project which extends outside Apertium, so there may be Constraint Grammar code available to disambiguate languages.
+While reusing and building on the work of others is a generous idea, the Apertium module has problems. The grammar is not consistent with other Apertium grammar, the power of the module leads to obscure code, and the module uses a lot of computing power. So we are not writing about it. If you are using a language pair which contains Constraint Grammar, or have a difficult language to translate, see the link below. Or ask on IRC.
+====Technical Description====
+Resolve ambiguous segments (e.e. where there is more than one match) by choosing, by rule building, a match.
+==Links==
+[http://wiki.apertium.org/wiki/Constraint_Grammar Constraint Grammar]
+===POS tagger and lexical selection===
+The 'POS Tagger' module is not used often. In Apertium, it is preferred, if possible, to use the 'Lexical Selection' module. The module is placed after the morphological analysis (the first monodix) and attempts to disambiguate text.
+The module disambiguates text by testing word placements against statistics. The statistics are generated by feeding pre-translated text into the module. Another way of saying this; 'Ok, this word is ambiguous. Now, the module has been told about many real-world translations. When it saw those translations, how often, and where, was the ambiguous word translated to one form or another? Based on that, what is the best translation?'.
+This module is not under direct control of a configuration file. It is trained by input of data. If the 'POS tagger' module is not trained, then data passes unchanged.
+The tagger has some configuration. The TSX format is used to configure a tagging process, identifying tags from the initial analysis, setting them as 'coarse tags'. These tags are used for the statistic building.
+'Coarse tags' are not much more than an overall name (e.g. "the word shrink as a noun"). The coarse tagging process can also accept or avoid some adjacent tag names (so "shrink" after "the" is a noun).
+====Technical Description====
+Resolve ambiguous segments (e.e. where there is more than one match) by choosing, by statistical training, a match.
+====Example====
+: "THE stage is more beholding to love"
+This group of words can be read in several ways. 'The stage' may refer to a 'stage' mentioned earlier in the text; for example, 'the meeting'. And 'more beholding' may mean 'a place that creates love'. In the original text, 'the stage' means 'the theatre', and 'more beholding' means 'a place for love to be presented'.
+It is very unlikely that a general translating software is trained in this old form of English. In one translation engine, a translation from English to Spanish and back resulted in "The scenario is looking more to love". The engine has tried, using a POS tagging/statistics-based analysis, to guess the intention of the words,
+: "(means "steps in a process")[THE stage] is (means "to look towards")[more beholding] to love"
+Here, the engine failed, but often it will succeed.
+====Typical stream output====
+The format of the output is a standard Apertium text stream. But the output may be changed from the morphological output in quality of analysis and volume. Here is tagger output, using the above example and the current English monodix,
+: ^THE/THE<det><def><sp>$ ^stage/stage<n><sg>/stage<vblex><inf>/stage<vblex><imp>$ ^is/be<vbser><pres><p3><sg>$ ^more/more<adv>/more<preadv>/more<n><sg>/more<det><qnt><sp>$ ^beholding/*beholding$ ^to/to<pr>$ ^love/love<n><sg>/love<vblex><inf>/love<vblex><pres>/love<vblex><imp>$^./.<sent>$
+but the tagger may have decided how to handle ambiguities and added new tags. Here is the output from the bidex tagger mode using the es-en pair,
+: ^the<det><def><sp>$ ^stage<n><sg>$ ^be<vbser><pri><p3><sg>$ ^more<preadv>$ ^*beholding$ ^to<pr>$ ^love<vblex><inf>$^.<sent>$
+You can see some ambiguities, such as the words 'stage' and 'love', which can be a noun or a verb, have been resolved.
+====Tool used====
+(not directly configurable)
+: apertium-tagger
+====Auto Mode====
+In a monodix,
+: xxx-tagger
+In a bidix,
+: xxx-yyy-tagger
+====Configuration Files====
+For the filter,
+: apertium-xxx-yyy.xxx.tsx
+(a new pair template template contains no .tsx files)
+====Links====
+[[Tagger training]]
+[[TSX format]]

@@ Line 56: / Line 56: @@
 ^An/a<det><ind><sg>$ ^ant/ant<n><sg>$ ^is/be<vbser><pres><p3><sg>$ ^a/a<det><ind><sg>$ ^wise/wise<adj><sint>$ ^creature/creature<n><sg>$ ^for/for<cnjadv>/for<pr>$ ^itself/itself<prn><ref><p3><nt><sg>$^./.<sent>$
 </pre>
 ====Tool used====
 : lt-proc
 Note this option/switch,
 :'-a', --analysis (default behaviour)
 For this stage, you will often see this option used (the mode scripts all configure with this switch),
 : -w, --dictionary-case:  use dictionary case instead of surface case
 ====Auto Mode====
 In a monodix,
 : xxx-morph
 Other modes exist which output results after the 'POS Tagger' or 'Constraint Grammar' modules are used. If these modules are not used (they are unusual), these modes will output the same as the 'morph' mode.
 In a bidex,
 : xxx-yyy-pgen
 There is also a mode 'pretransfer'. The Pretransfer module adds some fixes before the bidex. The output will often be the same as 'pgen'.
 ====Configuration Files====
 The monolingual dictionary,
 : apertium-xxx-yyy.xxx.dix
 ====Other Files====
 The .acx file is used to define equivalent characters (and can grab apostrophes). Not necessary for making a basic pair,
 : apertium-xxx-yyy.xxx.acx
 ====Links====
-[[Monodix_basics]]
+* [[Monodix_basics]]
-[[Apertium_New_Language_Pair_HOWTO]]
+* [[Apertium_New_Language_Pair_HOWTO]]
-[[Acx]]
+* [[Acx]]
@@ Line 150: / Line 165: @@
 ====Links====
-[[Apertium_New_Language_Pair_HOWTO]]
+* [[Apertium_New_Language_Pair_HOWTO]]
-[[Bilingual_dictionary]]
+* [[Bilingual_dictionary]]
@@ Line 196: / Line 211: @@
 ====Links====
-[[How_to_get_started_with_lexical_selection_rules]]
+* [[How_to_get_started_with_lexical_selection_rules]]
-[[Constraint-based lexical selection module]]
+* [[Constraint-based lexical selection module]]
@@ Line 267: / Line 282: @@
 ====Links====
-[[Chunking]]
+* [[Chunking]]
-[[Chunking:_A_full_example]]
+* [[Chunking:_A_full_example]]
@@ Line 311: / Line 326: @@
 ====Links====
-[[Chunking:_A_full_example]]
+* [[Chunking:_A_full_example]]
@@ Line 360: / Line 375: @@
 ====Links====
-[[Chunking:_A_full_example]]
+* [[Chunking:_A_full_example]]
@@ Line 407: / Line 422: @@
 ====Links====
-[[Monodix_basics]]
+* [[Monodix_basics]]
-[[Apertium_New_Language_Pair_HOWTO]]
+* [[Apertium_New_Language_Pair_HOWTO]]
@@ Line 515: / Line 530: @@
 The post generator usually only contains a handful of rules, often common, so is not covered by much documentation. For depth, try the [https://wiki.apertium.org/w/images/d/d0/Apertium2-documentation.pdf Apertium 2.0: Official documentation] (Sect. 3.1.2). For a quick-reference example,
-[[Post-generator]]
+* [[Post-generator]]

@@ Line 56: / Line 56: @@
 ^An/a<det><ind><sg>$ ^ant/ant<n><sg>$ ^is/be<vbser><pres><p3><sg>$ ^a/a<det><ind><sg>$ ^wise/wise<adj><sint>$ ^creature/creature<n><sg>$ ^for/for<cnjadv>/for<pr>$ ^itself/itself<prn><ref><p3><nt><sg>$^./.<sent>$
 </pre>
 ====Tool used====
 : lt-proc
 Note this option/switch,
 :'-a', --analysis (default behaviour)
 For this stage, you will often see this option used (the mode scripts all configure with this switch),
 : -w, --dictionary-case:  use dictionary case instead of surface case
 ====Auto Mode====
 In a monodix,
 : xxx-morph
 Other modes exist which output results after the 'POS Tagger' or 'Constraint Grammar' modules are used. If these modules are not used (they are unusual), these modes will output the same as the 'morph' mode.
 In a bidex,
 : xxx-yyy-pgen
 There is also a mode 'pretransfer'. The Pretransfer module adds some fixes before the bidex. The output will often be the same as 'pgen'.
 ====Configuration Files====
 The monolingual dictionary,
 : apertium-xxx-yyy.xxx.dix
 ====Other Files====
 The .acx file is used to define equivalent characters (and can grab apostrophes). Not necessary for making a basic pair,
 : apertium-xxx-yyy.xxx.acx
 ====Links====
 [[Monodix_basics]]