User:Rcrowther

(proposed page 'Apertium workflow reference', or similar title)

You can invent your own tags to pass information along.

Another example:

"I may give only this advice"

If this is being translated to a Subject-Object-Verb language (English is a Subject-Verb-Object language, but many languages are not), it will need rearranging. If the wards are simple in grammar, they may be left as the morphological analysis has found them. More complex word sequences are gathered into a chunk and, if necessary, tagged,

I <a verb>{may give only} <some object>{this advice}

The "may" and "only" words also need to be handled. They could be cut, which is also a job for this module. If they are not cut, then they will need to be tagged, so they can be re-ordered. To see what happens next, look at the next module, 'interchunk'.

Typical stream output

From,

'a poppy',

The curly brackets are the stream representation for a 'chunk',

^Det_nom<SN><DET><f><sg>{^uno<det><ind><3><4>$ ^amapola<n><3><4>$}$

Tool used

apertium-transfer

Auto Mode

xxx-yyy-chunker

Configuration Files

apertium-xxx-yyy.xxx-yyy.t1x

Links

Chunking Chunking:_A_full_example

Transfer 2/InterChunk

In a three-stage transfer, the second transfer stage orders chunked and tagged items from stage one.

Configuring this stage is not necessary for making a basic pair.

The detection is of patterns/sequences of chunks. This module can not match words in chunks, only the marks added to chunks.

For language pairs with no major reordering between chunks, this module is not needed. If the 't2x' file is not configured, the module passes data unaltered. For example, 'en-es' has a Postchunk module (see next section), but not an Interchunk module.

Technical Description

Reorder or modify chunk sequences (e.g. transfer noun gender to related adjectives).

Example

From the previous example,

"I may give only this advice"

If this is being translated to a Subject-Object-Verb language (English is a Subject-Verb-Object language, but many languages are not), it will need rearranging. At the very least, the target dictionary will need,

I advice give

and it is in this module the words are rearranged.

Typical stream output

The module only reorders chunks. It has no effect on the form of the stream. But see 'mode'.

Tool used

apertium-interchunk

Auto Mode

xxx-yyy-interchunk

Configuration Files

apertium-xxx-yyy.xxx-yyy.t2x

Links

Chunking:_A_full_example

Transfer 3/PostChunk

In a three-stage transfer, the third transfer stage interferes with the resolution and writing of chunks.

Configuring this stage is not necessary for making a basic pair.

Detection is not by pattern matching, it is by the name/lemma of the chunk itself. Position marks refer to the words/lexical units inside the chunks. The module will not write chunks, only lexical units and blanks.

So PostChunk is less abstracted than Transfer 2/InterChunk processing.

For language pairs with no rewriting of chunks, this module is not needed. If the 't3x' file is not configured, the module defaults to resolving and removing chunk data.

Technical Description

Substitute fully-tagged target-language forms into the chunks.

Example

Reducing a previous example, text arrives, prepared by the Chunker, labelled as feminine and singular. The following stripped version of the input shows the 'chunk' marks,

^<f><sg>{^uno<det><ind>$ ^amapola<n>$}$

Now the postchunk module must render this. In English, the chunk had no gender, so neither did the 'a' word/determiner. Now it has a gender, and the chunker stages have defined where these tags should be applied. In some cases of translation, the chunks may also be reordered.

Typical stream output

From,

"a poppy"

Chunk marks have been removed from the stream (compare to 'chunker' output above), and tags distributed,

^Uno<det><ind><f><sg>$ ^amapola<n><f><sg>$

The output looks much the same as before the chunker stages. However, tags may have been added and deleted, and chunks of words recordered, to suit the target language.

Tool used

apertium-postchunk

Auto Mode

xxx-yyy-postchunk

Configuration Files

apertium-xxx-yyy.xxx-yyy.t3x

Links

Chunking:_A_full_example

Morphological Generator

'Generate' the surface forms of the translated words.

At this point, the text stream contains target language lemmas and tags, perhaps modified and prepared by the Lexical Selector and/or chunker stages. But this is not the final form. The Morphological Generator stage needs to takes the lemma and tags, then generate the target-language surface form e.g. it needs to take the lemma 'knife', and the tag '<pl>' (for plural), then generate 'knives'.

For this, the target language monodix is used in the direction right/left (in reverse of left->right reading). 'surface form' <- 'lexical unit'.

Technical Description

Use the lemma and tags ('lexical unit') to deliver the correct target-language surface form.

Example

The output, now translated,

"He that travels into a country, before he has some entrance into the language, goes to school, and not to travel"

becomes,

"Él aquello viaja a un país, antes de quei tiene alguna entrada a la lengua, va a escuela, y no para viajar"

This may be changed, for a few surface forms, by the Post Generator, and then formatted.

Typical stream output

The translation, now stripped of stream formatting, but before Post Generator and formatting. See the example above.

Tool used

lt-proc -g

The switch/option,

-g, --generation: morphological generation

Auto Mode

Post Chunker debug output (before Post Generator),

xxx-yyy-dgen

Configuration Files

apertium-xxx-yyy.yyy.dix

Links

Monodix_basics Apertium_New_Language_Pair_HOWTO

PostGenerator

Corrects or localises spelling where the adjustment relies on the next word.

Configuring this stage is not necessary for making a basic pair.

The post-generator uses a dictionary very similar to a mono-dictionary, used as a generator. So it is capable of creating, modifying and removing text. It can use paradigms. However, please read the rest of this section! The generator must be triggered using an '<a/>' tag in the bidex.

The module was originally provided to convert Spanish-like 'de el' into 'del'. It also performs a good job on placing 'a'/'an' determiners before English nouns ('an apple'). Here you can see the two main features of the post-generator. First, it works on the text as generated, so can be used when the form of a word is closely related to the final form of the following word. Note that 'de el' and 'a'/'an' could not be handled earlier in the text stream/Apertium workflow, because we have no idea what the surface forms will be. These forms are only available after the generating monodix. The second feature is that, in general, the post-generator works by inflecting/selecting/replacing a word based on the following word.

The post-generator is sometimes referred to in documentation as intended for 'orthography'. Orthography is conventions of spelling, hyphenation, and other graphical display features i.e. the language side of typography. Perhaps that was the original intention for the Post Generator, but the module at the time of writing is unsuitable for many orthographic tasks. It displays several unexpected behaviours. Attempts at elision and compression, other than a 'de el'->'del' style of elision across the forward word boundary, are likely to fail. However, the module is so useful for these two cases alone that it is an established stage in the workflow.

Technical Description

Make final adjustments where a generated surface form relies on the next surface form.

Example

The example from the manual is Spanish,

"de el"

which becomes,

"del"

And the template includes an example in English,

"a"

which becomes "an" before a vowel,

"an apple" (but "a car")

Both examples are beyond pure orthography, but depend on the final surface forms and that they are next to each other,

The Post Generator handles difficult cases. For example, we translate into English,

Un peligro inminente

The Post Generator will successfully handle the determiner, translating to,

An imminent danger

It would be very difficult handle this action earlier in the Apertium workflow. It may also confuse intentions in the code, and maybe limit other work we needed to do.

But the Post Generator is not useful for some actions. English sometimes hyphenates groups of words,

"But all this while, when I speak of vain-glory..."

Other common hyphenated groups are "misty-eyed", "follow-up", and "computer-aided". Finding a rule for this form of hyphenation is not easy. Let us imagine a rule exists. Unfortunately, the Post Generator could not handle the insertion of the hyphen, because it is made to recognise the following blank and either replace the first word, or the words as a whole. Language pair en-es handles the above examples by recognising these word groups in the initial monodix, not by Post Generator manipulation.

For the same reason, the Post-Generator can not handle another orthographic action in English; the use of the apostrophe. For example,

"that is but a circle of tales"

will often become,

that's but a circle of tales

The rule is clear and, for this example, reaches across a following blank. But the the Post Generator can give unintended results when manipulating a single letter (such as an 's') and is more predictable with a full replacement. Also, there are many such compressions and elisions in English ('where is' -> 'where's' etc.), so may be better to handle these with more general rules earlier in the workflow, or consider if the translation is better without them.

Typical stream output

The translation, now stripped of stream formatting e.g.

"that is but a circle of tales"

gives,

Aquello es pero un círculo de cuentos"

Tool used

lt-proc -p

The switch/option,

'-p', --postgeneration

Auto Mode

Input,

xxx-yyy-dgen

which is the debug output of the Post Chunker. Output, excluding formatting, is the finished product, so,

xxx-yyy

Configuration Files

The files are in the mono-dictionary folders. For the source language,

apertium-xxx.post-xxx.dix

For the target language,

apertium-yyy.post-yyy.dix

Links

The post generator usually only contains a handful of rules, often common, so is not covered by much documentation. For depth, try the Apertium 2.0: Official documentation (Sect. 3.1.2). For a quick-reference example,

Post-generator

References

Text examples from,

• Francis Bacon (1625). THE ESSAYS OR COUNSELS, CIVIL AND MORAL. Project Gutenberg