Format handling

Format handling in Apertium is done with special programs to encapsulate and de-encapsulate formatting information in "superblanks", which are delimited by the characters [ and ], so for example, for processing HTML, the program apertium-deshtml encapsulates the formatting information, while apertium-rehtml de-encapsulates (restores) it, as in the following example:

$ echo "<em>this is</em> a <b>test</b>" | apertium-deshtml
[<em>]this is[<\/em> ]a[ <b>]test.[][<\/b>]

$ echo "<em>this is</em> a <b>test</b>" | apertium-deshtml | apertium-rehtml
<em>this is</em> a <b>test</b>

To tell the whole translator to use a certain format, use -f, e.g. apertium -f html oc-ca infile outfile.

Official formats carried out by Apertium

Currently, deformatters and reformatters are available for:

• plain text: -f txt (apertium-destxt)
• HTML: -f html or -f html-noent (apertium-deshtml)
  • html prints non-ASCII chars using entities, html-noent keeps non-ASCII as-is; the difference is in the reformatter
• RTF: -f rtf (apertium-desrtf)
• OpenOffice.org Writer ODT: -f odt (apertium-desodt)
• Microsoft Word DOCX, WXML: -f wxml (apertium-deswxml)
• Microsoft Powerpoint PPTX: -f pptx (apertium-despptx)
• Microsoft Excel XLSX: -f xlsx (apertium-desxlsx)
• QuarkXPress XpressTag: -f xpresstag (apertium-desxpresstag)
• MediaWiki: -f wikimedia (apertium-desmediawiki -- still a work in progress, see Translating wikimedia)

There is as of yet no built-in handling of gettext .po files or subtitle formats, but see Translating gettext and Translating subtitles for very simple solutions.

Some "special" features and gotcha's:

• apertium-destxt adds a full stop before any line-break that's not followed by text, meaning you sometimes get two full stops, apertium-deshtml does this with paragraph markup. To avoid this, you currently need to patch apertium (some discussion).
• apertium-deshtml and other xml-based formatters accept the tag <apertium-notrans> to mean "don't translate this"; so if you have text that is not markup that you don't want translated, wrap it in that element like this:

  text to be translated<apertium-notrans>don't translate me</apertium-notrans> translate again

• To translate document formats like ODT, you may have to pass the infile/outfile on the command line instead of piping, e.g. do apertium -f odt oc-ca in.odt out.odt, not cat in.odt | apertium -f odt oc-ca > out.odt.

Formats carried out by separate packages

Other deformatters and reformatters were written directly in C or C++ language without using XML files. So, they don't follow format specification described in the following chapters. Therefore, they are distributed in separate packages.

• apertium-mediawiki is a package written in C++ that handles format for wikimedia documents with a better support of links.

• apertium-c-formatters is a package written in C that handles formats for :
  • man pages
  • mnemonic files (an alternative to .po files for multilingual user interfaces).

Limitations

The apertium tools can not deal with reordered superblanks. The following example shows what happens when superblanks exist betweeen reordered words/chunks:

$ echo '<i>Perro</i> <b>blanco</b>' |apertium es-en -f html
<i>White</i> <b>dog</b>

This is currently a hard problem to fix and will require changes to both the transfer engine and the transfer rules.

Read more at Reordering superblanks.

Format specification

This paragraph and the followings apply only to formats officially supported by Apertium.

This section describes how the de-formatter and re-formatter are generated from a format specification in XML. Rules for format, like linguistic data, are specified in XML, and they contain regular expressions with flex syntax. The specification is divided in three parts (see its DTD in the Appendix A.6):

• Configuration options. Here one specifies the value for the maximum length of a non-extensive superblank, the input and output encodings, whether case must be considered, and the regular expressions for escape characters and space characters.
• Format rules. Describes the set of tags belonging to a specific format which have to be included in a block of format by the de-formatter. These tags may, optionally, indicate a sentence end, in which case the de-formatter will insert an artificial punctuation mark (followed by an empty block of format, as explained in the previous section). One has to specify the priority of application of the rules, although, when this is not relevant, it is possible to give the same priority to all the rules by assigning them the same value (any number). Everything that is not specified as format will be left without encapsulation and, therefore, will be considered as translatable text.
• Replacement rules. Allow to replace special characters in the text. A regular expression will recognise a set of special characters, and will replace it with the specified characters. For example, in HTML, the characters specified in hexadecimal have to be replaced with the corresponding entity or ASCII character. For example, cami&oacute;n corresponds to camión.

Root of the specification file

The attribute name contains the name of the format.

<?xml version="1.0" encoding="UTF-8"?>
<format name="html">
  <options>
    ...
  </options>
  <rules>
    ...
  </rules>
</format>

It has to include the options and rules, an example of which is presented next:

Options

The element <largeblocks> specifies the maximum length of a non-extensive superblank, through the value of the attribute size. The elements <input> and <output> specify the input and output encoding of the text, through the attribute encoding. The element escape-chars specifies, by means of a regular expression declared in the value of the attribute regexp, which characters must be escaped with a backslash. The element <space-chars> specifies the set of characters that must be considered as blanks. Finally, the element case-sensitive specifies if case is relevant in the specifications of format attributes in which regular expressions are contained.

Example

  <options>
    <largeblocks size="8192"/>
    <input encoding="UTF-8"/>
    <output encoding="UTF-8"/>
    <escape-chars regexp=’[\[\]ˆ$\\]’/>
    <space-chars regexp=’[ \n\t\r]’/>
    <case-sensitive value="no"/>
  </options>

Rules

There are format rules and replacement rules.

  <rules>
    <format-rule ... >
      ...
    </format-rule>
      ... 
    <replacement-rule>
      ...
    </replacement-rule>
      ...
    </rules>

The two types are described in the following points.

Format rules

The de-formatter will encapsulate in blocks of format the tags indicated by these rules in the field regexp. If they are begin and end tags, and everything delimited by them is format, one has to specify a regexp both for begin and for end:

  <format-rule eos="no" priority="1">
    <begin regexp=’"\<!--"’/>
    <end regexp=’"--\>"’/>
  </format-rule>

Otherwise only one begin-end element is used:

  <format-rule eos="yes" priority="3">
    <begin-end regexp=’"<"[/]?"li"[ˆ>]*">"’/>
  </format-rule>

Besides, in priority you have to specify a priority to tell the system in which order the format rules must be applied (the absolute value is not relevant, only the order resulting from the values). In “eos” you indicate, with yes or no, whether the block of format that contains the detected pattern must be preceded by an artificial punctuation mark or not.

Replacement rules

Are used to replace special characters in the text. The regular expression in the attribute regexp will recognise a set of special characters and will replace them with the specified characters in the text to be translated. The correspondence between original and replacement characters is stated in the attributes source and target of the replace elements, which can be multiple:

  <replacement-ruleregexp='"&"[ˆ;]+;'>
    <replacesource="À"target="À"/>
    <replacesource="À"target="À"/>
    <replacesource="À"target="À"/>
    <replacesource="À"target="À"/>
    <replacesource="Á"target="Á"/>
    <replacesource="Á"target="Á"/>
    <replacesource="Á"target="Á"/>
    <replacesource="Á"target="Á"/>
    ...
  </replacement-rule>

Regular expressions of regexp attributes

They have the syntax used in flex. As example of a format specification, we will give that for HTML. The explanation given in the following paragraphs can be followed looking at Figure 3.49. In the first place, we find the format rule that specifies in a general way all the HTML tags: it considers as HTML tag everything that begins with the sign < and ends with the sign >.

These rules have the priority 4, which is the lowest priority so that the more specific rules are applied preferentially. But before considering a tag in a general way by applying this rule, some of the higher priority rules will be applied.

In the case of HTML,

• Priority 1: The highest priority is for comments <!-- ... -->
• Priority 2: The marks for beginning and end <script> </script> and <style> </style>, where everything included by them is considered to be format.
• Priority 3: is for tags that indicate end of sentence (artificial punctuation), which are <br/>, ,
  , etc.
• Priority 4: Last of all are the replacement rules, which replace all the codes that begin with &, as specified in the regular expression. Then, each one of the replacements is defined: À, as well as À, À and À are replaced with À. The remaining special characters are declared in the same way.

Example

 <?xml version="1.0" encoding="UTF-8"?>
 <format name="html">
   <options>
     <largeblocks size="8192"/>
     <input encoding="UTF-8"/>
     <output encoding="UTF-8"/>
     <escape-chars regexp='[\[\]ˆ$\\]'/>
     <space-chars regexp='[ \ n\ t\ r]'/>
     <case-sensitive value="no"/>
   </options>
   <rules>
    <format-rule eos="no" priority="1">
       <begin regexp='"<!--"'/>
      <end regexp='"-->"'/>
    </format-rule>
    <format-rule eos="no" priority="2">
      <begin regexp='"<script"[ˆ>]*">"'/>
      <end regexp='"</script"[ˆ>]*">"'/>
    </format-rule>
    <format-rule eos="no" priority="2">
      <begin regexp='"<style"[ˆ>]*">"'/>
      <end regexp='"</style"[ˆ>]*">"'/>
    </format-rule>
    <format-rule eos="yes" priority="3">
      <begin-end regexp='"<"[/]?"br"[ˆ>]*">"'/>
    </format-rule>
    <!-- Here come more declarations of format-rule eos="yes"-->
    <!-- ...                                                -->
    <format-rule eos="no" priority="4">
      <begin-end regexp='"<"[a-zA-Z][ˆ>]*">"'/>
    </format-rule>
    <replacement-rule regexp=’"&"[ˆ;]+;’>
      <replace source="&Agrave;" target="À"/>
      <replace source="&#192;" target="À"/>
      <replace source="&#xC0;" target="À"/>
      <replace source="&#xc0;" target="À"/>
      <!-- Here come more replace elements                  -->
      <!-- --                                               -->
    </replacement-rule>
  </rules>
</format>

See also

• Apertium stream format
• Translating gettext