Format handling

Format handling in Apertium is done with special programs to encapsulate and de-encapsulate formatting information in "superblanks", which are delimited by the characters [ and ], so for example, for processing HTML, the program apertium-deshtml encapsulates the formatting information, while apertium-rehtml de-encapsulates (restores) it, as in the following example:

$ echo "<em>this is</em> a <b>test</b>" | apertium-deshtml
[<em>]this is[<\/em> ]a[ <b>]test.[][<\/b>]

$ echo "<em>this is</em> a <b>test</b>" | apertium-deshtml | apertium-rehtml
<em>this is</em> a <b>test</b>

Currently, deformatters and reformatters are available for:

• plain text (apertium-destxt)
• HTML (apertium-deshtml)
• RTF (apertium-desrtf)
• OpenOffice.org Writer ODT (apertium-desodt)
• Microsoft Word DOCX, WXML (apertium-deswxml)
• Microsoft Powerpoint PPTX (apertium-despptx)
• Microsoft Excel XLSX (apertium-desxlsx)
• QuarkXPress XpressTag (apertium-desxpresstag)
• MediaWiki (apertium-desmediawiki -- still a work in progress, see Translating wikimedia)

There is as of yet no built-in handling of gettext .po files or subtitle formats, but see Translating gettext and Translating subtitles for very simple solutions.

Note: apertium-destxt adds a period before any line-break that's not followed by text, meaning you sometimes get double periods.

Format specification

This section describes how the de-formatter and re-formatter are generated from a format specification in XML. Rules for format, like linguistic data, are specified in XML, and they contain regular expressions with flex syntax. The specification is divided in three parts (see its DTD in the Appendix A.6):

• Configuration options. Here one specifies the value for the maximum length of a non-extensive superblank, the input and output encodings, whether case must be considered, and the regular expressions for escape characters and space characters.
• Format rules. Describes the set of tags belonging to a specific format which have to be included in a block of format by the de-formatter. These tags may, optionally, indicate a sentence end, in which case the de-formatter will insert an artificial punctuation mark (followed by an empty block of format, as explained in the previous section). One has to specify the priority of application of the rules, although, when this is not relevant, it is possible to give the same priority to all the rules by assigning them the same value (any number). Everything that is not specified as format will be left without encapsulation and, therefore, will be considered as translatable text.
• Replacement rules. Allow to replace special characters in the text. A regular expression will recognise a set of special characters, and will replace it with the specified characters. For example, in HTML, the characters specified in hexadecimal have to be replaced with the corresponding entity or ASCII character. For example, camión corresponds to camión.

Root of the specification file

The attribute name contains the name of the format.

<?xml version="1.0" encoding="UTF-8"?>
<format name="html">
  <options>
    ...
  </options>
  <rules>
    ...
  </rules>
</format>

It has to include the options and rules, an example of which is presented next:

Options

The element <largeblocks> specifies the maximum length of a non-extensive superblank, through the value of the attribute size. The elements <input> and <output> specify the input and output encoding of the text, through the attribute encoding. The element escape-chars specifies, by means of a regular expression declared in the value of the attribute regexp, which characters must be escaped with a backslash. The element <space-chars> specifies the set of characters that must be considered as blanks. Finally, the element case-sensitive specifies if case is relevant in the specifications of format attributes in which regular expressions are contained.

Example

  <options>
    <largeblocks size="8192"/>
    <input encoding="UTF-8"/>
    <output encoding="UTF-8"/>
    <escape-chars regexp=’[\[\]ˆ$\\]’/>
    <space-chars regexp=’[ \n\t\r]’/>
    <case-sensitive value="no"/>
  </options>

Rules

There are format rules and replacement rules.

  <rules>
    <format-rule ... >
      ...
    </format-rule>
      ... 
    <replacement-rule>
      ...
    </replacement-rule>
      ...
    </rules>

The two types are described in the following points.

Format rules

The de-formatter will encapsulate in blocks of format the tags indicated by these rules in the field regexp. If they are begin and end tags, and everything delimited by them is format, one has to specify a regexp both for begin and for end:

  <format-rule eos="no" priority="1">
    <begin regexp=’"\<!--"’/>
    <end regexp=’"--\>"’/>
  </format-rule>

Otherwise only one begin-end element is used:

  <format-rule eos="yes" priority="3">
    <begin-end regexp=’"<"[/]?"li"[ˆ>]*">"’/>
  </format-rule>

Besides, in priority you have to specify a priority to tell the system in which order the format rules must be applied (the absolute value is not relevant, only the order resulting from the values). In “eos” you indicate, with yes or no, whether the block of format that contains the detected pattern must be preceded by an artificial punctuation mark or not.

Replacement rules

Are used to replace special characters in the text. The regular expression in the attribute regexp will recognise a set of special characters and will replace them with the specified characters in the text to be translated. The correspondence between original and replacement characters is stated in the attributes source and target of the replace elements, which can be multiple:

  <replacement-ruleregexp='"&"[ˆ;]+;'>
    <replacesource="À"target="À"/>
    <replacesource="À"target="À"/>
    <replacesource="À"target="À"/>
    <replacesource="À"target="À"/>
    <replacesource="Á"target="Á"/>
    <replacesource="Á"target="Á"/>
    <replacesource="Á"target="Á"/>
    <replacesource="Á"target="Á"/>
    ...
  </replacement-rule>

Regular expressions of regexp attributes

They have the syntax used in flex. As example of a format specification, we will give that for HTML. The explanation given in the following paragraphs can be followed looking at Figure 3.49. In the first place, we find the format rule that specifies in a general way all the HTML tags: it considers as HTML tag everything that begins with the sign < and ends with the sign >.

These rules have the priority 4, which is the lowest priority so that the more specific rules are applied preferentially. But before considering a tag in a general way by applying this rule, some of the higher priority rules will be applied.

In the case of HTML,

• Priority 1: The highest priority is for comments <!-- ... -->
• Priority 2: The marks for beginning and end <script> </script> and <style> </style>, where everything included by them is considered to be format.
• Priority 3: is for tags that indicate end of sentence (artificial punctuation), which are <br/>, ,
  , etc.
• Priority 4: Last of all are the replacement rules, which replace all the codes that begin with &, as specified in the regular expression. Then, each one of the replacements is defined: À, as well as À, À and À are replaced with À. The remaining special characters are declared in the same way.

Example

 <?xml version="1.0" encoding="UTF-8"?>
 <format name="html">
   <options>
     <largeblocks size="8192"/>
     <input encoding="UTF-8"/>
     <output encoding="UTF-8"/>
     <escape-chars regexp='[\[\]ˆ$\\]'/>
     <space-chars regexp='[ \ n\ t\ r]'/>
     <case-sensitive value="no"/>
   </options>
   <rules>
    <format-rule eos="no" priority="1">
       <begin regexp='"<!--"'/>
      <end regexp='"-->"'/>
    </format-rule>
    <format-rule eos="no" priority="2">
      <begin regexp='"<script"[ˆ>]*">"'/>
      <end regexp='"</script"[ˆ>]*">"'/>
    </format-rule>
    <format-rule eos="no" priority="2">
      <begin regexp='"<style"[ˆ>]*">"'/>
      <end regexp='"</style"[ˆ>]*">"'/>
    </format-rule>
    <format-rule eos="yes" priority="3">
      <begin-end regexp='"<"[/]?"br"[ˆ>]*">"'/>
    </format-rule>
    <!-- Here come more declarations of format-rule eos="yes"-->
    <!-- ...                                                -->
    <format-rule eos="no" priority="4">
      <begin-end regexp='"<"[a-zA-Z][ˆ>]*">"'/>
    </format-rule>
    <replacement-rule regexp=’"&"[ˆ;]+;’>
      <replace source="&Agrave;" target="À"/>
      <replace source="&#192;" target="À"/>
      <replace source="&#xC0;" target="À"/>
      <replace source="&#xc0;" target="À"/>
      <!-- Here come more replace elements                  -->
      <!-- --                                               -->
    </replacement-rule>
  </rules>
</format>

See also

• Apertium stream format
• Translating gettext