Format handling

Data: format specification rules This section describes how the de-formatter and re-formatter are generated from a format specification in XML. Rules for format, like linguistic data, are specified in XML, and they contain regular expressions with flex syntax. The specification is divided in three parts (see its DTD in the Appendix A.6):

• Configuration options. Here one specifies the value for the maximum length of a non-extensive superblank, the input and output encodings, whether case must be considered, and the regular expressions for escape characters and space characters.
• Format rules. Describes the set of tags belonging to a specific format which have to be included in a block of format by the de-formatter. These tags may, optionally, indicate a sentence end, in which case the de-formatter will insert an artificial punctuation mark (followed by an empty block of format, as explained in the previous section). One has to specify the priority of application of the rules, although, when this is not relevant, it is possible to give the same priority to all the rules by assigning them the same value (any number). Everything that is not specified as format will be left without encap sulation and, therefore, will be considered as translatable text.
• Replacement rules. Allow to replace special characters in the text. A regular expression will recognize a set of special characters, and will replace it with the specified characters. For example, in HTML, the characters specified in hexadecimal have to be replaced with the corresponding entity or ASCII character. For example, camión corresponds to camión.

Rules are described in more detail next.

• Root of the specification file. The attribute name contains the name of the format.

      <?xml version="1.0" encoding="ISO-8859-1"?>
      <format name="html">
         <options>
         ...
         </options>
         <rules>
         ...
         </rules>
      </format>

It has to include the options and rules, an example of which is presented next:

Options

         <options>
           <largeblocks size="8192"/>
           <input encoding="ISO-8859-1"/>
           <output encoding="ISO-8859-1"/>
           <escape-chars regexp=’[\[\]ˆ$\\]’/>
           <space-chars regexp=’[ \n\t\r]’/>
           <case-sensitive value="no"/>
         </options>

The element <largeblocks> specifies the maximum length of a non-extensive superblank, through the value of the attribute size. The elements <input> and <output> specify the input and output encoding of the text, through the attribute encoding. The element escape-chars specifies, by means of a regular expression declared in the value of the attribute regexp, which characters must be escaped with a backslash. The element <space-chars> specifies the set of characters that must be considered as blanks. Finally, the element case-sensitive specifies if case is relevant in the specifications of format attributes in which regular expressions are contained.

Rules

There are format rules and replacement rules.

    <rules>
       <format-rule ... >
         ...
       </format-rule>
       ...
       <replacement-rule>
         ...
       </replacement-rule>
       ...
    </rules>

The two types are described in the following points.

Format rules

The de-formatter will encapsulate in blocks of format the tags indicated by these rules in the field regexp. If they are begin and end tags, and everything delimited by them is format, one has to specify a regexp both for begin and for end:

       <format-rule eos="no" priority="1">
         <begin regexp=’"\<!--"’/>
         <end regexp=’"--\>"’/>
       </format-rule>

Otherwise only one begin-end element is used:

       <format-rule eos="yes" priority="3">
         <begin-end regexp=’"<"[/]?"li"[ˆ>]*">"’/>
       </format-rule>

Besides, in priority you have to specify a priority to tell the system in which order the format rules must be applied (the absolute value is not relevant, only the order resulting from the values). In “eos” you indicate, with yes or no, whether the block of format that contains the detected pattern must be preceded by an artificial punctuation mark or not.

Replacement rules

Are used to replace special characters in the text. The regular expression in the attribute regexp will recognize a set of special characters and will replace them with the specified characters in the text to be translated. The correspondence between original and replacement characters is stated in the attributes source and target of the replace elements, which can be multiple:

           <replacement-rule regexp=’"&"[ˆ;]+;’>
              <replace source="&Agrave;" target="À"/>
              <replace source="&#192;" target="À"/>
              <replace source="&#xC0;" target="À"/>
              <replace source="&#xc0;" target="À"/> 
              <replace source="&Aacute;" target="Á"/>
              <replace source="&#193;" target="Á"/>
              <replace source="&#xC1;" target="Á"/>
              <replace source="&#xc1;" target="Á"/>
              ...
           </replacement-rule>

Regular expressions of regexp attributes

They have the syntax used in flex. As example of a format specification, we will give that for HTML. The explanation given in the following paragraphs can be followed looking at Figure 3.49. In the first place, we find the format rule that specifies in a general way all the HTML tags: it considers as HTML tag everything that begins with the sign < and ends with the sign >. This rule has the lowest priority (4) so that the more specific rules are applied preferentially. But before con- sidering a tag in a general way by applying this rule, some of the higher priority rules will be applied. In the case of HTML, the highest prior- ity is for comments . The marks for beginning and end <script> </script> and <style> </style>, where everything in- cluded by them is considered to be format, has priority 2. Priority 3 is for tags that indicate end of sentence (artificial punctuation), which are

, ,

, etc. Last of all are the replacement rules, which replace all the codes that

begin with &, as specified in the regular expression. Then, each one of the replacements is defined: &Agrave, as well as &#192, &#xC0 and &#xc0 are replaced with À. The remaining special characters are declared in the same way.

<?xml version="1.0" encoding="UTF-8"?>
<format name="html">
  <options>
    <largeblocks size="8192"/>
    <input encoding="UTF-8"/>
    <output encoding="UTF-8"/>
    <escape-chars regexp=’[\[\]ˆ$\\]’/>
    <space-chars regexp=’[ \ n\ t\ r]’/>
    <case-sensitive value="no"/>
  </options>
  <rules>
   <format-rule eos="no" priority="1">
      <begin regexp=’"<!--"’/>
     <end regexp=’"-->"’/>
   </format-rule>
   <format-rule eos="no" priority="2">
     <begin regexp=’"<script"[ˆ>]*">"’/>
     <end regexp=’"</script"[ˆ>]*">"’/>
   </format-rule>
   <format-rule eos="no" priority="2">
     <begin regexp=’"<style"[ˆ>]*">"’/>
     <end regexp=’"</style"[ˆ>]*">"’/>
   </format-rule>
   <format-rule eos="yes" priority="3">
     <begin-end regexp=’"<"[/]?"br"[ˆ>]*">"’/>
   </format-rule>
   <format-rule eos="no" priority="4">
     <begin-end regexp=’"<"[a-zA-Z][ˆ>]*">"’/>
   </format-rule>
   <replacement-rule regexp=’"&"[ˆ;]+;’>
     <replace source="&Agrave;" target="À"/>
     <replace source="&#192;" target="À"/>
     <replace source="&#xC0;" target="À"/>
     <replace source="&#xc0;" target="À"/>