Popcorndude: /* Restoring Capitalization */

2024-03-04T19:44:56Z

Restoring Capitalization

Popcorndude: document rules

2022-12-22T20:15:35Z

document rules

Popcorndude: start documenting new capitalization module

2022-12-22T18:32:40Z

start documenting new capitalization module

New page

This page describes the modules for extracting and restoring capitalization information which were added in apertium 3.9.0.

Using these modules, pipelines can be changed so that every step after the analyzer operates solely on dictionary case and there is no need to track the proper placement of case in transfer.

== Extracting Capitalization ==

The program <code>apertium-extract-caps</code> stores information about the source language capitalization in [[Wordbound blanks|word-bound blanks]] with the prefix <code>c:</code>.

echo '^THE/the<det>$ ^Green/green<adj>$ ^stuff/stuff<n>$ ^from/from<pr>$ ^Target/Target<np>$' | apertium-extract-caps
<nowiki>[[c:AA/aa]]^the<det>$ [[c:Aa/aa]]^green<adj>$ [[c:aa/aa]]^stuff<n>$ [[c:aa/aa]]^from<pr>$ [[c:Aa/Aa]]^Target<np>$</nowiki>

To add it to a [[Modes|modes.xml]] file, put the following after the part-of-speech tagger:

<program name="apertium-extract-caps"/>

And make the following changes to the preceding programs, if applicable:
* Add <code>-w</code> to <code>lt-proc</code> or <code>hfst-proc</code>
* Remove <code>-w</code> and <code>-n</code> from <code>cg-proc</code>
* Add <code>-p</code> to <code>apertium-tagger</code>

If a later stage of the pipeline needs the surface forms, add the <code>-s</code> option to <code>apertium-extract-caps</code>.

== Restoring Capitalization ==

@@ Line 47: / Line 47: @@
  	apertium-validate-crx $<
  	apertium-compile-caps $< $@
+And also add <code>$(PREFIX1).crx.bin</code> to <code>TARGETS_COMMON</code>.
 == Capitalization Restoration Rule Format ==

@@ Line 22: / Line 22: @@
 == Restoring Capitalization ==
+The program <code>apertium-restore-caps</code> uses the information from <code>apertium-extract-caps</code> and the dictionary case of the target language forms to determine the proper capitalization of the target language surface forms according to a set of rules whose format is described below.
+ # supposing an input like "Long cat"
+ <nowiki>echo '[[c:aa/aa]]^gata<n>/gata$ [[c:Aa/aa]]^larga<adj>/larga$' | apertium-restore-caps abc-xyz.crx.bin</nowiki>
+ <nowiki>[[c:aa/aa]]Gata[[/]] [[c:Aa/aa]]larga[[/]]</nowiki>
+This module converts from [[Apertium stream format|stream format]] to plain text with blanks, so all modules before it must output stream format and none of the modules after it should expect stream format. Specifically:
+* Generation should be done with <code>lt-proc -b</code> or <code>hfst-proc -b</code> rather than <code>-g</code>
+* Postgeneration using <code>lt-proc -p</code> should come after
+* Postgeneration using <code>lsx-proc -p</code> should come before
+* Preferences should come before, since <code>apertium-restore-caps</code> expects only 1 surface form
+To add it, include the following in modes.xml:
+ <program name="apertium-restore-caps">
+   <file name="abc-xyz.crx.bin"/>
+ </program>
+And add this recipe to Makefile.am:
+ $(PREFIX1).crx.bin: $(BASENAME).$(PREFIX1).crx
+ 	apertium-validate-crx $<
+ 	apertium-compile-caps $< $@
+== Capitalization Restoration Rule Format ==
+The format of the CRX rules is modeled after the [[Constraint-based lexical selection|LRX]] format.
+CRX files follow this format:
+‎<syntaxhighlight lang="xml" line>
+<?xml version="1.0"?>
+<capitalization>
+  <rules>
+    <rule weight="0.1">
+      <match select="aa"/>
+    </rule>
+    <rule weight="1.0">
+      <or>
+        <match lemma="iPhone" select="dix"/>
+        <match lemma="iPod" select="dix"/>
+      </or>
+    </rule>
+    <rule weight="0.9">
+      <match tags="np.*" select="Aa"/>
+    </rule>
+  </rules>
+</capitalization>
+</syntaxhighlight>
+Rule weights may be any positive real number. If two rules give conflicting instructions for a particular word, the one with the higher weight will be used (or the higher sum, if multiple rules agree).
+=== <code>&lt;match/&gt;</code> ===
+The <code><match/></code> tag matches a single token in the input stream.
+Given an input like
+ <nowiki>[[c:AA/Aa]]^xyz<n><sg>/abc$</nowiki>
+The various pieces of this can be matched with the following attributes:
+{| class="wikitable" style="margin:auto"
+|+ Pattern attributes for <code><match/></code>
+|-
+! Attribute !! Description !! Example Input !! Example Rule !! Notes
+|-
+| <code>lemma</code> || Target language lemma || <code>xyz</code> || <code>lemma="xyz"</code> || Case-sensitive and cannot be combined with <code>trglem</code>; use <code>*</code> to get prefix or suffix match
+|-
+| <code>tags</code> || Target language tags || <code><n><sg></code> || <code>tags="n.sg"</code> || <code>?</code> matches any single tag, <code>*</code> 0 or more, and <code>+</code> 1 or more
+|-
+| <code>surface</code> || Target language surface form || <code>abc</code> || <code>surface="abc"</code> || Case-sensitive and cannot be combined with <code>trgsurf</code>; use <code>*</code> to get prefix or suffix match
+|-
+| <code>srcsurf</code> || Source language surface case || <code>AA</code> || <code>srcsurf="AA"</code> || Currently can only be <code>AA</code>, <code>Aa</code>, or <code>aa</code>
+|-
+| <code>srclem</code> || Source language dictionary case || <code>Aa</code> || <code>srcsurf="Aa"</code> || Currently can only be <code>AA</code>, <code>Aa</code>, or <code>aa</code>
+|-
+| <code>trglem</code> || Target language dictionary case || <code>aa</code> (case of <code>xyz</code>) || <code>srcsurf="aa"</code> || Cannot be combined with <code>lemma</code>
+|-
+| <code>trgsurf</code> || Target language surface case || <code>aa</code> (case of <code>abc</code>) || <code>trgsurf="aa"</code> || Cannot be combined with <code>surface</code>
+|}
+In <code>trglem</code> and <code>trgsurf</code>, <code>A</code> will match capital letters, <code>a</code> will match any other letters, <code>*</code> will match any characters at all, and <code> </code> will match spaces. All of these must match at least 1 character.
+Thus words with unusual dictionary capitalization can be matched with
+<syntaxhighlight lang="xml" line>
+<or>
+  <match trglem="aA"/>   <!-- nnnTTT (I can't think of a real example right now) -->
+  <match trglem="*aA"/>  <!-- NooJ -->
+  <match trglem="aA*"/>  <!-- iPhone -->
+  <match trglem="*aA*"/> <!-- LaTeX -->
+</or>
+</syntaxhighlight>
+Finally, the output case of a word can be determined using the <code>select</code> attribute. The allowed values are <code>dix</code> (use the output of the generator directly), <code>AA</code> (make the word all-caps), <code>Aa</code> (capitalize the word), and <code>aa</code> (lowercase the word).
+=== <code>&lt;begin/&gt;</code> ===
+The <code><begin/></code> tag matches the beginning of the input or immediately after the null character. For example, the following rule sets the first word of a sentence to uppercase:
+<syntaxhighlight lang="xml" line>
+<rule weight="1.0">
+  <or>
+    <begin/>
+    <match tags="sent"/>
+  </or>
+  <match select="Aa"/>
+</rule>
+</syntaxhighlight>
+=== <code>&lt;or&gt;</code> ===
+<syntaxhighlight lang="xml" line>
+<or>
+  <match tags="sent"/>
+  <match tags="cm"/>
+</or>
+</syntaxhighlight>
+This will match either a period or a comma.
+=== <code>&lt;repeat&gt;</code> ===
+<syntaxhighlight lang="xml" line>
+<repeat from="1" upto="3">
+  <match tags="sent"/>
+</repeat>
+</syntaxhighlight>
+This will match 1, 2, or 3 period tokens. Both <code>from</code> and <code>upto</code> are required.

Capitalization restoration - Revision history

Popcorndude: /* Restoring Capitalization */

Popcorndude: document rules

Popcorndude: start documenting new capitalization module