Apertium-html-tools

Apertium Html-tools is a web application providing a fully localised interface for text/document/website translation, analysis, and generation powered by Apertium. Designed to use only static resources for quick and easy deployment with any web server, it is relatively lightweight and user-friendly. It is currently located on Github at github.com/apertium/apertium-html-tools. Html-tools relies on an Apertium HTTP API such as Apertium-apy or ScaleMT (to a lesser extent).

Usage

Getting Started

Using Html-tools is as easy as cloning from git, changing the configuration and running yarn.

$ git clone https://github.com/apertium/apertium-html-tools.git 
$ cd apertium-html-tools/ 
$ vim ./config.ts
$ yarnpkg install
$ yarnpkg build

The last step is pointing your web server of choice, e.g. Apache, to serve the directory dist or running python3 -m http.server in /build.

Configuration

Since Html-tools is powered entirely by static resources, a configuration file generated at apertium-html-tools/config.conf (based on config.conf.example) is maintained to allow users to modify basic environment parameters such as API location and enabled interfaces. Following is a summary of the currently supported arguments with some examples:

• APY_URL: the URL of an API instance (ScaleMT for basic functionality, APY for full) that the web application calls
  • https://www.apertium.org/apy
  • http://localhost:2737
  • http://turkic.apertium.org:8080
• ALLOWED_LANGS: an array of ISO 639-3 language codes that the application displays in the interface, useful for deploying websites that have only a subset of all the languages the API being used provides (e.g. a region specific website)
  • does not limit the languages displayed (everything the API supports will be shown).
  • tur, uzb, kaz'] limits to only language pairs between Turkish, Kazakh, and Uzbek appearing in the interface; analyzers and generators for the three languages will also be shown.
• ALLOWED_VARIANTS: an array of variant codes that the application displays in the interface, similar to the functionality of ALLOWED_LANGS
  • does not limit the variants displayed (everything the API supports will be shown).
  • US, GB limits languages displayed in the translation interface such that pairs either have no variant or a subset of {'US', 'GB'} (en_US-es and es-en_GB are fine but en_UK-anything is not); analyzers and generators are similarly restricted.
• ENABLED_MODES: an array of the enabled interfaces, a non-empty subset of ['translation', 'analyzation', 'generation', 'sandbox']
  • translation limits the interface to only translation support.
  • translation, analyzation, generation displays the three main interfaces as tabs.
  • translation, analyzation, generation, sandbox is generally not recommended outside of a development environment since users most probably do not need an API Sandbox.
• DEFAULT_MODE: the default landing interface for a new user, an element of the ENABLED_MODES array
  • translation lets the user land on the translation interface first (recommended option).
• SHOW_NAVBAR: whether the navigation bar that allows switching between interfaces is visible.
  • True shows the navbar and is recommended when more than one mode is enabled via ENABLED_MODES.
  • False hides the navbar and is recommended when only one mode is enabled (e.g. ENABLED_MODES = translation).
• PIWIK_SITEID: the Piwik site ID
  • undefined disables Piwik Analytics (recommended unless you know what you're doing).
• PIWIK_URL: the Piwik instance URL
  • undefined disables Piwik Analytics (recommended unless you know what you're doing).
• LIST_REQUEST_CACHE_EXPIRY: cache expiry time in hours for /list requests (/listPairs, /list?q=analyzers, /list?q=generators)
  • 24 ensures list requests are refreshed at least every day.
• LANGUAGE_NAME_CACHE_EXPIRY: cache expiry time in hours for /languageName requests
  • 24 ensures language name caches are refreshed at least very 24 hours.
• LOCALIZATION_CACHE_EXPIRY: cache expiry time in hours for localised string asset requests (e.g. /assets/strings/fra.json)
  • 24 ensures the interface strings for a locale are refreshed at least every day (browser caching may interfere).
• AVAILABLE_LOCALES_CACHE_EXPIRY: cache expiry time in hours for available locales
  • 24 ensures the available locales are refreshed at least every day (browser caching may interfere).

Minification

is currently not done, since enabling gzip compression on the server gives almost-as-good results with easier debuggability, less build complexity and less dependencies. (The old minification build rules are in SVN revision 52413.) Files are still named min.css/min.js even though they're not minified.

However, JS and CSS files are concatenated on typing make, which makes for fewer requests.

Production

In a production environment, make will generate all the necessary files including the build/index.html linking to the concatenated JS and CSS. So, see the instructions in the above section and you should be set for deployment.

Development

Since running make generates an index.html with links to concatenated content, development is a little bit more complicated. After making changes to any CSS or JS resource and before re-running make, the changes will not be reflected in / since index.html still links to the stale minified CSS or JS resources. While this could be amended by running make after every change, that's a terrible workflow unless it's automated somehow (e.g. in emacs you would have to put (add-hook 'after-save-hook 'compile) in ~/.emacs). To alleviate this problem, running make also generates index.debug.html which links to the raw CSS/JS resources that you modify. So, use /index.debug.html while debugging/adding features in the CSS or JS. Modifying the HTML does require running make every time but such changes are less frequent and usually involve less debugging. If you add a static resource (e.g. a new JS file for a spell checker interface), be sure to edit the Makefile and debug-head.html to ensure the concatenation process still runs smoothly. The commands below might help with auto-making after changes:

while true; do inotifywait . -r -e MODIFY && make debug -j32; done;      # Linux
while true; do fswatch -o -r . | make debug -j32; done;                  # OS X

Current Status

Features

Translation

Text translation is supported through an intuitive interface with support for instant translation (translation as you type) and support for disabling the marking of unknown tokens. Language detection is supported with CLD with plans to migrate to a solution that supports all the Apertium languages. Document translation is available for almost all modern browsers and supports less than 32MB files in DOCX, XLSX, PPTX, ODT, TXT, RTF, HTML, and LATEX formats with plans to support older Microsoft Office formats (DOC, XLS, PPT).

Morphological Analysis and Generation

Morphological analysis and generation are supported via two language selectors, one for the language desired (e.g. English) and another for the Apertium package used (e.g. English-Spanish or English-Esperanto).

Theming / Customisation

Html-tools uses Bootstrap as its frontend framework and supports using any of the Bootstrap themes provided by Bootswatch through the Bootstrap CDN. Currently, there are sixteen available themes (Cerulean, Cosmo, Cyborg, Darkly, Flatly, Journal, Lumen, Paper, Readable, Sandstone, Simplex, Slate, Spacelab, Superhero, United and Yeti), based on bootstrap themes, in addition to the default theme. To use the Cosmo theme,

   $ make cosmo -B

To switch back to the default theme,

   $ make -B

By running make, the theme's Bootstrap CSS is downloaded from the CDN. It is then combined with the default styles used on the page (located in /assets/css/style.css). However, sometimes you may want to override the rules set by these two files. For that reason, each theme also has a custom CSS file which contains rules that adapt existing elements to the theme or override the theme's CSS itself. For example, /assets/css/themes/style.lumen.css contains a rule that switches the default Apertium logo with white letters to the one with black lettering. If desired, it can also be used to do things like changing the colors that Bootstrap sets, overriding their default values.

Deployed Examples

There are currently two deployed instances of Apertium-apy and Apertium-html tools

Apertium

Apertium maintains html-tools at www.apertium.org and APY at https://www.apertium.org/apy/ with support for most of the current trunk pairs. However, since it only contains the translation interface, it is not a good source of context for contributors looking to localise the interface.

Apertium Turkic

Apertium Turkic maintains html-tools at turkic.apertium.org and APY at turkic.apertium.org:8080 with support for several Turkic languages and the translation, analyzation, and generation modes available. As such, it is a good candidate for getting localisation string context.

Future Improvements

Localisation

The Html-tools interface is fully localisable and has support for RTL languages. The 26 currently available languages are Aragonese (arg), Avar (ava), Catalan (cat), English (eng), Basque (eus), French (fra), German (deu), Hebrew (heb), Karakalpak (kaa), Kazakh (kaz), Kyrgyz (kir), Nynorsk Norwegian (nno), Bokmål Norwegian (nob), Occitan (oci), Portuguese (por), Romanian (ron), Saami (sme), Spanish (spa), Tatar (tat), Thai (tha), Turkish (tur), Uzbek (uzb), Uyghur (uig), and Chinese (zho).

Contributing a new translation of the interface is encouraged and a very simple process.

Quick and easy

If you don't know what git is, or you'd rather not deal with it, follow these steps.

1. Download the eng.json file from this directory.
2. Download the file for your language too. If one doesn't exist, make a copy of the eng.json file for your language. You could also download the file for a language similar to your language or one that you're more comfortable with.
3. Base your translations off the eng.json file, as the English strings file is the only one guaranteed to be up-to-date.
4. Send us your finished translation! Or let us know if you have questions. Thanks!

Creating a new translation

1. Clone the apertium-html-tools repo, which has all the localisation files in apertium-html-tools: git clone git@github.com:apertium/apertium-html-tools.git.
2. Look at a deployed version of the html-tools interface, preferably one with all/most modes enabled. See the section on deployed sites for where to go.
3. • Starting from an existing translation (with reference)
   1. From the assets/strings directory, run ./localisation-tools.py new fin -p eng, replacing fin with your language's ISO 639-3 code and eng with the language you want to use as a reference while translating.
   2. Replace all the right-hand values in each pair to match your language, use the website you found from Step 2 and the reference language translations to help with context.
   • Starting from scratch (without reference)
   1. Run ./localisation-tools.py new fin -b, replacing fin with your language's ISO 639-3 code.
   2. Replace all the right-hand values in each pair to match your language, use the website you found from Step 2 to help with context.
4. Run ./localisation-tools.py all fin, replacing fin with your language's ISO 639-3 code and see whether completion is 100 (percent). If not, keep translating (what you're missing is shown in missing) and running this step's command.
5. If you didn't manage to run localisation-tools.py because of an exception, you might have made an error in the json-file. You can check your JSON with http://jsonlint.com/ or similar.
6. Optional: Add your name to the "authors" key and update the "last-updated" key. Add a line to assets/strings/locales.json with the language code mapped to the language's native name (endonym).
7. Commit your changes to git: git commit ./ -m "Your message..."
8. If you didn't edit locales.json, let us know in IRC so that we can edit it for you and update websites. Thanks!

Updating an existing translation

1. Clone the apertium-html-tools repo, which has all the localisation files in apertium-html-tools: git clone git@github.com:apertium/apertium-html-tools.git.

1. Look at a deployed version of the html-tools interface, preferably one with all/most modes enabled. See the section on deployed sites for where to go.
2. • Working with a reference
   1. From the assets/strings directory, run ./localisation-tools.py all fin -p eng, replacing fin with your language's ISO 639-3 code and eng with the language you want to use as a reference while translating.
   2. Change all the right-hand values in each pair that are marked as %%UNAVAILABLE%% to match your language, use the website you found from Step 2 and the reference language translations to help with context.
   • Working without a reference
   1. Run ./localisation-tools-py all fin -b, replacing fin with your language's ISO 639-3 code.
   2. Change all the right-hand values in each pair that are marked as %%UNAVAILABLE%% to match your language, use the website you found from Step 2 to help with context.
3. Run ./localisation-tools.py all fin, replacing fin with your language's ISO 639-3 code and see whether completion is 100 (percent). If not, keep translating (what you're missing is shown in missing) and running this step's command.
4. Optional: Add your name to the "authors" key (if it's not there already) and update the "last-updated" key.
5. Commit your changes to git: git commit ./ -m "Your message..."

Progress

Updated 8 January 2017. Generated by progresstable.sh.

code	CBE*	CBC**
eng	100%	100.00%
arg	98%	109.85%
uzb	98%	108.64%
swe	98%	102.15%
tur	98%	100.08%
rus	98%	97.09%
deu	96%	109.07%
nno	96%	95.95%
cat	94%	104.96%
kaa	94%	103.19%
nob	94%	96.43%
tha	94%	89.29%
fin	94%	88.98%
spa	86%	35.01%
kir	84%	33.11%
fra	82%	102.76%
oci	82%	99.34%
uig	82%	91.87%
kaz	82%	91.19%
heb	82%	80.42%
tat	69%	19.68%
sme	65%	21.78%
por	61%	19.55%
eus	57%	21.91%
ava	55%	18.54%
ron	50%	17.27%
zho	50%	5.04%

 *CBE: completion by entries
 **CBC: completion by characters (i.e., ratio of characters to English ~source)

Docs

• /Localisation

Credits

Apertium-html-tools grew out of simple-html as features were being added by Sushain Cherivirala during GCI 2013. It's now its own beast, and continues to be maintained primarily by Sushain.

The following people have helped with localisation:

• Unhammer (Nynorsk Norwegian, Bokmål Norwegian)
• Lene Antonsen (Northern Saami)
• firespeaker (Kyrgyz, Kazakh, Russian, Tatar)
• spectie (Spanish, Catalan, Aragonese)
• Ilnar (Russian, Tatar)
• Sushain (Spanish)
• Aida (Kazakh)
• hbwashington (Portuguese, Spanish)
• solinus (Avar, Russian)
• shopskasalata/kikero (Romanian)
• Cédric VALMARY (French)
• Lene Antonsen (Saami)
• Tolgonay (Kyrgyz)
• wei2912 (Chinese)
• Märdan (Uyghur)
• Flammie (Finnish)
• inariksit (Finnish)
• youssef (French)
• n0nick (Hebrew)
• vmax = Max Vorobev (Russian)
• bmansurov (Uzbek)
• Çağrı (Turkish)

Please cite

• https://www.aclweb.org/anthology/W18-2207/