Apertium on Windows

hic·sunt·dracones
Here be dragons! To make improvements, please dive right in. It is highly recommended that you hang out on the IRC channel (irc.oftc.net #apertium) and join the mailing list.

Introductory notes

Apertium is Unix software (this includes Mac OS/X). It's not formally supported on Windows - though we would all like it to run on Windows, it's not supported. It seems that Apertium is a difficult beast to get working, which is odd for us, because in a unix environment, it "just works".

Cygwin, as a 'Unix-like' environment for Windows, should be an easy target for us; however, Apertium makes heavy use of wstring in its C++ code, which is not supported by Windows versions of GCC earlier than 4.0.

Lttoolbox builds with Visual C++ without additional changes - provided that you follow the instructions in README.Win32.

You will also need a Unix-type shell, such as bash, to run it. Batch files simply will not work, as Windows does not have true support for program pipes ('|').

Getting the code

The lttoolbox changes are in the main SVN branch. Current win32 changes for apertium are in the branch apertium/win32

See instructions for using SVN, or the simplest way:

• Download and install TortoiseSVN
• Get a login/password for SourceForge at http://sourceforge.net/
• Create a directory C:\Apertium, and right click/TortoiseSVN/Import... in it
• Specify the URL of the repository: http://apertium.svn.sourceforge.net/svnroot/apertium and click OK
• Use the source code in the directory "trunk" (not "branches")

Dependencies

You will need to get Windows versions of the following tools and libraries to compile the code under Windows :

• Microsoft's C++ compiler - the compiler is part of Visual Studio 2008 Express, which is available free of charge.
• CMake - a cross platform build tool, download cmake-2.6.4-win32-x86.exe, at installation make sure you have CMakeSetup.exe in the bin directory.
• libxml - the Windows binaries are at http://xmlsoft.org/sources/win32/; you will need:
  • libxml2 (install it in C:\Program Files\LibXML) from libxml2-2.7.6.win32.zip
  • zlib (install it in C:\Program Files\zlib) from zlib-1.2.3.win32.zip
  • iconv (install it in C:\Program Files\iconv) from iconv-1.9.2.win32.zip
  • libxslt (install it in C:\Program Files\libxslt) from libxslt-1.1.26.win32.zip
• pcre - Download pcre-7.9.zip and unzip it (in C:\pcre for instance), next read Building and installing pcre below.
  • KDE/Win32 have packages for MinGW/MSVC here -- I haven't checked them yet
• flex for Windows - download and use the Windows installer version (http://sourceforge.net/projects/gnuwin32/files/flex/).

Insert the zlib bin directory in the path variable, for information it is needed by xsltproc.exe.

Setting up the required tools and libraries

Installing Visual Studio

Be sure to start downloading Visual Studio 2008 first, since it is a fairly big download.

Installing CMake

Next, install the Windows version of CMake. Let the installer place CMake in your path, since it will be much easier to run CMake from the command line if you do this.

Installing the libxml libraries

The zip files of libxml2, zlib, iconv and libxslt all have the familiar UNIX layout:

|- bin
|- include
|- lib
|- share

Unzip each of these archives and copy the bin, include, lib and share directories in each package into a common directory (C:\Program Files\LibXML by default).

Building and installing pcre

You have unziped the pcre archive to a directory C:\pcre-7.9 (likely), create a diretory C:\pcre\build (for instance).

Edit the file CMakeLists.txt and update the following by replacing OFF by ON:

SET(PCRE_SUPPORT_UTF8 OFF CACHE BOOL
    "Enable support for the Unicode UTF-8 encoding.")

To compile it, you will need to have Visual Studio AND CMake already installed on your system. Open the Visual Studio 8.0 Command Prompt (Start -> Microsoft Visual C++ 2008 Express Edition -> Visual Studio Tools -> Visual Studio 2008 Command Prompt). In this command prompt, go to the pcre source directory and run:

cmakesetup .

Set the source code location (C:\pcre-7.9) and the build location (C:\pcre-7.9\build). Click on "Configure" and select "NMake Makefiles", CMake will show you a number of build variables, all highlighted in red, click "Configure" once more, if everything goes well the "OK" button will become active, hit it: as a result your "build" directory will be populated with a number of files for compiling the software.

To compile pcre, go to your build directory (C:\pcre-7.9\build) and run:

nmake

When this is done, copy the file pcred.lib in your final pcre directory (C:\Program Files\pcre\lib for instance).

Installing flex

If you have downloaded the binary archive, unzip them into something like C:\Program Files\Flex. If you downloaded the Windows installer, then just follow the installation wizard.

Building lttoolbox

Before you can build lttoolbox, you need to tell CMake where to find the libxml files (under Linux, CMake will use pkg-config to find where the packages are installed). Open lttoolbox\CMakeLists.txt and modify the lines

SET (LIBXML2_BASE        "C:/Program\ Files/LibXML")
SET (LIBXML2_INCLUDE_DIR "${LIBXML2_BASE}/include")
SET (LIBXML2_LIBRARIES   "${LIBXML2_BASE}/lib/libxml2.lib")

to match your installation.

Now, in the Visual Studio command prompt (again, Start -> Microsoft Visual C++ 2008 Express Edition -> Visual Studio Tools -> Visual Studio 2008 Command Prompt) change the directory to lttoolbox and execute

cmakesetup .

Click on the configure button. As with libpcre, CMake will ask you to choose the type of build file to generate. Again, choose "Nmake". If you get an error message about LIBXML, then it means that LIBXML2_INCLUDE_DIR or LIBXML2_LIBRARIES in lttoolbox\CMakeLists.txt has an incorrect value.

If everything completes without errors after the first time that you clicked the "Configure" button, you will again see a few lines in the appear in the cmakesetup window. They should be highlighted with red. You have the opportunity to change the installation directory of lttoolbox by modifying the build variable CMAKE_INSTALL_PREFIX. To continue, click a second time on "Configure"; if everything is correct, then everything that was highlighted with red should now be highlighted with gray and the OK button becomes active.

Next, click OK; CMake should generate NMake files. To compile, simply execute:

nmake

To install lttoolbox, run:

nmake install

Building apertium

In order to build apertium, CMake needs to use the following tools and libraries:

• libxml
• libpcre
• lttoolbox
• xsltproc (you should have this since it is distributed with libxslt)
• flex

At this point you should have all of the above and you simply need to set the correct CMake variables in order to build apertium. Open apertium/CMakeLists.txt (from the trunk directory) and find the section which should resemble the following:

IF (WIN32)
        SET (WIN32_DIR ${PROJECT_SOURCE_DIR}/apertium/win32)
        INCLUDE_DIRECTORIES (${WIN32_DIR})
        LIST (APPEND EXTRA_SOURCES ${WIN32_DIR}/getopt.c ${WIN32_DIR}/libgen.c)

        SET(FLEX_EXECUTABLE        "${PROGRAM_FILES_PATH}/GnuWin32/bin/flex.exe")

        SET(LIBXML2_BASE_DIR       "${PROGRAM_FILES_PATH}/LibXML")
        SET(LIBXML2_INCLUDE_DIR    "${LIBXML2_BASE_DIR}/include")
        SET(LIBXML2_LIBRARIES      "${LIBXML2_BASE_DIR}/lib/libxml2.lib")
        
        SET(LIBXSLT_BASE_DIR       "${PROGRAM_FILES_PATH}/libxslt")
        SET(LIBXSLT_INCLUDE_DIR    "${LIBXSLT_BASE_DIR}/include")
        SET(LIBXSLT_LIBRARIES      "${LIBXSLT_BASE_DIR}/lib/libxslt.lib")
        SET(XSLTPROC_EXECUTABLE    "${LIBXSLT_BASE_DIR}/bin/xsltproc.exe")

        SET(LTTOOLBOX3_BASE_DIR    "${PROGRAM_FILES_PATH}/apertium-3.1")
        SET(LTTOOLBOX3_INCLUDE_DIR "${LTTOOLBOX3_BASE_DIR}/include/lttoolbox-3.1")
        SET(LTTOOLBOX3_LIBRARIES   "${LTTOOLBOX3_BASE_DIR}/lib/lttoolbox3.lib")

        SET(LIBPCRE_BASE_DIR       "${PROGRAM_FILES_PATH}/pcre")
        SET(LIBPCRE_INCLUDE_DIR    "${LIBPCRE_BASE_DIR}/include")
        SET(LIBPCRE_LIBRARIES      "${LIBPCRE_BASE_DIR}/lib/pcred.lib")

        ADD_DEFINITIONS (/D _CRT_SECURE_NO_WARNINGS /D STDC_HEADERS /D PCRE_STATIC)
ENDIF (WIN32)

The only variables you may need to adapt are:

• PROGRAM_FILES_PATH (at top of apertium/CMakeLists.txt)
• FLEX_EXECUTABLE
• LIBXML2_BASE_DIR
• LIBXSLT_BASE_DIR
• LTTOOLBOX3_BASE_DIR
• LIBPCRE_BASE_DIR

These variables point to the base directories where the respective required tools and libraries are installed.

To build apertium, open the Visual Studio command prompt (as described earlier in this document) and change the directory to apertium. Now as before, run :

cmakesetup .

Follow all of the build steps as described in the previous section which details how lttoolbox is built.

After the build completes, you should be able to install apertium by typing :

nmake install

Testing apertium

Go to the directory occitan-catalan directory (likely to be C:\Apertium\trunk\apertium-oc-ca) and do the following tests:

lt-comp lr apertium-oc-ca.ca.dix ca-oc.automorf.bin

expected results:

apostrophes@postblank 1001 1543
final@inconditional 28 537
main@standard 36360 66910

try now:

echo "Això és una prova" | lt-proc ca-oc.automorf.bin 

expected results:

^Això/Això<prn><tn><p3><nt>$ ^és/ser<vbser><pri><p3><sg>$ ^una/un<num><f><sp>/un<det><ind><f><sg>$ ^prova/prova<n><f><sg>/provar<vblex><imp><p2><sg>/provar<vblex><pri><p3><sg>$

next do:

echo "Això és una prova" | lt-proc ca-oc.automorf.bin  | apertium-tagger -g ca-oc.prob 

expected results:

^Això<prn><tn><p3><nt>$ ^ser<vbser><pri><p3><sg>$ ^un<det><ind><f><sg>$ ^prova<n><f><sg>$