User:Pankajksharma/Application

Personal Information

Name: Pankaj Kumar Sharma

E-mail address: sharmapankaj1992@gmail.com

Other information that may be useful to contact:

My alternative email: pankaj@pankajksharma.com

Interest in ML and Aperitum

Why is it you are interested in machine translation?

I am interested in Machine Translation (MT) because of two reasons. The first one is little Philosophical one, i.e., the ideology of making all the digital information present openly available to everyone regardless of the language in which it's written or regardless of the language that used by the recipients. Further this would also cause in decreasing the language barrier in the exchange process of ideas.

I did my minor in Text Classification and since then become interested in Machine Learning and took me closer to NLP (a pert of MT). To be honest and I've only only used MT only as an end-user until recently.

Why is it that they are interested in the Apertium project?

I am interested in Apertium because:

• It's open source.
• Very helping community (experienced this from my interaction during project discussion).
• All the technique used in Apertium are provided as research papers (so anyone could learn from them).
• Apertium works Offline as well (:P).

Proposal

Title

Command line Fuzzy-match repair from Translation Memory

Abstract

For a given sentence S in a source language and it's translation T in another language, the idea is to find the translation of another sentence S'. The condition that S and S' must hold is that S and S' must have high Fuzzy-match score (or Low Edit Distance) between them. Then depending upon what changes from S to S' we employ (t, t') repair operations to T to get our T'.

Another phase of the project is to preprocess an existing translation memory corresponding to the source and target languages and store validated (s,t) pairs (s is a sub-sequence of S, t is a sub-sequence of T and s translates to t). These pairs could be used for generating target more better and verified (s', t') pairs.

This idea was originally given by User:mlforcada.

Project Details

Finding fuzzy match score

The project would employ following method for finding the the fuzzy match score (FMS) between S and S':

FMS(S, S') = 1 - ED(S, S') / max(|S|, |S'|)

ED(S, S') is the edit distance between S and S'. We would employ Levenshtein Distance for sentence for calculating the edit distance.

If only the value of FMS > min-fms(specified by user), the program will proceed.

Finding what changed from S to S'

To find out the changes between S and S', we would employ the phrase-extraction algorithm to extract with slight modification to obtain pairs (s, s') where s and s are sub-segments of S and S' respectively and there is some non-alignment them.

Another point worth noting is that:

min-len <= |s|,|s'| <= max-len, (min-len, max-len being specified by the user).

Translating what changed from S to S'

For this we'd be using the clipping that we created in above steps, as in (s, s') papers.

To consider the context we'd be using double validation, i.e., would match the translations in target sentence T as well and would be considering those pairs (s, t) which have following properties: s is a sub-segment of S, s contains some mismatch in S and S', t is a sub-segment of T and s translates to t.

We'll refer set of such pairs by set A.

API Call

As the project is allows you to use any scripting language I'll be using Python.

The main program would have following API:

repair.py S S' T LP [--min-fms (default 80)] [--min-len (default 3)] [--max-len (default 3)] [-r] [-s] [-h] [-d Directory]

positional arguments:

 S           Source Language Sentence

 T           Target Language Sentence

 S'          Second Source Language Sentence

 LP          Language Pair (for example 'en-eo')

optional arguments:

 -h, --help  shows help message and exit

 -d D        Specify the language-pair installation directory

 -r          Check for pairs reversibly as well

 -s          Ignore single words

 --min-fms   Minimum Fuzzy match score required to process (default value: 80%)

 --min-len   Minimum length of the sub-segments (default value: 3)

 --max-len   Maximum length of the sub-segments  (default value: 3)

Headline text

Include time needed to think, to program, to document and to disseminate.

List your skills and give evidence of your qualifications. Tell us what is your current field of study, major, etc. Convince us that you can do the work. In particular we would like to know whether you have programmed before in open-source projects.

List any non-Summer-of-Code plans you have for the Summer, especially employment, if you are applying for internships, and class-taking. Be specific about schedules and time commitments. we would like to be sure you have at least 30 free hours a week to develop for our project. No I don't have any other engagement for the Summer and would be more than happy to devote 30+ hours every week for this project.