User:Darthxaher/Application2010

Name

Abu Zaher Md. Faridee

Affiliation

Final year undergraduate student, Department of Computer Science and Engineering, Bangladesh University of Engineering and Technology

Email Address

zaher14@gmail.com

Contact Information

IRC: darthxaher@irc.freenode.net

Cell Phone: +880 1714070147

Why is it you are interested in machine translation?

As a student of Computer Science, I'm personally very interested in the fields of artificial intelligence, machine learning and pattern recognition. I think machine translation is one of the most exciting applications in this field. The most interesting thing about machine translation is how fundamentally different the various machine translation techniques are. Whereas rule bases machine translation relies upon extensively on automata theory and pattern matching, statistical machine translation approach harnesses the essence of statistics and information theory. There have been extensive work in this field in the recent decades and there is much to be done, posing new challenges to developers every day.

Working on machine translation also involves the unique bonus of getting to know a lot of different languages, history and cultures, which is its own reward.

Why is it you are interested in the Apertium Project?

I successfully completed my last Google Summer of Code project (2009) titled Conversion of Anubadok: Creating an English Bengali Language Pair under Apertium. The project was a great experience for me. I had the wonderful opportunity of working with some of the experts in rule based machine translation technique. Though quite interested in working in this field, my knowledge on machine translation was not that much great. But during the course of the project I got the chance to understand the intricate details of rule based machine translation through my mentor and Apertium's helpful community. It goes without saying that Apertium's community is one of the most active open source communities out there and here I really feel at home.

One of the most exciting features of Apertium's development philosophy is its focus for endangered and less resourced languages. In a world of globalization, its hard to maintain cultural identity of minority groups. Many cultures are are on the verge of extinction. The machine translation work offered by Apertium for less resourced languages makes Apertium not only a challenging software project but also a great humanitarian effort on its own accord.

I have been long time supporter of the open source movement in my country. Adopting to open source philosophy is crucial for a developing country like Bangladesh where cost of proprietary software is unbearable for the most people. Open source machine translation that is being offered by Apertium will have far reaching effect in the local Bengali Language adoption and localization of open source softwares.

Which of the published tasks are you interested in? What do you plan to do?

I'm interested in VM for the transfer module idea, that is creating a virtual machine for the structural transfer stage in Apertium's pipeline.

As already mentioned in the idea's page, Apertium currently uses XML tree walking in the structural transfer stage. This is quite inefficient as XML parsing is quite CPU time consuming (specially in case of languages involving three stage transfer). The idea is to create a pseudo-assembly level instruction-set that embodies the rules stated in the XML files (t1x, t2x. T3x), then compile them to a easy to use byte-code format. A tiny and optimized virtual machine would need to be written to run the byte-code that will do the actual structural transfer work. Even a non JIT optimized byte-code driven VM could achieve several magnitude of performance over existing XML based solution.

Why should Google and Apertium Sponsor it?

The existing architecture of Apertium is very robust and already quite fast, but there are rooms to make it faster. The demand of faster and efficient machine translation system now is more than ever. Different machine translation systems are used everyday in the web and each day there is an ever growing load on the web servers running them. Apertium already has a highly scalable web architecture, but its performance will always be dependent on the core of Apertium's engine. The current project idea promises several magnitude of performance increase over the current implementation of the core engine by removing the bottleneck, thus freeing up more CPU time for the servers and allowing more service for the end users.

How and who will it benefit in society?

Every year European Union spend approximately one billion Euros for translating documents between the languages of the European nations. While recent advancements in machine translation field has somewhat leveraged the manual translation work, it has not reached that maturity to work without human intervention. On the other hand, most of the state of the translation systems are still closed source and costly. This makes them unsuitable for large scale deployment for this kind of government endeavour. Open source solutions like Apertium are better options as they are cost effective and flexible, and they can be readily modified to cope with ongoing demand. Every single step that increase the performance, efficiency or correctness of Apertium will have therefore long term effect its worldwide adoption and potentially save millions of dollars.

Work Plan

I have been keeping in touch with mentors Sergio Ortiz Rojas, Francis Tyers and Jimmy O'Regan regarding the details and plan for this project. So far I've noted the following things will need to be done:

• Separate the lexical transfer module from the structural transfer module without breaking the existing system. This part is necessary because the VM will take word pairs [source language, target language] as input, so we need something that can feed the VM that data. (Note: I have been notified by mentor Ortiz Rojas that he will take care of this part)

• Define the instruction-set and data structure for the pseudo assembly

• Define the byte-code format

• Create a python prototype for the stack based virtual machine:

This will be the test-bed for brainstorming and R&D. Primarily, I think only sticking with the first-level transfer stage will be a wise decision.

By the end of this stage we should be able to generate -

• Text pseudo-assembly from the transfer XML file (first only t1x, later include to t2x and t3x)
• Byte-code from the text pseudo-assembly i.e. compile the text file

• Port the VM into C++:

After implementing the python prototype, we'll have a clear view of the needed data structures, instructions and byte-code format. If desired, JIT optimization techniques could be carried out in later part this phase.

Community Bonding Period

As I'm already quite familiar with both the code and community of Apertium, I think I could start coding right after the community bonding period begins. That gives me a total of 16 weeks to get the project done. I'd like to use only the first week for reviewing the code structure of Apertium and Lt-toolbox and reserve the last 3 weeks for debugging, polishing, testing and any other emergencies. So that leaves me with 12 weeks for the coding phase.

Week 1: April 27 - May 2

• Reviewing of the code structure of Apertium and Lt-toolbox that involves the lexical transfer and structural transfer module. Also come up with some set of test cases (at first, small t1x files and accompanying input) with increasing order of complexity (include t2x and t3x) for a preferred language pair. These test cases will be the the basis of how we'll keep adding more and more complex instructions in the python prototype. Previous works from Jacob Nordfalk will be an excellent place to start.

Deliverable: Set of test cases

Week 2: May 3 - May 9

Define a very minimal set of instructions and data structure for the prototype VM, try to come up with a simple as possible schematics for the python implementation.

Deliverable: A minimal instruction set and data structure + schematics for the python VM

Week 3: May 10 - May 16

Start preliminary work on 't1x' to 'pseudo-assembly text' compiler.

Deliverable: A minimal compiler that can compile from 't1x' to 'pseudo-assembly' text

Week 4: May 17 - May 23

Start coding on the python prototype VM. It will be able to run the instructions from 't1x' pseudo-assembly text file and produce the desired structurally translated output.

Deliverable: A very very simple and working prototype.

Official Coding Period

Week 5: May 24 - May 30

Start working in t2x and t3x to pseudo-assembly compiler. This will involve adding more instructions to our instruction set.

Week 6: May 31 - June 6

Modify the VM so that now it can run t2x and t3x pseudo assembly files and produce desired output.

Deliverable: A functional compiler and VM that can handle t1x, t2x, t3x files.

Week 7: June 7 - June 13

By this time the interpreted VM would be in a workable state. So now we add 'txx' to 'byte-code' generation mode to our compiler.

Deliverable: The python implementation of the byte-code compile

Week 8: June 14 - June 20

Start adding byte-code aware mode to our VM, fix any remaining issues in the 'txx' to 'byte-code' compiler. Test this new module.

Week 9: June 21 - June 27

Finish working on the byte-code mode VM, also explore the possibilities of JIT optimization for the VM.

Deliverable: Fully working python prototype

Week 10: June 28 - July 24

Start primary work on the C++ implementation, focus of this implementation is fast and reliable VM and a compiler that is also binary compatible with the previously created python implementation.

Deliverable: Reimplementation of the python 'txx' to byte-code compiler in C++

Week 11: July 5 - July 11

Start working on porting the python VM into C++

Week 12: July 12 - July 18 (Mid-term Evaluation)

Final work on C++ VM

Deliverable: A fully functional C++ implementation of VM

Week 13: July 19 - July 25

Try to cut the loose ends of the C++ VM and explore the possibilities of JIT optimizations

Deliverable: A JIT optimized (if possible) VM

Week 14: July 26 - August 1

Reserved for testing, debugging and any other emergencies

Week 15: August 2 - August 8

Reserved for testing, debugging and any other emergencies

Week 16: August 9 - August 16

Reserved for testing, debugging and any other emergencies

Final Evaluation: August 9 - August 16

Final evaluation

Development Note

From my previous GSoC experience I have realized that a lot of unexpected problems arise during the course of the project which are not part of the original plan. In those cases I'll try to modify the project plan to fit our needs but try to maintain the deadlines as much as possible.

List your skills and give evidence of your qualifications

As I've already mentioned, I successfully completed my Google Summer of Code project titled Conversion of Anubadok: Creating an English Bengali Language Pair under Apertium last year. It was a really ambitious project given the fact that there was little linguistic data available for Bengali other than another open source machine translation project called Anubadok. The project had three stages, building a Bengali morphological analyser/generator, creating a English to Bengali bilingual dictionary and the creating a transfer system. Building the morphological analyser/generator proved to be tougher than we originally comprehended as Apertium needs more information for each lexical category than what was tagged in Anubadok's dataset. Therefore by the end of the project we had a morphological analyser with 68% coverage of the most used 20 thousand words. The post GSoC report discussed the outcome in further details.

The project was followed up by a successful paper submission at freeRBMT09 by me and my mentor Francis Tyers. The paper can be accessed from here. I also tried to remain active in Apertium's community after summer of code. Several of the maintenance enhancements were carried out long after the project was officially over.

Right now I’m in my final year of my undergraduate in Computer Science and Engineering in Bangladesh University of Engineering and Technology. I have attended both theoretical and practical courses in Algorithm, Automata Theory and Compiler. I took a compiler design course and as a part of term project successfully created a compiler that could run a subset of Pascal. I also attended basic OS course, where I had to go through Linux's source code implement a custom system call. implement some new features in 'Nachos' and solve some classic concurrency problem using POSIX thread. So I believe that I have both the basic theoretical and practical knowledge for this project.

I have been an open source advocate in my country from my college years. I have been working with Ankur, a not-profit organization since then. With them I have conducted numerous open source camps. I participated in creating and beta testing of several products from Ankur, notably Firefox spell-checking dictionary and off-line add-on CD for several localized Ubuntu versions.

I’m the developer of several open source applications. Netaccess-squid has been created as an open source alternative to Cyberoam System. Aural Aurora is a Spring/Oracle based music discovery and social collaboration system.

I had worked as a part time software developer for AfriGISBD, an off-shore development house of AfriGIS 2 years. I mainly worked in high level languages like python, php, javaEE there and had to do a lot of system-admin on Linux, Solaris and Mac OS X. Until last year I was also working at MuktoSoft (part-time) which involved iPhone/iPod softwares.

I maintain a public blog here. Although its not a day-to-day blog, I try to keep it updated when I get free time with the interesting technical things I come across.

My resume can be viwed from here.

List any non-Summer-of-Code plans you have for the Summer

Right now, as I write this application, I'm attending my first semester final exams at university. My exams will be over in April 14 and my next semester does not start until July. So after this exam is over, I will be able to fully concentrate on my work of the project. My class schedules will overlap at the ending phase of the project, but normally there is not much pressure at the beginning of the semester. Keeping all these in mind, I plan to finish the project up as soon as possible while I can give it full attention and at the later stage do bug fixing and enhancements. And if need arises, I could always do extra work in the weekend to minimize the effect of overlapping.

Conclusion

I’d like to thank all the developers of Apertium for putting up such a great effort. I’d also like to thank Google for organizing Google Summer of Code and flourishing Open Source community.