User:Darthxaher/Application2010

Name

Abu Zaher Md. Faridee

Affiliation

Final year undergraduate student, Department of Computer Science and Engineering, Bangladesh University of Engineering and Technology

Email Address

zaher14@gmail.com

Contact Information

IRC: darthxaher@irc.freenode.net

Cell Phone: +880 1714070147

Why is it you are interested in machine translation?

As a student of Computer Science, I'm personally very interested in fields of artificial intelligence, machine learning and pattern recognition. I think machine translation is one of the most exciting applications in this field. The most interesting thing about machine translation is how fundamentally different the various machine translation techniques are. Whereas rule bases machine translation relies upon extensively on automata theory and pattern matching, statistical machine translation approach harnesses the essence of statistics and information theory. There have been extensive work in this field in the recent decades and there is much to be done, posing new challenges to developers every day.

Working on machine translation also involves the unique bonus of getting to know a lot of different languages, history and cultures, which is its own reward.

Why is it you are interested in the Apertium Project?

I successfully completed my last Google Summer of Code project (2009) titled Conversion of Anubadok: Creating an English Bengali Language Pair under Apertium. The project was a great experience for me. I had the wonderful opportunity of working with some of the experts in rule based machine translation technique. Though quite interested in working in this field, my knowledge on machine translation was not that much great. But during the course of the project I got the chance to understand the intricate details of rule based machine translation through my mentor and Apertium's helpful community. It goes without saying that Apertium's community is one of the most active open source communities out there and here I really feel at home.

One of the most exciting features of Apertium's development philosophy is its focus for endangered and less resourced languages. In a world of globalization, its hard to maintain cultural identity of minority groups. Many cultures are are on the verge of extinction. The machine translation work offered by Apertium for less resourced languages makes Apertium not only a challenging software project but also a great humanitarian effort on its own accord.

I have been long time supporter of the open source movement in my country. Adopting to open source philosophy is crucial for a developing country like Bangladesh where cost of proprietary software is unbearable for the most people. Open source machine translation that is being offered by Apertium will have far reaching effect in the local Bengali Language adoption and localization of open source softwares.

Which of the published tasks are you interested in? What do you plan to do?

I'm interested in VM for the transfer module idea, that is creating a virtual machine for the structural transfer stage in Apertium's pipeline.

As already mentioned in the idea's page, Apertium currently uses XML tree walking in the structural transfer stage. This is quite inefficient as XML parsing is quite CPU time consuming (specially in case of languages involving three stage transfer). The idea is to create a pseudo-assembly level instruction-set that embodies the rules stated in the XML files (t1x, t2x. T3x), then compile them to a easy to use byte-code format. A tiny and optimized virtual machine would need to be written to run the byte-code that will do the actual structural transfer work. Even a non JIT optimized byte-code driven VM could achieve several magnitude of performance over existing XML based solution.

Why should Google and Apertium Sponsor it?

The existing architecture of Apertium is very robust and already quite fast, but there are rooms to make it faster. The demand of faster and efficient machine translation system now is more than ever. Different machine translation systems are used everyday in the web and each day there is an ever growing load on the web servers running them. Apertium already has a highly scalable web architecture, but its performance will always be dependent on the core of Apertium's engine. The current project idea promises several magnitude of performance increase over the current implementation of the core engine by removing the bottleneck, thus freeing up more CPU time for the servers and allowing more service for the end users.

How and who will it benefit in society?

Every year European Union spend approximately one billion Euros for translating documents between the languages of the European nations. While recent advancements in machine translation field has somewhat leveraged the manual translation work, it has not reached that maturity to work without human intervention. On the other hand, most of the state of the translation systems are still closed source and costly. This makes them unsuitable for large scale deployment for this kind of government endeavour. Open source solutions like Apertium are better options as they are cost effective and flexible, and they can be readily modified to cope with ongoing demand. Every single step that increase the performance, efficiency or correctness of Apertium will have therefore long term effect its worldwide adoption and potentially save millions of dollars.

Work Plan

NOTE: just a draft, a lot fix needed, comments are always welcome

I have been keeping in touch with mentors Sergio Ortiz Rojas, Francis Tyers and Jimmy O'Regan regarding the details and plan for this project. So far I've noted the following things will need to be done:

• Define the instruction-set for the pseudo assembly

• Define the byte-code format

• Create a python prototype for the VM:

This will be the testbed for brainstorming. Primarily I think only sticking with the pre-transfer first-level transfer stage will be a wise decision.

do you mean pretransfer or first-level transfer? - Francis Tyers 17:27, 30 March 2010 (UTC)

Ah, I meant first level transfer. - Abu Zaher

By the end of this stage we should be able to generate -

• Text pseudo-assembly from the transfer XML file (first only t1x, later include to t2x and t3x)
• Byte-code from the text pseudo-assembly i.e. compile the text file
• Separate the lexical transfer module from the structural transfer module

Yes, bilingual word-by-word translation will be before structural transfer. Each word coming to the transfer will have the next format:

^sl_word/tl_word$

that is, source language lexical form/ target language lexical form - From Sergio's Mail

• Port the VM into C++:

After implementing the python prototype, we'll have a clear view of the needed data structures, instructions and byte-code format. If desired, JIT optimization techniques could be carried out in this phase of the implementation.

Community Bonding Period

As I'm already quite familiar with both the code and community of Apertium, I think I could start coding right after the community bonding period begins. That gives me a total of 16 weeks to get the project done. I would like to reserve the last 3 weeks for debugging, testing and any other emergencies. So that leaves me 13 weeks for the coding phase.

Coding Period

Week 14: July 26 - August 1

Reserved for testing, debugging and any other emergencies

Week 15: August 2 - August 8

Reserved for testing, debugging and any other emergencies

Week 16: August 9 - August 16

Reserved for testing, debugging and any other emergencies

Final Evaluation: August 9 - August 16

Final evaluation

List your skills and give evidence of your qualifications

As I've already mentioned, I successfully completed my Google Summer of Code project titled 'Conversion of Anubadok: Creating an English Bengali Language Pair' under Apertium last year. It was a really ambitious project given the fact that there was little linguistic data available for Bengali other than another open source machine translation project called Anubadok. The project had three stages, building a Bengali morphological analyser/generator, creating a English to Bengali bilingual dictionary and the creating a transfer system. Building the morphological analyser/generator proved to be tougher than we originally comprehended as Apertium needs more information for each lexical category than what was tagged in Anubadok's dataset. Therefore by the end of the project we had a morphological analyser with 68% coverage of the most used 20 thousand words. The post GSoC report discussed the outcome in further details.

The project was followed up by a successful paper submission at freeRBMT09 by me and my mentor Francis Tyers. The paper can be accessed from here. I also tried to remain active in Apertium's community after summer of code was over. Several of the maintenance enhancements were carried out long after the project was officially over.

Right now I’m in my 4th year / 2nd Semester of my undergraduate in Computer Science and Engineering in Bangladesh University of Engineering and Technology. I have attended both theoretical and practical courses in Algorithm, Automata Theory and Compiler. I took a compiler design course and as a part of term project successfully created a compiler that could run a subset of Pascal. So I believe that I have both the basic theoretical and practical knowledge for this project.

I have been an open source advocate in my country from my college years. I have been working with Ankur, a not-profit organization since then. With them I have conducted numerous open source camps. I participated in creating and beta testing of several products from Ankur, notably Firefox spell-checking dictionary and off-line add-on CD for several localized Ubuntu versions.

I’m the developer of several open source applications. Netaccess-squid has been created as an open source alternative to Cyberoam System. Aural Aurora is a Spring/Oracle based music discovery and social collaboration system.

I had worked as a part time software developer for AfriGISBD, an off-shore development house of AfriGIS 2 years. I mainly worked in high level languages like python, php, javaEE there and had to do a lot of system-admin on Linux, Solaris and Mac OS X. Until last year I waas also working at MuktoSoft (part-time) which involved iPhone/iPod softwares.

I maintain a public blog here. Although its not a day-to-day blog, I try to keep it updated when I get free time with the interesting technical things I come across.

My resume can be viwed from here.

List any non-Summer-of-Code plans you have for the Summer

I don’t have any other plans beside Google Summer of Code this summer. However, my class schedule do overlap with some parts of GSoC’s schedule but I think it won’t conflict with the work plan. I could always do extra work in the weekend to minimize the overlapping.

Conclusion

I’d like to thank all the developers of Apertium for putting up such a great effort. I’d also like to thank Google for organizing Google Summer of Code and flourishing Open Source community.