User:Iamas/GSoC13 Application: "Improved Bilingual Dictionary Induction" - Revision history

Iamas: /* Coding Challenge */

2013-05-03T17:50:12Z

Coding Challenge

Iamas: /* Biography */

2013-05-03T17:46:08Z

Biography

Iamas: /* Week Plan */

2013-05-03T16:11:30Z

Week Plan

Iamas: /* Week Plan */

2013-05-03T16:10:39Z

Week Plan

Iamas: /* Week Plan */

2013-05-03T16:09:29Z

Week Plan

Iamas: /* Week Plan */

2013-05-03T16:08:12Z

Week Plan

Iamas: /* Week Plan */

2013-05-03T16:06:19Z

Week Plan

Iamas: /* Which of the published tasks am I interested in? What do I plan to do? */

2013-05-03T15:46:40Z

Which of the published tasks am I interested in? What do I plan to do?

Iamas: /* Week Plan */

2013-05-03T09:56:46Z

Week Plan

Iamas: /* Week Plan */

2013-05-02T22:11:44Z

Week Plan

@@ Line 41: / Line 41: @@
 * Link can be found on github [https://github.com/arnavsharma93/CodingChallengeApertium here].
 * Please refer to the [https://github.com/arnavsharma93/CodingChallengeApertium/blob/master/README.md README] for further details.
 === Interim period and community bonding period ===

@@ Line 103: / Line 103: @@
 == Biography ==
 I am currently pursuing Bachelor of Technology in Computer Science and MS by Research in Computational Linguistics at IIIT-H. I have just finished my second year in that. I have been studying the various fields of Computational Linguistics for the past two years and I can not wait to study more. I am proficient in Python, C/C++, Bash, SQL and HTML5. I have developed an Urdu-Hindi transliterator using NLP tools. It gave an accuracy of 75%.
 == Non-Summer-of-Code plans for the summer ==
 I might have to go for a social entrepreneurship trip for 3 days in July. Also, I plan on improving my programming skills by taking part in algorithmic coding competitions. Otherwise, I have nothing else planned for the summer. This project will be my main priority.

@@ Line 71: / Line 71: @@
 | '''Deliverable #1'''
 |
-|Developed the first level of bilingual dictionary
+|Developed the first level of bilingual dictionary based on statistical parameters.
 |-
 |05 & 06

@@ Line 69: / Line 69: @@
 |Evaluate the dictionary created manually by random sampling a few entries against a web dictionary. If accuracy satisfactory, continue with those factors for the language pair if not chose other factors and re-evaluate.
 |-
-| '''D #1'''
+| '''Deliverable #1'''
 |
 |Developed the first level of bilingual dictionary
@@ Line 81: / Line 81: @@
 |Write a script to make bidix entries in an incremental fashion by checking to see whether the source language paradigm has a template with the transfer language paradigm..
 |-
-|'''D #2'''
+|'''Deliverable #2'''
 |
 |Script that makes bidix entries.
@@ Line 97: / Line 97: @@
 |Combine the scripts to automate as much of the process as possible. Improve code, remove bugs, add lots of comments and write wiki for all of the scripts usage.
 |-
-|'''D #3'''
+|'''Deliverable #3'''
 |
 |Final project

@@ Line 69: / Line 69: @@
 |Evaluate the dictionary created manually by random sampling a few entries against a web dictionary. If accuracy satisfactory, continue with those factors for the language pair if not chose other factors and re-evaluate.
 |-
-| '''Deliverable #1'''
+| '''D #1'''
 |
 |Developed the first level of bilingual dictionary
@@ Line 81: / Line 81: @@
 |Write a script to make bidix entries in an incremental fashion by checking to see whether the source language paradigm has a template with the transfer language paradigm..
 |-
-|'''Deliverable #2'''
+|'''D #2'''
 |
 |Script that makes bidix entries.
@@ Line 97: / Line 97: @@
 |Combine the scripts to automate as much of the process as possible. Improve code, remove bugs, add lots of comments and write wiki for all of the scripts usage.
 |-
-|'''Deliverable #3'''
+|'''D #3'''
 |
 |Final project

@@ Line 57: / Line 57: @@
 !PLANS
 |-
-|Week 01
+|01
 |06.17-06.23
 |Choose at least three language pairs with varying degree of relatedness and make word aligned data after running morph analyzer on a parallel corpus of these language pairs. Also, decide on the various factors which will act as filters for the entries such as probability/frequency that will be used to create a first level of dictionary.
 |-
-|Week 02
+|02
 |06.24-06.30
 |Write a script to generate the list of word mappings. Use the filters decided above to get the mappings in both directions.
 |-
-|Week 03 & 04
+|03 & 04
 |07.01-07.14
 |Evaluate the dictionary created manually by random sampling a few entries against a web dictionary. If accuracy satisfactory, continue with those factors for the language pair if not chose other factors and re-evaluate.
@@ Line 73: / Line 73: @@
 |Developed the first level of bilingual dictionary
 |-
-|Week 05 & 06
+|05 & 06
 |07.15-07.28
 | Write the script which generates templates for user's selection by using the most frequent combinations of paradigms in Source Language - Transfer Language.
 |-
-|Week 07 & 08
+|07 & 08
 |07.29-08.11
 |Write a script to make bidix entries in an incremental fashion by checking to see whether the source language paradigm has a template with the transfer language paradigm..
@@ Line 85: / Line 85: @@
 |Script that makes bidix entries.
 |-
-|Week 09
+|09
 |08.12-08.18
 |Create a mini-testvoc for the added words and see that the entries pass it.
 |-
-|Week 10 & 11
+|10 & 11
 |08.19-09.01
 |Create a mini-testvoc for the added words and see that the entries pass it. Compare the bidix dictionary against some online web dictionaries
 |-
-|Week 12
+|12
 |09.02-09.08
 |Combine the scripts to automate as much of the process as possible. Improve code, remove bugs, add lots of comments and write wiki for all of the scripts usage.

@@ Line 17: / Line 17: @@
 == Which of the published tasks am I interested in? What do I plan to do?  ==
-I am interested in the project '''Improved Bilingual Dictionary Induction'''. The aim is to write a set of scripts that generate valid entries for a bidix from the word aligned parallel corpus and also to evaluate the reliability of the extracted translations.
+I am interested in the project '''Improved Bilingual Dictionary Induction'''. The aim is to write a set of scripts that generate valid entries for a bidix from the word aligned parallel corpus and also to evaluate the reliability of the extracted translations. There are no methods that could enable the fully automatic production of dictionaries. Thus, the creation of a completely clean lexicographical resource
+with an appropriate coverage requires a manual post-editing phase. Accordingly, my goal is to provide lexicographers with resources diminishing as much as possible the amount of labor required to prepare full-fledged dictionaries for Apertium's usage.
+===Advantages of using parallel corpora in dictionary creation===
+*High-quality dictionaries are based on corpora. This linguistic data decreases the role of human intuition during lexicographic process.
+*Corpus-driven nature of this method ensures that human insight is eliminated also when hunting for possible translation candidates, that is, when establishing possible pairings of the source language and the target language expressions.
+*The method we will be using will rank the translation candidates according to how likely they are based on automatically determined translational probabilities. This in turn renders possible to determine which sense of a given lemma is the most frequently used. Thus, representative corpora guarantees that not only the most important source lemmata will be included in the dictionary – as in traditional corpus-based lexicography – but also the translations of their most relevant senses.
 ==Proposal Title==

@@ Line 54: / Line 54: @@
 |Week 01
 |06.17-06.23
+|Make word aligned data after running morph analyzer on a parallel corpus of at least three language pairs.
-|
 |-
 |Week 02
 |06.24-06.30
+|Write a script to generate the list of word alignments.
-|
 |-
 |Week 03
 |07.01-07.07
+|Write a script to extract the most frequent combinations of paradigms in Source Language - Transfer Language.
-|
 |-
 |Week 04
 |07.08-07.14
+|Complete the script which generates templates for user's selection.
-|
 |-
 | '''Deliverable #1'''
 |
+|Developed templating system
-|
 |-
-|Week 05
+|Week 05 & 06
-|07.15-07.21
+|07.15-07.28
+|Write a script which checks to see whether the source language paradigm has a template with the transfer language paradigm.
-|
 |-
-|Week 06
+|Week 07 & 08
-|07.22-07.28
+|07.29-08.11
+|Write a script to make bidix entries in an incremental fashion.
-|
-|-
-|Week 07
-|07.29-08.04
-|
-|-
-|Week 08
-|08.05-08.11
-|
 |-
 |'''Deliverable #2'''
 |
+|Script that makes bidix entries in an incremental fashion.
-|
 |-
 |Week 09
 |08.12-08.18
+|EVALUATION: MANUAL: Check a few dictionaries by randomly sampling a few hundred words.
-|
 |-
-|Week 10
+|Week 10 & 11
-|08.19-08.25
+|08.19-09.01
+|EVALUATION: AUTOMATIC: Compare the bidix dictionary against some online web dictionaries
-|
 |-
 |Week 11
-|08.26-09.01
+|08.25-09.01
+|Combine all of the scripts above to automate all the tasks.
-|
 |-
 |Week 12
 |09.02-09.08
+|Improve code, add lots of comments and write wiki for all of the scripts usage.
-|
 |-
 |'''Deliverable #3'''
 |
+|Final project
-|
 |}