User:Ksingla025/Application - Revision history

Francis Tyers at 21:23, 14 March 2014

2014-03-14T21:23:36Z

Francis Tyers: Francis Tyers moved page User:Ksingla025 to User:Ksingla025/Application

2014-03-14T21:22:58Z

Francis Tyers moved page User:Ksingla025 to User:Ksingla025/Application

Ksingla025 at 17:39, 14 March 2014

2014-03-14T17:39:06Z

Ksingla025 at 17:26, 14 March 2014

2014-03-14T17:26:32Z

Ksingla025 at 16:35, 14 March 2014

2014-03-14T16:35:27Z

Show changes

Ksingla025: Normalisation of Non-Standard Text Input

2014-03-14T00:25:54Z

Normalisation of Non-Standard Text Input

Ksingla025: Normalisation of Non-Standard Text Input

2014-03-14T00:22:03Z

Normalisation of Non-Standard Text Input

Ksingla025: Normalisation of Non-Standard Text Input

2014-03-14T00:05:44Z

Normalisation of Non-Standard Text Input

Ksingla025: Normalisation of Non-Standard Text Input

2014-03-14T00:03:08Z

Normalisation of Non-Standard Text Input

Ksingla025: Normalisation of Non-Standard Text Input

2014-03-14T00:01:48Z

Normalisation of Non-Standard Text Input

← Older revision		Revision as of 21:23, 14 March 2014
Line 92:		Line 92:

	Handle the problems and solve them with regard to apertium, along with this there can be a possibility to work on 7. , and report results for improvement in Apertium’s output and also other MT system trained on Europal or standard datasets reported in WMT’14		Handle the problems and solve them with regard to apertium, along with this there can be a possibility to work on 7. , and report results for improvement in Apertium’s output and also other MT system trained on Europal or standard datasets reported in WMT’14

			[[Category:GSoC 2014 Student proposals\|Ksingla025]]

@@ Line 56: / Line 56: @@
 WFST part is quite similar to the part, I used in my paper “Enhancing ASR by MT using Hindi WordNet” in ICON-2013 with Aniruddha Tammewar,Srinivas Bangalore and Michael Carl
-Training data for 3.2 and 3.3 : There is a long list of abbreviations available on the web, but for this they  need to filtered to find the one’s which satisfy 4.2(a & b) conditions.
+Training data for 3.2 and 3.3 : There is a long list of abbreviations available on the web, but for this they  need to filtered to find the one’s which satisfy 3.2(a & b) conditions.
 . Hashtags( easily hand-able using regex )

@@ Line 18: / Line 18: @@
 Codes can be found in the GitHub Repository
 GitHub Repository: https://github.com/ksingla025/Normalizer
+ Phase 2 : Phase 1 reduced the tokens into possible candidates for that token so that they can be matched to a word in the morphological dictionary.
-Phase 2 : Phase 1 reduced the tokens into possible candidates for that token so that they can be matched to a word in the morphological dictionary.
       ===> Make a dictionary look up and take the first word from it.
@@ Line 39: / Line 38: @@
 . Abbreviations
 .1) Frequent [non dictionary words]{ abbreviations which are available online and don't have a dictionary entry}
-          sample : “abbreviations” file in the github repository
+sample : “abbreviations” file in the github repository
 .2)  Train a Linear reg model from “words” to abbreviations that have a single word mapping, and extend the dictionary to new dictionary. ( not a good idea )
@@ Line 46: / Line 46: @@
                 a) Delete vowels and possibly sonorant consonants (hdwd for hardwood)
                 b) Delete all but the ﬁrst syllable (ceil for ceiling)
-	     but works well for  deletion(but not very good for substitution).
+                * but works well for  deletion(but not very good for substitution).
 Ex-> But -> bt
-        Sample Code : available in the github repository “sample_linear_regression.py”, it works fine, try it or modify it to train on a larger abbriviated paralllel data to get best results
+Sample Code : available in the github repository “sample_linear_regression.py”, it works fine, try it or modify it to train on a larger abbriviated paralllel data to get best results
 .3) Predicting words from a abbreviation using “Decision tree based search” using WFST(Weighted FST), will work for some.
@@ Line 64: / Line 64: @@
      Solution : First type is easy to handle, for second, who need to generate the possible words. So a dictionary look up for possible words or FST made of a dictionary can be used.
-) Unknown Characters in Words( works as 3)
+. Unknown Characters in Words( works as 3)
     Problem : Sh**, f**k, ki**
     Solution : Solve them as elongated words were handled, or a domain can be kept in which people use these words most oftenly.
                    Further disambiguation can be done on the basics of a language model, made respectively from monolingual corpus of each language in Apertium.
-) Years
+. Years
     Problem :  Apertium doesn’t handles 1980s, 80s, #1980
@@ Line 77: / Line 76: @@
-) Other Research Ideas Involved :
+. Other Research Ideas Involved :
 If one can use moses, to train a system, then one can train a system on parallel data that looks like :
@@ Line 89: / Line 88: @@
 ALTERNATE : Instead of language modelling take the output, which is there in dictionary and is most widely used in the language, possible in Apertium
-End Goal
+. End Goal
-Handle the problems and solve them with regard to apertium, along with this there can be a possibility to work on  9) , and report results for improvement in Apertium’s output  and also other MT system trained on Europal or standard datasets reported in WMT’14
+Handle the problems and solve them with regard to apertium, along with this there can be a possibility to work on  7. , and report results for improvement in Apertium’s output  and also other MT system trained on Europal or standard datasets reported in WMT’14

@@ Line 2: / Line 2: @@
 My Research Interests mainly include : Machine Translation, Machine Learning, Parsing, Speech Recognition
+I am working in LTRC, IIIT-Hyderabad and has worked in the SEECAT Project,CBS Denmark.
 In Apertium I am working on building a normalizer for Non-Standard text, if you have suggestions or comments i will be thankful !!

@@ Line 1: / Line 1: @@
+I am Karan Singla, currently persuing my MS in Computational Linguiscts from IIIT-Hyderabad, India.
+My Research Interests mainly include : Machine Translation, Machine Learning, Parsing, Speech Recognition
+In Apertium I am working on building a normalizer for Non-Standard text, if you have suggestions or comments i will be thankful !!
 Do we need to add a tokenizer to the normalizer( can include basic cleaning like extra punctuation marks( ... ), unwanted characters(#karan) etc ??

← Older revision	Revision as of 21:22, 14 March 2014
(No difference)

@@ Line 39: / Line 39: @@
 Already handled in the coding challenge(lllooooovvveee, nasssaaaa)
-) Abbreviations
+) Abbreviations(Need Effort)
 .1) Frequent [non dictionary words]{ abbreviations which are available online and don't have a dictionary entry}
@@ Line 67: / Line 67: @@
 Other Noted Problems in the Apertium
 ) special symbols (like @, a mapping needs to be made)
 ) Hashtags( easily hand-able using regex )
-) Sh*t( works as 3)
-) years 80s,1980s,#1980( can be handled)
+) Sh*t( works as 3)
+) years 80s,1980s,#1980( can be handled)