User:Fpetkovski/GSoC-2012 Application - Revision history

Fpetkovski: /* The classification process */

2012-04-05T18:18:59Z

The classification process

Fpetkovski at 18:18, 5 April 2012

2012-04-05T18:18:23Z

Fpetkovski: /* Personal Info */

2012-04-05T17:50:15Z

Personal Info

Fpetkovski at 16:43, 5 April 2012

2012-04-05T16:43:28Z

Fpetkovski: /* Which of the published tasks are you interested in? What do you plan to do? */

2012-04-05T16:33:48Z

Which of the published tasks are you interested in? What do you plan to do?

Fpetkovski: /* Work already done */

2012-04-05T02:33:35Z

Work already done

Fpetkovski: /* Work already done */

2012-04-05T02:31:41Z

Work already done

Francis Tyers: /* Non-GSoC activities */

2012-03-31T20:39:07Z

Non-GSoC activities

Fpetkovski: /* TO-DO */

2012-03-31T12:02:43Z

TO-DO

Fpetkovski: /* Prior to May 21 */

2012-03-31T12:02:13Z

Prior to May 21

@@ Line 130: / Line 130: @@
 === The classification process ===
-The first step is to extract all the possible features that one can think of. These include all the tokens surrounding the target word, all of the target word's tags, all of the tags of the surrounding words, is a word capitalized, how many commas are there in a specified window left and right of the target word etc. Once these features are extracted, a tool like RapidMiner or Weka will be used to test the performance of different classifiers on the whole feature set. Feature selection will be done if needed. Once the optimal feature set has been determined, the classification process will be implemented in a separate module and integrated in Apertium.
+The first step is to extract all the possible features one can think of. These include all the tokens surrounding the target word, all of the target word's tags, all of the tags of the surrounding words, is a word capitalized, how many commas are there in a specified window left and right of the target word etc. Once these features are extracted, a tool like RapidMiner or Weka will be used to test the performance of different classifiers on the whole feature set. Feature selection will be done if needed. Once the optimal feature set has been determined, the classification process will be implemented in a separate module and integrated in Apertium.
 === Training, testing and evaluation ===

@@ Line 127: / Line 127: @@
 Čaša vode. -> A glass ''of'' water <br />
 Čaša moje sestre. -> ''My sister's'' glass <br />
+=== The classification process ===
+The first step is to extract all the possible features that one can think of. These include all the tokens surrounding the target word, all of the target word's tags, all of the tags of the surrounding words, is a word capitalized, how many commas are there in a specified window left and right of the target word etc. Once these features are extracted, a tool like RapidMiner or Weka will be used to test the performance of different classifiers on the whole feature set. Feature selection will be done if needed. Once the optimal feature set has been determined, the classification process will be implemented in a separate module and integrated in Apertium.
+=== Training, testing and evaluation ===
+Training and testing will be done using a k-fold cross validation. The whole data set will be split into two parts, an optimization set and a validation set. The optimization set will be used for optimizing classifier parameters[:)], by making a training and testing set out of it, training the classifier on the training set using one set of parameters and testing it on the testing set.
+Once the optimal parameter values have been determined, the performance will be tested on a set that has not been seen by the classifier yet (the validation set).
 == Skills, qualifications and field of study ==
@@ Line 136: / Line 145: @@
 == Non-GSoC activities ==
 I have final exams at the beginning of June, but I will be able to work more than 30 hours / week.

@@ Line 3: / Line 3: @@
 First name: Filip <br />
 Last name: Petkovski <br />
-email: filip.petkovsky@gmail.com <br />
+email: filpetkovski@gmail.com <br />
 fpetkovski on IRC: #apertium <br />

@@ Line 14: / Line 14: @@
 Apertium is a great project. It is obvious that a ton of work has been put into both developing the platform and creating the language resources. However, there is always more work to be done and being a part of this project is a perfect opportunity to make a big contribution to society.
+== Why should Google and Apertium sponsor it? ==
+Capturing context is an important part of machine translation. It enables fluency and translations which are much closer to human-generated ones. However, it is quite difficult to achieve such a thing by relying solely on rules and a statistical/corpus based approach yields better results for solving this problem.
+This project is intended to be an extension to the Apertium platform and will not only improve the existing Serbo-Croatian - Macedonian pair, but also serve as a prototype for implementing similar techniques in some of the other language pairs.

@@ Line 18: / Line 18: @@
 == Which of the published tasks are you interested in? What do you plan to do? ==
-I'm interested in building a Corpus-based lexicalised feature transfer module which will set tags based on a corpus-generated model.
+I'm interested in building a Corpus-based lexicalised feature transfer module which will set tags based on a corpus-generated model. The existing sh-mk language pair will be used.
 == Work already done ==

@@ Line 22: / Line 22: @@
 == Work already done ==
-started the apertium-sh-en language pair in ''incubator''.
+* started the apertium-sh-en language pair in ''incubator''.
-created a stream-processor for the output of apertium-transfer that reads character by character (/branches/gsoc2012/fpetkovski/stream-processor).
+* created a stream-processor for the output of apertium-transfer that reads character by character (/branches/gsoc2012/fpetkovski/stream-processor).
-created a stream processor for the output of apertium-transfer that removes stop words specified in a dictionary (/branches/gsoc2012/fpetkovski/stopwords-filter).
+* created a stream processor for the output of apertium-transfer that removes stop words specified in a dictionary (/branches/gsoc2012/fpetkovski/stopwords-filter).
-started a corpus based classifier (/branches/gsoc2012/fpetkovski/corpus-based-classifier)
+* started a corpus based classifier (/branches/gsoc2012/fpetkovski/corpus-based-classifier)
 == Work to do (Incomplete) ==

← Older revision		Revision as of 20:39, 31 March 2012
Line 131:		Line 131:

	I have final exams at the beginning of June, but I will be able to work more than 30 hours / week.		I have final exams at the beginning of June, but I will be able to work more than 30 hours / week.

			[[Category:GSoC 2012 Student Proposals\|Fpetkovski]]

@@ Line 35: / Line 35: @@
 b. More about modularity, user friendliness. <br />
 . Something about training, testing and crossvalidation. <br />
-. How will the baselines be made. <br />
+. <s> How will the baselines be made. </s> <br />
-. More on the n-gram model. <br />
+. <s> More on the n-gram model. </s> <br />
 === The big picture ===

@@ Line 58: / Line 58: @@
 * '''Finish the stream-processor so it will parse lemmas and tokens in a data structure.''' This will be useful later for extracting features from lexical units in the stream.
-* Set baselines for definiteness, correct preposition, aspect and genitive type. A simple, and often used, way of setting a baseline is to take the class with the higher prior probability, and assign that label to all the examples in the training set. For the definiteness problem we would take the form of the noun / pronoun that is the most common (say definite), and say that all the nouns / pronouns  are definite and measure performance on a testing set. The same method can be used for setting the other three baselines.
+* '''Set baselines for definiteness, correct preposition, aspect and genitive type.''' A simple, and often used, way of setting a baseline is to take the class with the higher prior probability, and assign that label to all the examples in the training set. For the definiteness problem we would take the form of the noun / pronoun that is the most common (say definite), and say that all the nouns / pronouns  are definite and measure performance on a testing set. The same method can be used for setting the other three baselines.
 * '''Create a simple n-gram model and see how it performs.''' The idea is to use only the target side to determine whether a noun is definite. A simple way we can try is to count n-grams (bigrams, trigrams, fourgrams etc.) of the POS tags of the words before a noun/pronoun. Then, a back-off or interpolation technique can be used to determine the definiteness based on the counts in the model. For example, if we have a sequence of <pronoun> <verb> <adjective> (and then <noun>), we would check in the model and take the definiteness tag that a noun which follows that trigram most often has. If that trigram has never appeared in the training set, we would back-off to a bigram model and, in this case, disregard the <pronoun> tag.