User:Francis Tyers/Sandbox2 - Revision history

Francis Tyers at 08:11, 30 September 2011

2011-09-30T08:11:15Z

Francis Tyers at 11:42, 28 September 2011

2011-09-28T11:42:20Z

Francis Tyers at 10:46, 19 September 2011

2011-09-19T10:46:45Z

Francis Tyers at 10:46, 16 September 2011

2011-09-16T10:46:47Z

Francis Tyers at 09:10, 15 September 2011

2011-09-15T09:10:25Z

Francis Tyers at 11:49, 14 September 2011

2011-09-14T11:49:11Z

Francis Tyers at 10:00, 12 September 2011

2011-09-12T10:00:39Z

Francis Tyers at 11:05, 7 September 2011

2011-09-07T11:05:16Z

Francis Tyers at 15:11, 2 September 2011

2011-09-02T15:11:57Z

Francis Tyers at 11:43, 4 August 2011

2011-08-04T11:43:31Z

@@ Line 46: / Line 46: @@
 * Take the top scoring analysis:analysis results where the POS matches
 * Where the word is already ambiguous in the Apertium dictionaries, add the possibilities from GIZA to the dictionary so that they may be chosen -- only added with POS tag.
+==Annotation process==
+# Translate corpus (native speaker of English, competent Catalan), adding missing translations to bilingual dictionary options.
+#* Words with too many tagging errors, or MWE errors are left out.
+# Proofread corpus
+# Run corpus up to lexical transfer stage
+# Annotate output of lexical transfer

@@ Line 26: / Line 26: @@
 * Rationale: Dictionary doesn't provide good enough coverage to produce statistically significant results over a whole corpus.
-* Discarding because of bad tagging/MWE recognition: 'to', 'sol', 'portada', 'cap', 'cop' (less than 60% correct)
+* Discarding because of bad tagging/MWE recognition: 'to', 'sol', 'portada', 'cap', 'cop', 'marxa' (less than 60% correct)
 Training corpus:

@@ Line 26: / Line 26: @@
 * Rationale: Dictionary doesn't provide good enough coverage to produce statistically significant results over a whole corpus.
-* Discarding because of bad tagging: 'to', 'sol', 'portada', 'cap' (less than 60% correct)
+* Discarding because of bad tagging/MWE recognition: 'to', 'sol', 'portada', 'cap', 'cop' (less than 60% correct)
 Training corpus:

@@ Line 26: / Line 26: @@
 * Rationale: Dictionary doesn't provide good enough coverage to produce statistically significant results over a whole corpus.
-* Discarding because of bad tagging: 'to', 'sol', 'portada' (less than 60% correct)
+* Discarding because of bad tagging: 'to', 'sol', 'portada', 'cap' (less than 60% correct)
 Training corpus:

@@ Line 26: / Line 26: @@
 * Rationale: Dictionary doesn't provide good enough coverage to produce statistically significant results over a whole corpus.
-* Discarding because of bad tagging: 'to', 'sol' (less than 50% correct)
+* Discarding because of bad tagging: 'to', 'sol', 'portada' (less than 60% correct)
 Training corpus:

@@ Line 38: / Line 38: @@
 * Full analysis:Full analysis dic from Giza++
 * Rules from phrase table
+Process for using GIZA++:
+* Tag both sides of the corpus (europarl, en-ca, first 1,700,000 sentences) with the Apertium language pair.
+* Extract the model/lex.f2e file.
+* Take the top scoring analysis:analysis results where the POS matches
+* Where the word is already ambiguous in the Apertium dictionaries, add the possibilities from GIZA to the dictionary so that they may be chosen -- only added with POS tag.

@@ Line 1: / Line 1: @@
+Constraint-based lexical selection for rule-based machine translation
 <pre>
 Corpus: cawiki-20110616-pages-articles.xml.bz2
@@ Line 17: / Line 19: @@
 Test corpus:
+* 150 test words
-* 2,000 sentences
+* 1,500 sentences
 * 10 per test word
 * Randomly selected from the subset of sentences which were found in the corpus.
 * Only words with >100 example sentences included
+* Rationale: Dictionary doesn't provide good enough coverage to produce statistically significant results over a whole corpus.
+Training corpus:
-Baseline:
+Baselines:
-* Idea: Full analysis:Full analysis dic from Giza++
-*: This would require a parallel corpus.
+* TL Frequency-best
-Rationale:
+* TLM-best
+* Linguist set
+* Full analysis:Full analysis dic from Giza++
-* Dictionary doesn't provide good enough coverage to produce statistically significant results over a whole corpus.
+* Rules from phrase table
-==Testing==
-;Input: Les Carmelites el veneren com a sant patró seu.
-<pre>
-^El<det><def><f><pl>/The<det><def><f><pl>$
-^*Carmelites/*Carmelites$
-^prpers<prn><pro><p3><m><sg>/prpers<prn><obj><p3><nt><sg>$
-^venerar<vblex><pri><p3><pl>/venerate<vblex><pri><p3><pl>$
-^com a<pr>/as a<pr>$ ^sant<adj><m><sg>/saint<adj><m><sg>$
-^patró<n><m><sg>/patron<n><sg>/owner<n><sg>/master<n><sg>/head<n><sg>/pattern<n><sg>/employer<n><sg>$
-^seu<adj><pos><m><sg>/his<adj><pos><m><sg>$^.<sent>/.<sent>$^.<sent>/.<sent>$
-</pre>
-;Reference:
-<pre>
-	]^El<det><def><f><pl>/The<det><def><f><pl>$
-^*Carmelites/*Carmelites$
-^prpers<prn><pro><p3><m><sg>/prpers<prn><obj><p3><nt><sg>$
-^venerar<vblex><pri><p3><pl>/venerate<vblex><pri><p3><pl>$
-^com a<pr>/as a<pr>$
-^sant<adj><m><sg>/saint<adj><m><sg>$
-^patró<n><m><sg>/patron<n><sg>$
-^seu<adj><pos><m><sg>/his<adj><pos><m><sg>$^.<sent>/.<sent>$[
-</pre>
-;Test 1 (1/6)
-<pre>
-^patró<n><m><sg>/patron<n><sg>/owner<n><sg>/master<n><sg>/head<n><sg>/pattern<n><sg>/employer<n><sg>$
-</pre>
-;Test 2 (1/1)
-<pre>
-^patró<n><m><sg>/patron<n><sg>$
-</pre>
-;Test 3 (1/4)
-<pre>
-^patró<n><m><sg>/patron<n><sg>/owner<n><sg>/master<n><sg>/employer<n><sg>$
-</pre>