Users guide and notes Jacob - Revision history

Hectoralos: /* Why do pairs with the same language (f.eks English) not share the English monodix? */

2011-12-29T05:45:00Z

Why do pairs with the same language (f.eks English) not share the English monodix?

Francis Tyers at 21:40, 28 December 2011

2011-12-28T21:40:54Z

Francis Tyers at 21:40, 28 December 2011

2011-12-28T21:40:05Z

Jacob Nordfalk: /* TODO */

2009-02-04T01:20:09Z

TODO

Jacob Nordfalk at 03:28, 30 September 2008

2008-09-30T03:28:17Z

Jacob Nordfalk: /* #include clause */

2008-09-01T03:26:43Z

#include clause

Jacob Nordfalk at 08:38, 26 August 2008

2008-08-26T08:38:20Z

Jacob Nordfalk at 07:32, 26 August 2008

2008-08-26T07:32:56Z

Jacob Nordfalk at 07:31, 26 August 2008

2008-08-26T07:31:43Z

Jacob Nordfalk at 18:02, 25 August 2008

2008-08-25T18:02:08Z

@@ Line 85: / Line 85: @@
-==Why do pairs with the same language (f.eks English) not share the English monodix?==
+==Why do pairs with the same language (e.g. English) not share the English monodix?==
 Why can't for example en-ca, en-es and en-eo all share the SAME English dictionary?
-> Then we could all contribute to this gian dict for the advantage of
+> Then we could all contribute to this giant dict for the advantage of
 > all 3 projects?
 >
@@ Line 104: / Line 104: @@
 always like that. For example, there are some things it makes sense to
 distinguish in some language pairs and not in others.
 ==#include clause==

@@ Line 85: / Line 85: @@
-==Why projects concerning the same languages (f.eks English) not share the English monolingual dictionary?==
+==Why do pairs with the same language (f.eks English) not share the English monodix?==
 Why can't for example en-ca, en-es and en-eo all share the SAME English dictionary?
 > Then we could all contribute to this gian dict for the advantage of

@@ Line 1: / Line 1: @@
+{{TOCD}}
-These are my notes for making the English-Esperanto translator but I might be usefull to the same kind of people like me who knows next to nothing about linguistics.
+These are my notes for making the English-Esperanto translator but I might be useful to the same kind of people like me who knows next to nothing about linguistics.
 Ive installed standtard Ubuntu packages and theyre working fine:

@@ Line 212: / Line 212: @@
  - add treeview (and others added to apertium-eo-en.eo-en.dix) to the English monodix
  - make some wiki notes.
+<pre>
+<jacobn> Ok, Ill try the web doc translator more, find the systematics, report a bug and attach files etc.
+</pre>

@@ Line 234: / Line 234: @@
   we have 93% coverage with ~7,000 words
   so we can get 99% coverage with probably 20,000
+[[Category:English and Esperanto]]

@@ Line 108: / Line 108: @@
 Q: In general, is there a way to do something like an #include clause so that I could keep my additions seperate for the rest?
-A: See apertium-en-es/apertium-en-es.en.metadix.xml:
+===Using <nowiki><xi:include/></nowiki>===
+A: You could use <nowiki><xi:include/></nowiki> as is done in apertium-en-es.
+There are, however, a lot of limitations, which make it cumbersome:
+* The apertium tools doesen't support <nowiki><xi:include/></nowiki> directly, so instead of working on the files directly they will have to be preprocessed and then the result of this can be used in Apertium.
+* You will therefore have to make significant changes to the Makefile
+* The included files need to have a en enclosing tag, like <sdefs> below. If not then you'll have to invent one.
+See apertium-en-es/apertium-en-es.en.metadix.xml:
 <pre>
 <?xml version="1.0" encoding="UTF-8"?>
+Here's how to do:
 <dictionary>
   <alphabet>·ÀÁÂÄÇÈÉÊËÌÍÎÏÑÒÓÔÖÙÚÛÜàáâäçèéêëìíîïñòóôöùúûüABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz</alphabet>
@@ Line 135: / Line 145: @@
 </pre>
+===Using shell tools, like cat, head and tail===
+If you just want something simple like the #include in C/C++ then it might be much easier for you to just use cat, head and tail Unix shell commands.
+Imagine that you want to add in the end of the files, at the third last line:
+<pre>
+  #include your file here
+  </section>
+</dictionary>
+</pre>
+then you could just change your Makefile like (original has been prefixed by #):
+<pre>
+$(PREFIX1).automorf.bin: $(BASENAME).$(LANG1).dix tradukunet.$(LANG1).dix
+        (head -n -3 $(BASENAME).$(LANG1).dix; cat tradukunet.$(LANG1).dix; tail -n -3 $(BASENAME).$(LANG1).dix) > tmp.dix
+        apertium-validate-dictionary tmp.dix
+        lt-comp lr tmp.dix $@
+#       apertium-validate-dictionary $(BASENAME).$(LANG1).dix
+#       lt-comp lr $(BASENAME).$(LANG1).dix $@
+$(PREFIX1).autobil.bin: $(BASENAME).$(PREFIX1).dix tradukunet.$(PREFIX1).dix
+        (head -n -3 $(BASENAME).$(PREFIX1).dix; cat tradukunet.$(PREFIX1).dix; tail -n -3 $(BASENAME).$(PREFIX1).dix) > tmp.dix
+        apertium-validate-dictionary tmp.dix
+        lt-comp lr tmp.dix $@
+#       apertium-validate-dictionary $(BASENAME).$(PREFIX1).dix
+#       lt-comp lr $(BASENAME).$(PREFIX1).dix $@
+$(PREFIX1).autogen.bin: $(BASENAME).$(LANG2).dix tradukunet.$(LANG2).dix
+        (head -n -3 $(BASENAME).$(LANG2).dix; cat tradukunet.$(LANG2).dix; tail -n -3 $(BASENAME).$(LANG2).dix) > tmp.dix
+        apertium-validate-dictionary tmp.dix
+        lt-comp rl tmp.dix $@
+#       apertium-validate-dictionary $(BASENAME).$(LANG2).dix
+#       lt-comp rl $(BASENAME).$(LANG2).dix $@
+$(PREFIX1).autopgen.bin: $(BASENAME).post-$(LANG2).dix
+        apertium-validate-dictionary $(BASENAME).post-$(LANG2).dix
+        lt-comp lr $(BASENAME).post-$(LANG2).dix $@
+$(PREFIX2).automorf.bin: $(BASENAME).$(LANG2).dix tradukunet.$(LANG2).dix
+        (head -n -3 $(BASENAME).$(LANG2).dix; cat tradukunet.$(LANG2).dix; tail -n -3 $(BASENAME).$(LANG2).dix) > tmp.dix
+        apertium-validate-dictionary tmp.dix
+        lt-comp lr tmp.dix $@
+#       apertium-validate-dictionary $(BASENAME).$(LANG2).dix
+#       lt-comp lr $(BASENAME).$(LANG2).dix $@
+$(PREFIX2).autobil.bin: $(BASENAME).$(PREFIX1).dix  tradukunet.$(PREFIX1).dix
+        (head -n -3 $(BASENAME).$(PREFIX1).dix; cat tradukunet.$(PREFIX1).dix; tail -n -3 $(BASENAME).$(PREFIX1).dix) > tmp.dix
+        apertium-validate-dictionary tmp.dix
+        lt-comp rl tmp.dix $@
+#       apertium-validate-dictionary $(BASENAME).$(PREFIX1).dix
+#       lt-comp rl $(BASENAME).$(PREFIX1).dix $@
+$(PREFIX2).autogen.bin: $(BASENAME).$(LANG1).dix tradukunet.$(LANG1).dix
+        (head -n -3 $(BASENAME).$(LANG1).dix; cat tradukunet.$(LANG1).dix; tail -n -3 $(BASENAME).$(LANG1).dix) > tmp.dix
+        apertium-validate-dictionary tmp.dix
+        lt-comp rl tmp.dix $@
+#       apertium-validate-dictionary $(BASENAME).$(LANG1).dix
+#       lt-comp rl $(BASENAME).$(LANG1).dix $@
+</pre>
+Here the included files are files are called <code>tradukunet.*.dix</code>.
 ==TODO==

@@ Line 82: / Line 82: @@
  mig: yes, all declinations (ways of conjugation) are all the same in Esperanto
+==Why projects concerning the same languages (f.eks English) not share the English monolingual dictionary?==
+Why can't for example en-ca, en-es and en-eo all share the SAME English dictionary?
+> Then we could all contribute to this gian dict for the advantage of
+> all 3 projects?
+>
+For each project if we want to add it to one dictionary, we need to add
+it to all of them. For example, if you want to add a word to es-en, you
+need to add it to all three dictionaries (en, en-es, es) -- in the
+appropriate form. Otherwise you get the @ # * symbols.
+Because of this, and because not every has the time to edit, or speaks
+all of the languages, we find it more convienient to work with them
+separately, as language pairs, and then merge when/where possible.
+You'll note that most of the paradigm names, for example, are shared.
+Although the ideal is for each dictionary to be "isolated", it isn't
+always like that. For example, there are some things it makes sense to
+distinguish in some language pairs and not in others.

@@ Line 33: / Line 33: @@
 ===Understanding the files===
+<pre>
 <e r="LR"><p><l>kataluno<s n="n"/><s n="f"/></l><r>Catalan<s n="n"/></r></p></e>
 <e r="LR"><p><l>kataluno<s n="n"/><s n="m"/></l><r>Catalan<s n="n"/></r></p></e>
@@ Line 56: / Line 57: @@
  francis.tyers: left-to-right = esperanto to english
+</pre>
-==Words==
-which is the lemma ?
-.08 mig: what is "lemma" ? (a lemming? :-)
- francis.tyers: haha!
-  no
-  the base form
-  root form
-  citation form
-  etc.

@@ Line 93: / Line 93: @@
+==#include clause==
+Q: In general, is there a way to do something like an #include clause so that I could keep my additions seperate for the rest?
+A: See apertium-en-es/apertium-en-es.en.metadix.xml:
+<pre>
+<?xml version="1.0" encoding="UTF-8"?>
+<dictionary>
+  <alphabet>·ÀÁÂÄÇÈÉÊËÌÍÎÏÑÒÓÔÖÙÚÛÜàáâäçèéêëìíîïñòóôöùúûüABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz</alphabet>
+        <!-- symbols -->
+        <xi:include xmlns:xi="http://www.w3.org/2001/XInclude"  href="apertium-en-es.symbols.xml"/>
+        <!-- paradigms -->
+        <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="apertium-en-es.en.pardefs.xml"/>
+</pre>
+And then in apertium-en-es.symbols.xml:
+<pre>
+<?xml version="1.0" encoding="UTF-8"?>
+  <sdefs>
+    <sdef n="comp" />
+    <sdef n="detnt" />
+    <sdef n="predet" />
+    <sdef n="past" />
+    <sdef n="atn" />
+</pre>
@@ Line 99: / Line 128: @@
  - add treeview (and others added to apertium-eo-en.eo-en.dix) to the English monodix
  - make some wiki notes.
+===File from traduku.net===
+convert it into EN : EO
+  then tag the EO side
+  and strip out the nouns and adjectives
+  those are most important to start with
+  then grab a corpus
+  (wikipedia, or euro parl or something)
+.15 and order them by frequency of the english word
+ mig: why reorder?
+ francis.tyers@gmail.com: higher frequency words are more important
+.16 if you translate "the" correctly, you cover ~50% of the text, if you translate "gable" correctly you cover maybe 0.5%
+.17 mig: yes, yes, but why bother if all words get in?
+ francis.tyers@gmail.com: because someone has to add the inflection for the english side
+  the esperanto side is regular, but the english is not always regular
+.18 mig: OK, so reording is important because we probably wont make all 110000.
+ francis.tyers@gmail.com: yep
+  but the good news is we don't need to make 110000
+  we have 93% coverage with ~7,000 words
+  so we can get 99% coverage with probably 20,000

@@ Line 96: / Line 96: @@
 ==TODO==
+ - go through http://wiki.apertium.org/wiki/Monodix_basics and review the file (the apertium-eo-en.eo.dix file)
+ - add treeview (and others added to apertium-eo-en.eo-en.dix) to the English monodix
-.11 mig: Ill try to go http://wiki.apertium.org/wiki/Monodix_basics and review the file.
+ - make some wiki notes.
-  this is the apertium-eo-en.eo.dix file
-.13 Fine, Ill review it.