Difference between revisions of "User talk:Muki987"

Revision as of 19:05, 20 May 2009

Basque

This conversation is a bit heated for me, but note that in our Basque→Spanish system we do something similar with Basque cases. For example, a typical way of representing "hegoak" would be:

^hegoak/hego<n>+a<det><art><pl>/
        hego<n>+a<det><art><sg>+k<post>$

"<hegoak>"
  "hego"  IZE ARR DEK ABS NUMP MUGM
  "hego"  IZE ARR DEK ERG NUMS MUGM

Note how in our representation the case is marked as a postfix k<post> where in the more traditional analysis it is marked as a case ERG (ergative). Compared with Basque→Spanish, Hungarian→English would be easier in terms of word order:

S                                         O                      V
Txinako Poliziak, datu ofizialen arabera, 1.317 pertsona atzeman zituen 
la Policía de China, según los datos oficiales, 1.317 personas capturó
The Chinese police, according to official data, 1,317 people detained.

`According to official data the Chinese police detained 1,317 people.'

- Francis Tyers 10:13, 9 April 2009 (UTC)

I think, the discussion gets calmer, since I start to see the pont of xxx, (and he mine) which is helpful and valid. I see, Matxin can handle both Basque-Spanish and Spanish-Basque, so I'll look throughoutly into that. Basque is also a hun language, as far as I know, very similar to Hungarian. Muki987 10:32, 9 April 2009 (UTC)

Actually, the Matxin system cannot handle Basque→Spanish, as there is no dependency analysis for Basque. Apertium is used for Basque→Spanish and Matxin for Spanish→Basque. As far as I know, Basque does not have any living relatives. - Francis Tyers 10:44, 9 April 2009 (UTC)

That's important for me to know, thanks. Muki987 11:30, 9 April 2009 (UTC)

Is there any difference between the main diagram "How Apertium works" between Apertium and Matxin? If yes, where, if not: What is the difference between Matxin and Apertium (except of character coding)? Muki987 12:37, 9 April 2009 (UTC)

Basque has lots of living relatives, Hungarian, Armenian, Turkish, Aserbaidshan, Uigur, Finnish, Estonian, Persian, Japanese (thru Ainu = hunnish influence), Ketchua (Inka language in south America), ancient Egyptian (no more living, but hieroglyphes show a great past), Etruscian (also no more living, but great past), Hindi, and more. Muki987 11:30, 9 April 2009 (UTC)

Considerations for prefix groups and possessions

Prefix groups

In English, one prefix can handle more nouns, for example: I travel to England, France and Spain. This will be translated as: Utazom Angliá-ba, Franciaország-ba és Spanyolország-ba. ("-" added for clarification). Utazom: I travel, Angliába: to England, ... , Spanyolországba: to Spain

In English the prefix ...nouns structure will be closed by:

a dot (finishing the sentence)
a verb - I travel to england an spain and will carry a bag- the word "will" closes the scope of to.
a new prefix - I travel to england and spain with train or aeroplane- the word "with" closes the scope of to.

Co-ordinated noun phrase with case agreement. I would probably do this kind of thing in pre-transfer with a constraint grammar. Basically write a rule which does: "add accusative case to nouns following the preposition 'to' until a new preposition, verb or end-of-sentence". - Francis Tyers 10:22, 9 April 2009 (UTC)

Possesions

In English the possessor may be before the possesion: Peter's coffee and tee

but also behind it: the coffee and tee of Peter

In Hungarian the possessor is always strictly before the possesion, both sentences above must be translated as: Péter kávé-ja és teá-ja. (again "-" just for clarity).

In English the possession structure will be closed by

a dot (finishing the sentence)
a verb - Peters coffee and tee looks like a bag - the word "looks" closes the scope of possession structure
a new prefix - Peters coffee and tee with sugar - the word "with" closes the scope of possession structure

In the case of "the coffee and tee of Peter" type possession relation: If an noun enumeration starts, the translator must watch. If the enumeration ends with "of", this is a possession structure, and must be translated, os such.

Combination of possession and prefix

With Peter's coffee and tee - Péter kávé-já-val és teá-já-val - ja is possession, val, vel is with
With the coffe and tee of Peter - as above

Adding plural

With Peter's coffee and tees - Péter kávé-já-val és teá-i-val - "i" is plural possession for tea

Remark

These kind of structures caused for me the most manual work when translated texts from English/German, therefore it is very important to set up their proper translation. Thanks in advance for any critics/thought/comments. Muki987 10:12, 9 April 2009 (UTC)

An interesting example. This is a fourth kind of structure closing signal: noun immediately followed a verb also stops the structure:

I drank Peter's coffee and children played near to us.

I saw Peter's coffee and tee smell like sugar - this sentence is even in English is ambiquous - what smells like sugar, both or only tee? Would a comma after coffee limit possession to coffee? Muki987 11:48, 9 April 2009 (UTC)

I'm not sure if this is ambiguous. The ambiguity is resolved by the inflection of the verb in this particular case.

?I saw Peter's coffee and tee smell like sugar

I saw [Peter's coffee] and [tea] smell like sugar

I saw [Peter's coffee and tea] smell like sugar

- Francis Tyers 13:36, 9 April 2009 (UTC)

Some tests

Peter's coffee and tee are sweet

el café de Peter y tee es dulce

Peter's mice and cats are sweet

los ratones de Peter y los gatos son dulces

Peter's mice and Peter's cats are sweet

los ratones de Peter y los gatos de Peter son dulces

This clearly shows, that in case of possesion apertium would not consider multiple possessions. Since it is also ambiguious in English, this is a feature.

Peter's coffee and tea are sweet

el café de Peter y el té son dulces

In this case, rules are not available for co-ordinated noun phrases, and so the translations come out rather badly. It is not to say that it is impossible to do in Apertium, just that so far for English→Spanish we have had more important things to work on. - Francis Tyers 13:27, 10 April 2009 (UTC)

Same is true for attributes:

Little boys and girls were playing

Pocos chicos y las chicas jugaban

Little boys and little girls were playing

Pocos chicos y pocas chicas jugaban

attribute is only taken as attribute of the neighbouring noun. Again, a feature. Muki987 13:14, 10 April 2009 (UTC)

Part of the problem above is in part-of-speech tagging "Little" is being taken as as a quantifier. If you try with 'small' the result is better.

Small boys and girls were playing

Chicas y chicos pequeños jugaban

Also in small case only the boys are concerned. Apertium is rather consistent here. But let's consider this as a feature, since English is also ambiguous here. Muki987 09:48, 11 April 2009 (UTC)

The ambiguity is preserved in the sentence — this is a feature. - Francis Tyers 10:41, 11 April 2009 (UTC)

Small boys and small girls were playing

Los Chicos pequeños y las chicas pequeñas jugaban

Here both of the translations are ok. - Francis Tyers 13:27, 10 April 2009 (UTC)

Expressions

What about expressions? For example "look after one's fences" this can be in the form of looking, looked, however, at present not handled at all:

Peter looked after Martha's fences
Peter miraba después de las vallas de Martha

The expression will be not at all recognized (Peter handled in the interest of Martha).

Is there something planned for this? Are there working examples available? 20-30% of our speech are expressions!!!! Muki987 13:38, 10 April 2009 (UTC)

Further subjects

>'házas (married- repeat all previous for this up to here, except the last 2) 1680' -- a married house? Really?

That word is a bit exception, since it has two meanings házas means married, and also a man/woman, who has a house In case if ing (shirt) inges means someone, who wears a shirt

Not specific to English, but sharper in English, than in any other cultur language. What about your ideas to solve it? Muki987 11:54, 8 April 2009 (UTC)

Word sense disambiguation

We don't currently have a good working lexical selection module, but it is one of the ideas we're hoping to get implemented through GSOC. - Francis Tyers 21:48, 8 April 2009 (UTC)

From Wikipedia:

In some cases, the diminutive suffix has become part of the basic form. These are no longer regarded as diminutive forms:

Animals

-ka/ke: fóka (seal), róka (fox), csóka (jackdaw), pulyka (turkey), szarka (magpie)
-cska/cske: macska (cat), kecske (goat), fecske (swallow), szöcske (grasshopper)

You see, you get better answers in fickipedia. You are right, this is an issue for translations, however one of the issues, that can easily be covered. Muki987 11:54, 8 April 2009 (UTC)

Sure, and a lot of others also not. One after the other. Muki987 18:14, 8 April 2009 (UTC)

This is rather off the topic of the discussion, this page is more to discuss methods of representing agglutinative morphology in Apertium, rather than the translation problems of agglutinative languages (which are also interesting, but better reserved for another page, or the mailing list). :) - Francis Tyers 08:21, 7 April 2009 (UTC)

Glad to hear, that you are convinced, apertium technology is suitable for agglutinative languages. Having gone thru the English-SerboCroatian example I was not that sure. I am at the moment in the evaluation phase, and I am looking for all existing technologies. At present in my opinion google translation technology with its statistical, grammar free approach will never have the quality of a grammar oriented one, like apertium. It will for ever remain on the surface, with no real improvement perspective. However, for some situations it is very helpful. That was my first step in the direction. We can continue this subject on my discussion page, if xxx wants. Muki987 10:02, 7 April 2009 (UTC)

Regarding other free grammar-focussed MT engines, you might also check out and Matxin. Open Logos has the downside of not supporting UTF-8 and not having very active development, while Matxin requires a dependency grammar to be written in Freeling format. If you want to go from English→Hungarian then this might be the answer, as they already have one written for English, but for Hungarian→English, it might take some extra development time. The Constraint grammar formalism for disambiguation and syntactic annotation might also be interesting. I'm quite happy to discuss other options and if you have any questions, please contact us on the mailing list, personally or through IRC. - Francis Tyers 10:36, 7 April 2009 (UTC)

PS. Are you the one asking on the hunmorph list about generation ('morp visszafele')? :) - Francis Tyers 12:00, 7 April 2009 (UTC)

Yes, I am an old language "rabbit" :-) Peter H. says, hunlex knows something similar, we are waiting for Victor, the author, he might know..... Muki987 18:04, 7 April 2009 (UTC)

Resources

Perhaps we could make a page of free resources for Hungarian ? - Francis Tyers 12:59, 9 April 2009 (UTC)

Sure, why not. As I go ahead, I'll think of the idea, and collect things. Muki987 13:31, 9 April 2009 (UTC)

Lexicon

I took a look at the lexicon you pointed me to. It looks ok, although it isn't tagged for part-of-speech. There are lots of "set phrases" (which we can extract into a translation memory) and "multiwords" (nice!) Some questions on the format:

writer
szerzõ
írnok
író

write up
elõnyös színben tüntet fel
feldicsér
feldolgoz
kidolgoz
megír
naprakész állapotba hoz

Is it:

<word1 in English>
<trans1 in Hungarian>
<trans2 in Hungarian>
 
<word2 in English> 
<trans1 in Hungarian>
...

E.g. blank line, English, Hungarian ... blank line ?

blank line is the sign for "next word" Muki987 20:11, 10 April 2009 (UTC)

Is the first translation always the most frequent one ?

No, they are alphabeticalMuki987 20:11, 10 April 2009 (UTC)

The first step in conversion might be to take out the entries which only have one translation into a file which looks like

program ; program
workweek ; munkahét
workshop ; mûhely
...

Then analyse the left side with the English analyser from Apertium, the right side with the hunmorph.

^program/program<n><sg>$ ;  program/NOUN
^workweek/*workweek$ ; munkahét/NOUN
^workshop/workshop<n><sg>$ ; mûhely/NOUN
^work/work<n><sg>/work<vblex><inf>/work<vblex><pres>$ ; alkotás/NOUN @ alkot/VERB[GERUND]/NOUN @ alkot/VERB[GERUND]/NOUN
...

Then extract the entries which have analyses in both analysers which agree for parts of speech.

^program/program<n><sg>$ ;  program/NOUN
^workshop/workshop<n><sg>$ ; mûhely/NOUN
^work/work<n><sg>$ ; alkotás/NOUN
...

And convert that to Apertium format:

  <e><p><l>program<s n="n"/></l><r>program<s n="NOUN"/></r></p></e>
  <e><p><l>workshop<s n="n"/></l><r>mûhely<s n="NOUN"/></r></p></e>
  <e><p><l>work<s n="n"/></l><r>alkotás<s n="NOUN"/></r></p></e>
  ...

This could largely be done automatically, but would need to be manually checked. I would focus on the most frequent open-category words, closed-category words can be done better by hand from scratch. - Francis Tyers 13:48, 10 April 2009 (UTC)

I've emailed you a "first pass" along with some of the commands I used for creating it. If you have Apertium installed you will be able to see basic noun transfer. I still haven't figured out a way to get hunmorph to do generation though. PS. I would like to ask your permission to put the generated file in our incubator - Francis Tyers 14:47, 10 April 2009 (UTC)

Received, thanks a million. :-) I have to install apertium on my pc, learn how to use this, and test the results. Sounds very promising!!!! Muki987 20:11, 10 April 2009 (UTC)

Ok cool :) Can I put the dictionary I extracted in our incubator? It isn't mandatory, but it might be useful to someone at some point. - Francis Tyers 20:21, 10 April 2009 (UTC)

Sure, you can. I think, however, it needs a lot of work before it is really usable. Muki987 21:12, 10 April 2009 (UTC)

Thanks, and yes it definitely does, but one of my principles, is release as soon as possible :) A lot of times I've come across stuff that would be useful to me, but people don't want to release it until it is "finished" -- which sometimes doesn't happen then you lose all the work... If you rifle through the incubator you'll see a lot of junk that I've just played around with and put up there... in case someone else finds it useful some day. - Francis Tyers 21:20, 10 April 2009 (UTC)

Look after

I just spoke to another developer and he says that "vigilar" is a better translation than "cuidar". So i've added it to the SVN and the change should be live in ~12 hours.

English dictionary:

    <e lm="look after"><i>look</i><par n="accept__vblex"/>      <p>        <l><b/>after</l><r><g><b/>after</g></r></p></e>

English--Spanish bilingual dictionary:

    <e srl="look_after"><p><l>look<g><b/>after</g><s n="vblex"/></l><r>vigilar<s n="vblex"/></r></p></e>

And now:

$ echo "Peter looked after Martha's fences" | apertium -d . en-es
Peter vigiló las vallas de Martha

Much better :) - Francis Tyers 20:33, 10 April 2009 (UTC)

Not really. Completely missing the expression meaning, he cares for her interest. vallas is fence, and not interest.

Please also see the examples on your page, not a single working expression :-( Muki987 21:27, 10 April 2009 (UTC)

I don't agree. I am a native speaker of English, and from that expression I take "Peter looked after the fences of Martha". If you would like to talk by telephone, I'll email you my phone number, but I think you have quite an unusual idea of what English is. - Francis Tyers 21:52, 10 April 2009 (UTC)

By the way, in case you had any doubt, I wouldn't expect the translator to be able to translate "alri' there ma' gizzu gleg", "here y'ar" or "this int nowt" either. - Francis Tyers 22:12, 10 April 2009 (UTC)

I also took the opportunity to ask a speaker of American English and he suggested that you might have wanted to say "mend fences" ? Which means "to resolve past conflicts, or put differences aside" -- personally I've never heard of that one, but perhaps you got mixed up? - Francis Tyers 07:53, 11 April 2009 (UTC)

Ahh, so what I think, is a usual English expression (look after one's fences == take care of one's interest) is not a common expression at all? I must rely on dictionaries I find here and there, like Jim pertaining Hungarian grammar. So now I changed my picture, thank for your input, as native English speaker. I use skype, and we could talk over skype at any time, however, writing is probably more effective. Muki987 09:56, 11 April 2009 (UTC)

No, it definitely isn't a common expression! :) Did you ever see the Monty Python sketch "Hungarian phrasebook (youtube)"? I don't use Skype as there is unfortunately not a free client for it. - Francis Tyers 10:13, 11 April 2009 (UTC)

Good to know (the expression), thanks!

That sketch is not funny for me at all.

Skype is completely free, very-very easy to use. Telephon abroad costs always a lot, no matter how you try. Muki987 10:24, 11 April 2009 (UTC)

I have an idea that might prevent you spending your time on something that isn't what you want or need. Could you send me ten sentences in Hungarian of varying levels of complexity, with linguist's glosses in English and translation in English. e.g.

1. John lát egy almát

~~ John see+VERB an+ART apple+ACC

~~ John sees an apple

This would be an "easy" sentence to translate, could you send 10 examples of others from "easy" to hard? I'll then look at them and let you know which ones will be able to be translated with Apertium, which with Matxin, and which will not be able to be translated (with machine translation). - Francis Tyers 12:20, 11 April 2009 (UTC)

Examples

I have put some addresses onto my page.

Yes, I saw :) Incidentally, our lead developer has a print out of one of the descriptions of the MetaMorfo parser on his desk. We're planning to do something similar one of these days... - Francis Tyers 21:26, 11 April 2009 (UTC)

Glosses

John and Martha's apple and pear were sweet

John és Martha almája és körtéje édesek voltak

John and Martha apple-his and pear-his sweet-s be+past+plural,3 person

A remark: John és Martha almája és körtéje édes- if in present - John and Martha apple+his and pear-his sweet. (no need for verb in this case, the adjektiv is in singular)

The ski jumpers were on the top of the hill

A síugrók a hegy tetején voltak

The ski-jumpers the hill top-its-on be+past+plural,3 person

In present: A síugrók a hegy tetején vannak -- the ski-jumpers the hill top-its-on be+plural,3.person

however: The ski jumpers ont the top of the hill are friendly

A síugrók a hegy tetején barátságosak - the ski-jumpers the hill top-its-on friendly-s -- No need for verb here.

He travelled in a nice coach

Jó kocsiban utazott

good coach-in travel+past+singular,3 person

Interesting here that the verb comes final. How would you say, for example "John saw Martha" and "John saw Martha through his telescope"? - Francis Tyers 19:17, 12 April 2009 (UTC)

John saw Martha

John látta Marthát

John see+past+sng.3person Martha+ACC

John saw Martha through his telescope

John látta Mártát teleszkópján

John see+past+sng.3person Martha+ACC telescope+his+on

John saw Martha through his telescope

John látta Mártát teleszkópján keresztül

John see+past+sng.3person Martha+ACC telescope+his+on through

-- this version is a bit overkill with the "keresztül", the previous is more practical

Ok, so it seems that with adverbial phrases "in a nice coach", "on the top of the hill" and with adjective complements, they come before the verb. How about:

John saw through his telescope

Would be:

John teleszkópján látta -- John telescope-his-on see+sng+3.pers

? - Francis Tyers 20:14, 12 April 2009 (UTC)

It depends on the context.

John saw through his telescope that the Moon is yellow

John teleszkópján látta, hogy a hold sárga -- John telescope-his-on see+sng+3.pers, that the moon yellow.

John looked through his telescope to see a part of the moon.

John teleszkópjába nézett, hogy lássa a hold egy részét. -- John telescope-his-into look+past,sng,3.pers because see+imperativ+sng+3.pers the moon one part.

It is also important, that each verb has 2 conjugated form: with and without subject

I watch = nézek or nézem. Nézek: I watch down: generally, nothing special. Nézem: I watch the cat.

... (each person)

They watch = néznek or nézik. They watch down: néznek they watch the cat: nézik

1. example genitiv and grouping with "and"

Ok, let's start:

~~John's and Martha's apple and pear were sweet~~
John and Martha's apple and pear were sweet
John y Martha la manzana y la pera eran dulces - Apertium
John y la manzana de Martha y la pera eran agradables - promt
John és Martha almája és körtéje édesek voltak - webforditas - John and Martha apple-his and pear-his sweet-s were.

errors:
Ignores the 's after John.- apertium
Ignores that both apple and pear belong to John and Martha - apertium & prompt

"John's and Martha's apple and pear were sweet" I would qualify as ungrammatical.

"John and Martha's apple and pear were sweet". The 's possessive in English is clitic and applies to the whole phrase.

Not that it makes a real difference to the translation quality. Co-ordinated noun phrases require extra rules that we haven't written in English→Spanish. This is not an issue of the power of the engine, but rather the number of rules. "Martha's apple was sweet" → "La manzana de Martha era dulce". The principle is the same. Incidentally, FreeLing (and therefore Matxin) also make a mess of parsing this).

I'm calculating how frequent this construction is in the Europarl corpus, and came up with this example:

The resources and capabilities of this country's agriculture and industry

Los recursos y capacidades de la agricultura de este país e industria

Los recursos y capacidades de la agricultura de este país e industria - promt

Ennek az országnak a mezőgazdaságának és iparának az erőforrásai és képességei - webforditas

It is still broken, but less so, again. The fixed length pattern for co-ordinated phrases after a genitive. The rule which fires above matches the pattern "DET NOM1 GENITIU NOM2", where we would need "DET NOM1 GENITIU NOM2 CC NOM3". - Francis Tyers 21:06, 11 April 2009 (UTC)

This pattern appears in approximately 0.4% of the two million sentences in the Europarl corpus. - Francis Tyers 22:02, 11 April 2009 (UTC)

John and Martha's apple and pear were sweet
John y la manzana de Martha y la pera eran agradables - promt
John y Martha la manzana y la pera eran dulces - apertium
John és Martha almája és körtéje édesek voltak

errors:

prompt ignores that John is also owner, ignores that apple and pear are together
apertium ignores both as owners

2 . example word combination

The ski jumpers were on the top of the hill
~~Los jerseys de esquí eran en la parte superior del cerro~~ Los saltadores de esquí eran en la parte superior del cerro - apertium
Los saltadores de esquí estaban en la cumbre de la colina - promt
A síugrók a domb tetõjén voltak webforditas, should be: A síugrók a hegy tetején voltak, The ski-jumpers the hill top-its-on were.

errors:
~~Apertium does not know "ski jumpers" combination.~~
webforditas generates false form tetõjén ->(should be)-> tetején

Here the Prompt output is much better. Adding the "ski jumper" multiword is easy, but that doesn't fix the problem with "en la parte superior", we'd have to look at the dictionary for that one. Again, this would be an incremental improvement, nothing "insurmountable". - Francis Tyers 21:33, 11 April 2009 (UTC)

You teached apertium? web testing shows: (http://xixona.dlsi.ua.es/apertium-www/index.php?id=translatetext):

Los jerseys de esquí eran en la parte superior del cerro Muki987 21:41, 11 April 2009 (UTC)

Let me update that (it updates every ~12 hours), but now I can do it manually. - Francis Tyers 21:46, 11 April 2009 (UTC)

PS. You can use the testing interface here. It also allows you to see the translation steps (see "print intermediate representation"). - Francis Tyers 19:23, 12 April 2009 (UTC)

3. example single word

He travelled in a nice coach
Viajó en un entrenador guapo - apertium
Él viajó en un entrenador agradable - promt
Egy jó edzõben utazott - webforditas - A good in-coach travelled, should be: Jó kocsiban utazott: good coach-in travelled.

Errors:

none recognizes, that coach is not only trainer, but also a coach, where I can sit. This shows, word selection (3.4 in docs) does not work reliable. Apertium also misses el. Besides that Egy is unnecessary and ugly in webforditas.Muki987 21:37, 11 April 2009 (UTC)

Word selection is not turned on, because the current module we have does not work. I mentioned this a couple of days ago (see above in the talk page). - Francis Tyers

I see. I will not test word selection. It is one of the most critical features generally, besides expression selection (multiword selection) Muki987 21:42, 11 April 2009 (UTC)

Yes, when dealing with distant languages it is one of the most important features. This is why we have it as #1 on our projects list for Google Summer of Code :) - Francis Tyers 21:45, 11 April 2009 (UTC)

Strongly agree. I would say, word selection is prio 1, expression selection immediately after word selection, and after these comes the rest. The mechanisms in apertium look good and well configurable. Muki987 21:53, 11 April 2009 (UTC)

My main complaint after word selection (when working on unrelated languages) is that we can't (so far) do recursive pattern matching. But we're working on it... - Francis Tyers 22:01, 11 April 2009 (UTC)

I start now installation and set up on my machine, I do not see any insurmountable problems. We shall see after setting up, how to continue. I must understand the mechanisms and handling first. If you need tests for word selection or expressions, just let me know, I am glad to help at testing. Muki987 21:53, 11 April 2009 (UTC)

Ok, thanks. I have been working on some tricks for limited rule-based lexical selection, you can find them in category:Lexical selection. These are just proofs of concept though. Working with unrelated languages is quite new for us (yes, even after 3-4 years!), most of our work goes on related languages. PS. I would be interested in seeing glosses for the Hungarian translations above (if you have time) - Francis Tyers 22:01, 11 April 2009 (UTC)

Done Muki987 18:48, 12 April 2009 (UTC)

Can you check if my changes to annotations are correct? - Francis Tyers 19:17, 12 April 2009 (UTC)

I could not find any changes. Where are they? The first was not gott, I corrected it. Muki987 19:28, 12 April 2009 (UTC)

I put them at the top of this section. - Francis Tyers 19:51, 12 April 2009 (UTC)

Found thanks. Also corrected then, the on was missing at the end. Muki987 20:03, 12 April 2009 (UTC)

Corpora

You can also add Hungarian Wikipedia to that list. We have some scripts for processing it too, see Calculating coverage. - Francis Tyers 11:20, 12 April 2009 (UTC)

Hunspell for generation

I get the same error that you posted to the list "NO DATA". The analysis works reasonably well. We would probably need to change the tagset, but that wouldn't be more than a few search/replace operations on the .aff and .dic files. "[" and "/" are reserved characters, and "<" cannot be embedded (for more information see Apertium stream format). In terms of analysis it is quite unfortunate that it doesn't do tokenise-as-you-analyse (as lttoolbox does), but it can probably be adapted to do this. - Francis Tyers 19:12, 12 April 2009 (UTC)

Yes, here some analysis results with hunpos:

en@anonymous:~/tmp/download/forditas/hunpos/hunpos-1.0-linux$ ./hunpos-tag ../hu_szeged_kr.model
model loaded
tagger compiled
holnap
elmegyek
moziba
.

holnap  ADV
elmegyek        VERB<PERS<1>>
moziba  NOUN<CAS<ILL>>
.       PUNCT

Márta
és
János
almái
és
körtéi
édesek
.

Márta   NOUN
és      CONJ
János   NOUN
almái   NOUN<PLUR><POSS>
és      CONJ
körtéi  NOUN<PLUR><POSS>
édesek  ADJ<PLUR>
.       PUNCT

Jó
kocsiba
ülj
!

Jó      ADJ
kocsiba NOUN<CAS<ILL>>
ülj     VERB<SUBJUNC-IMP><PERS<2>>
!       PUNCT

Tagset name change is absolutely no problem. I am very experienced in awk, and such changes are done in minutes.

I got on the Oo languagetool list from Német László 2 weeks ago working analysis results. The problem is with encoding of dictionary, it must be 8 bit. Looking for that. Muki987 19:53, 12 April 2009 (UTC)

Yes, I found that if I switched to an 8bit terminal the analysis worked fine. I think it might be something to do with the encoding of the aff/dic files for Hungarian (which seem to be in 8bit). I was intending to get around to change it, but didn't try it yet.

PS. Here is how I imagine Apertium tags:

 elmegyek  VERB<PERS<1>>                    →  ^elmegyek/el<VERB><PRES><PERS1>$
 moziba    NOUN<CAS<ILL>>                   →  ^moziba/mozi<NOUN><ILL>$
 almái     NOUN<PLUR><POSS>                 →  ^almái/alma<NOUN><PLUR><POSS>$
 édesek    ADJ<PLUR>                        →  ^édesek/édes<ADJ><PLUR>$
 kocsiba   NOUN<CAS<ILL>>                   →  ^kocsiba/kocsi<NOUN><ILL>$
 ülj       VERB<SUBJUNC-IMP><PERS<2>>       →  ^ülj/ül<VERB><SUBJ_IMP><PERS2>$

- Francis Tyers 20:15, 12 April 2009 (UTC)

How to switch to an 8 bit terminal (or how to convert the aff/dic files)? Muki987 21:06, 12 April 2009 (UTC)

In gnome-terminal, you can go to "Terminal -> Change character encoding". To convert the aff/dic files I'd use iconv, e.g.

$ cat morphdb_hu.aff | iconv -f latin2 -t utf-8 > morphdb_hu.aff.u8

- Francis Tyers 21:18, 12 April 2009 (UTC)

Actually, I just tried this and it doesn't seem to work. - Francis Tyers 22:41, 12 April 2009 (UTC)

Got the solution: download the last hu_HU.aff and hu_HU.dic from magyarispell.sf.net, and they work fine:

en@anonymous:~/program/humorph$ ls
analyze  hu_HU.aff  morphdb_hu.aff     morphdb_hu.dic
chmorph  hu_HU.dic  morphdb_hu.aff.u8  morphdb_hu.dic.u8
en@anonymous:~/program/humorph$ echo program | ./chmorph hu_HU.aff hu_HU.dic /dev/stdin NOM ACC
programot

en@anonymous:~/program/humorph$ echo programot | ./analyze hu_HU.aff hu_HU.dic /dev/stdin
> programot
analyze(programot) =  st:program po:noun ts:NOM is:ACC
stem(programot) = program

Could you give me a direct link to those files? I couldn't find them — my Hungarian really is non-existent :( - Francis Tyers 20:44, 16 April 2009 (UTC)

Slowness is because of affix tables and also flags of dic words are much more complicated, than for prefix languages like English, Spanish etc. Read in is much slower, however working is the same speed. For example Hungarian spell checking has the same speed as German or English spell checking. Start up is slower for Hungarian dictionaries.

Muki987 20:01, 15 April 2009 (UTC)

Ok, I'll check this out when I get home. Is there a binary format for the .aff/.dic files? Perhaps this might speed it up? - Francis Tyers 08:16, 16 April 2009 (UTC)

No. Even if there were, the amount of data is much higher for Hungarian, than for prefix languages. To read in 55 MB is slower than to read in 3 MB. Muki987 08:42, 16 April 2009 (UTC)

Yes, but there are ways to reducing the size. For example, a compiled Apertium dictionary typically takes up ~10% of the size due to the re-use of suffixes. E.g. if you have 'house +s -> house +n+pl' , 'computer +s -> 'computer +n+pl' , 'cat +s -> 'cat +n+pl' you only need to store the part '+s -> +n+pl' once. Also, parsing binary files can be quicker than parsing text files. - Francis Tyers 09:02, 16 April 2009 (UTC)

Yes, I agree with you. I shall propose to add an optional translation phase in hunmporph group. For example to write all structures after having read in into a file "froze them" and then, when called the next time, simply read in the frozen data. Muki987 09:13, 16 April 2009 (UTC)

Speed

Having checked chmorph source, there is no need for speed optimization. The program creates a class hunspell, and uses its generate method to get the morphologica data. We- as the translator application- can simply link libhunspell to our application, create the hunspell class once (that's expensive = slow), and then use that class for our translation procedure until we are finished. What do you think? Muki987 20:15, 16 April 2009 (UTC)

It is fine but it will be better for translating long documents than short ones, as the apertium pipeline loads the data each time. PS. I got Matxin to build!! - Francis Tyers 20:29, 16 April 2009 (UTC)

I use hunspell every day for all kinds documents, no load slowness problem at all. We shall see when we work with the translation system, I am optimistic. Muki987 20:51, 16 April 2009 (UTC)

Ok, great. - Francis Tyers 22:11, 22 April 2009 (UTC)

I saw, you documented Matxin, I'll follow the suggestions asap. Thanks! Muki987 20:51, 16 April 2009 (UTC)

POS taggers

I am looking for tagger, that best suits for English-Hungarian and German-Hungarian. I tested treetagger (http://www.ims.uni-stuttgart.de/projekte/corplex/TreeTagger/), freeling and apertium. Treetagger is not open source, and fails to show lemmas for German, however at the moment the only working German tagger. It is very fast.

I think there are corpora tagged for German, so it shouldn't be hard to train any tagger. I'll have to go looking. There are also morphological resources for German lying around somewhere, it is just a matter of changing them to Apertium or Freeling format. - Francis Tyers 22:06, 22 April 2009 (UTC)

Also see apertium-de-en in the incubator. There is a dictionary partially converted, morphological analysis for German here. - Francis Tyers 22:13, 22 April 2009 (UTC)

In my opinion apertium gives a more stable and more detailed analysis than freeling.

This was because you were using freeling "tagged" output, but apertium "analysed" output. See below. - Francis Tyers 22:06, 22 April 2009 (UTC)

Freeling

Henry and Martha went home when they saw an aeroplane fly through the sky.
Henry henry NN
and and CC
Martha martha NP
went go VBD
home home NN
when when NN
they they PRP
saw see VBD
an an DT
aeroplane aeroplane NN
fly fly VBP
through through IN
the the DT
sky sky NN
. . Fp

** OutputFormat=morfo in en.cfg **

$ echo "Henry and Martha went home when they saw an aeroplane fly through the sky." | ./analyzer -f ../../data/config/en.cfg
Henry henry NNP 1
and and CC 0.999723 and JJ 0.000163167 and NN 0.000114217
Martha martha NNP 1
went go VBD 0.499505 wend VBD 0.499505 wend VBN 0.00098912
home home NN 0.953125 home RB 0.046875
when when WRB 0.999667 when IN 0.000333111
they they PRP 1
saw see VBD 0.947674 saw NN 0.0523256
an an DT 1
aeroplane aeroplane NN 1
fly fly VB 0.698718 fly VBP 0.211538 fly NN 0.0833333 f RB 0.00641026
through through IN 0.2 through JJ 0.2 through RB 0.2  through 0.2 RP RP 0.2
the the DT 0.9998 the JJ 0.000121792 the NN 3.72143e-05 the VB 2.02987e-05 the VBP 2.02987e-05
sky sky NN 1
. . Fp 1

** OutputFormat=tagged in en.cfg **

$ echo "Henry and Martha went home when they saw an aeroplane fly through the sky." | ./analyzer -f ../../data/config/en.cfg
Henry henry NNP 1
and and CC 0.999723
Martha martha NNP 1
went go VBD 0.499505 wend VBD 0.499505
home home NN 0.953125
when when WRB 0.999667
they they PRP 1
saw see VBD 0.947674
an an DT 1
aeroplane aeroplane NN 1
fly fly VB 0.698718
through RP RP 0.2
the the DT 0.9998
sky sky NN 1
. . Fp 1

** OutputFormat=dep in en.cfg **

echo "Henry and Martha went home when they saw an aeroplane fly through the sky." | ./analyzer -f ../../data/config/en.cfg

grup-n/top/(Henry henry NNP -) [
  CC(and)/modnorule/(and and CC -)
  grup-n/modnorule/(Martha martha NNP -)
  verb/modnorule/(went go VBD -)
  grup-n/modnorule/(home home NN -)
  WRB(when)/modnorule/(when when WRB -)
  PRP(they)/modnorule/(they they PRP -)
  verb/modnorule/(saw see VBD -)
  sn/modnorule/(aeroplane aeroplane NN -) [
    DT/modnorule/(an an DT -)
  ]
  verb/modnorule/(fly fly VB -)
  RP(through)/modnorule/(through RP RP -)
  sn/modnorule/(sky sky NN -) [
    DT/modnorule/(the the DT -)
  ]
  Fp(.)/modnorule/(. . Fp -)
]

Apertium

en@anonymous:~/tmp/download/forditas/apertium-en-es-0.6$ apertium -d . en-es-anmor /tmp/x
^Henry/Henry<np><ant><m><sg>$ ^and/and<cnjcoo>$ ^Martha/Martha<np><ant><f><sg>$ ^went/go<vblex><past>$ ^home/home<adv>/home<n><sg>$^,/,<cm>$ 
^when/when<cnjadv>/when<adv><itg>$ ^they/prpers<prn><subj><p3><mf><pl>$ ^saw/saw<n><sg>/saw<vblex><inf>/saw<vblex><pres>/see<vblex><past>$^,/,<cm>$ 
^that/that<cnjsub>/that<det><dem><sg>/that<prn><tn><mf><sg>/that<rel><an><mf><sp>$ ^an/a<det><ind><sg>$ ^aeroplane/*aeroplane$ ^flied/*flied$ 
^on/on<adv>/on<pr>$ ^the/the<det><def><sp>$ ^sky/sky<n><sg>$^./.<sent>$^./.<sent>$

$  echo "Henry and Martha went home when they saw an aeroplane fly through the sky."  | lt-proc en-ca.automorf.bin 
^Henry/Henry<np><ant><m><sg>$ ^and/and<cnjcoo>$ ^Martha/Martha<np><ant><f><sg>$ ^went/go<vblex><past>$ ^home/home<adv>/home<n><sg>$ 
^when/when<cnjadv>/when<adv><itg>$ ^they/prpers<prn><subj><p3><mf><pl>$ ^saw/saw<n><sg>/saw<vblex><inf>/saw<vblex><pres>/see<vblex><past>$ 
^an/a<det><ind><sg>$ ^aeroplane/aeroplane<n><sg>$ ^fly/fly<n><sg>/fly<vblex><inf>/fly<vblex><pres>$ ^through/through<pr>$ 
^the/the<det><def><sp>$ ^sky/sky<n><sg>$^./.<sent>$

$  echo "Henry and Martha went home when they saw an aeroplane fly through the sky."  | lt-proc en-ca.automorf.bin | apertium-tagger -g en-ca.prob 
^Henry<np><ant><m><sg>$ ^and<cnjcoo>$ ^Martha<np><ant><f><sg>$ ^go<vblex><past>$ ^home<adv>$ 
^when<adv><itg>$ ^prpers<prn><subj><p3><mf><pl>$ ^see<vblex><past>$ 
^a<det><ind><sg>$ ^aeroplane<n><sg>$ ^fly<n><sg>$ ^through<pr>$ 
^the<det><def><sp>$ ^sky<n><sg>$^.<sent>$

The question

For me apertium seems to be the better one. Do you know any example, where freeling does a better job? Thanks, Muki987 21:07, 22 April 2009 (UTC)

The benefit with freeling is that it does parsing too.. see above examples. - Francis Tyers 22:02, 22 April 2009 (UTC)

Is your en.cfg file of freeling the standard one I use? If not, could you please email it to me? Thanks. Muki987 22:43, 22 April 2009 (UTC)

Yes, the standard one, just changing the output format. Francis Tyers 22:48, 22 April 2009 (UTC)

If there are differences between my version and your version, you can try changing the tagger from:

#### Tagger options
Tagger=relax

to:

#### Tagger options
Tagger=hmm

- Francis Tyers 07:33, 23 April 2009 (UTC)

dependency

Why should I have taken a raincoat and an umbrella while my aunt, living in Georgia, told, it will be nice weather.
grup-n/top/(Why why NN) [
  MD(should)/modnorule/(should should MD)
  NP(i)/modnorule/(I i NP)
  verb/modnorule/(taken take VBN) [
    VB*<have>/modnorule/(have have VBP)
  ]
  IN(a)/modnorule/(a a IN)
  grup-n/modnorule/(raincoat raincoat NN)
  CC(and)/modnorule/(and and CC)
  sn/modnorule/(umbrella umbrella NN) [
    DT/modnorule/(an an DT)
  ]
  IN(while)/modnorule/(while while IN)
  PP$(my)/modnorule/(my my PP$)
  grup-n/modnorule/(aunt aunt NN)
  Fc(,)/modnorule/(, , Fc)
  verb/modnorule/(living live VBG)
  IN(in)/modnorule/(in in IN)
  NP(georgia)/modnorule/(Georgia georgia NP)
  Fc(,)/modnorule/(, , Fc)
  verb/modnorule/(told tell VBD)
  Fc(,)/modnorule/(, , Fc)
  grup-n/modnorule/(it it NN)
  verb/modnorule/(be be VBP) [
    MD/modnorule/(will will MD)
  ]
  grup-n/modnorule/(weather weather NN) [
    adj/modnorule/(nice nice JJ)
  ]
  Fp(.)/modnorule/(. . Fp)
]

I get the above output when using freeling from my keyboard. However http://garraf.epsevg.upc.es/freeling/demo.php shows a very different picture. For example, top is "tell", Why is of type WRB, func is fast always modnorule, while on web output cmod, ncmod, ... How can I get the same output on my console output, as the web output is? Muki987 08:12, 23 April 2009 (UTC)

Which version are you using, SVN or 2.0? When I put the example below into the web interface I get the same result. - Francis Tyers 08:20, 23 April 2009 (UTC)

$ echo "Why should I have taken a raincoat and an umbrella while my aunt, living in Georgia, said that the weather would be nice." | \
./analyzer -f ../../data/config/en.cfg
DEPENDENCIES: NO HEAD Found!!! Check your chunking grammar and your dependency-building rules.
WRB(why)/top/(Why why WRB -) [
  MD(should)/modnorule/(should should MD -)
  PRP(i)/modnorule/(I i PRP -)
  verb/modnorule/(taken take VBN -) [
    VB*<have>/modnorule/(have have VBP -)
  ]
  Z(a)/modnorule/(a 1 Z -)
  grup-n/modnorule/(raincoat raincoat NN -)
  CC(and)/modnorule/(and and CC -)
  sn/modnorule/(umbrella umbrella NN -) [
    DT/modnorule/(an an DT -)
  ]
  IN(while)/modnorule/(while while IN -)
  PRP$(my)/modnorule/(my my PRP$ -)
  grup-n/modnorule/(aunt aunt NN -)
  Fc(,)/modnorule/(, , Fc -)
  verb/modnorule/(living live VBG -)
  IN(in)/modnorule/(in in IN -)
  grup-n/modnorule/(Georgia georgia NNP -)
  Fc(,)/modnorule/(, , Fc -)
  verb/modnorule/(said say VBD -)
  IN(that)/modnorule/(that that IN -)
  sn/modnorule/(weather weather NN -) [
    DT/modnorule/(the the DT -)
  ]
  verb/modnorule/(be be VB -) [
    MD/modnorule/(would will MD -)
  ]
  adj/modnorule/(nice nice JJ -)
  Fp(.)/modnorule/(. . Fp -)
]

I use Freeling 1.5. Please copy your console output for this sentence to this page. Muki987 08:26, 23 April 2009 (UTC)

For which sentence? I'm using Freeling 2.0 - Francis Tyers 08:32, 23 April 2009 (UTC)

Why should I have taken a raincoat and an umbrella, when my aunt, living in Georgia, told, it will be nice weather. Muki987 08:46, 23 April 2009 (UTC)

Here it is, although it isn't really grammatical English. - Francis Tyers 08:50, 23 April 2009 (UTC)

$ echo "Why should I have taken a raincoat and an umbrella, when my aunt, living in Georgia, told, it will be nice weather." |  \
./analyzer -f ../../data/config/en.cfg
DEPENDENCIES: NO HEAD Found!!! Check your chunking grammar and your dependency-building rules.
WRB(why)/top/(Why why WRB -) [
  MD(should)/modnorule/(should should MD -)
  PRP(i)/modnorule/(I i PRP -)
  verb/modnorule/(taken take VBN -) [
    VB*<have>/modnorule/(have have VBP -)
  ]
  Z(a)/modnorule/(a 1 Z -)
  grup-n/modnorule/(raincoat raincoat NN -)
  CC(and)/modnorule/(and and CC -)
  sn/modnorule/(umbrella umbrella NN -) [
    DT/modnorule/(an an DT -)
  ]
  Fc(,)/modnorule/(, , Fc -)
  WRB(when)/modnorule/(when when WRB -)
  PRP$(my)/modnorule/(my my PRP$ -)
  grup-n/modnorule/(aunt aunt NN -)
  Fc(,)/modnorule/(, , Fc -)
  verb/modnorule/(living live VBG -)
  IN(in)/modnorule/(in in IN -)
  grup-n/modnorule/(Georgia georgia NNP -)
  Fc(,)/modnorule/(, , Fc -)
  verb/modnorule/(told tell VBD -)
  Fc(,)/modnorule/(, , Fc -)
  PRP(it)/modnorule/(it it PRP -)
  verb/modnorule/(be be VB -) [
    MD/modnorule/(will will MD -)
  ]
  grup-n/modnorule/(weather weather NN -) [
    adj/modnorule/(nice nice JJ -)
  ]
  Fp(.)/modnorule/(. . Fp -)
]

Yes, exactly. The same as for freeling 1.5. If we compare this to the web screen output, we see, this is very different from that. For example, here all words ar modnorule, except the top one, on the web screen there are different func types.

It could be a difference between SVN (or Freeling 2.1) and Freeling 2.0. The person to ask would be Lluís Padró, I've emailed you his email address. - Francis Tyers 09:07, 23 April 2009 (UTC)

Also the tree itself is completely different. Top of it is "told" on web, here "why". And so on.... Why that? Muki987 09:00, 23 April 2009 (UTC)

I'm not sure, it could have something to do with it not being a grammatical sentence in English ? - Francis Tyers 09:07, 23 April 2009 (UTC)

OK. Change it to a grammatical one, and you still see the big differences. Why? Muki987 09:22, 23 April 2009 (UTC)

I entered this same question on freeling forum, however it seems to be dead.

http://garraf.epsevg.upc.es/freeling/index.php?option=com_simpleboard&Itemid=55&func=view&catid=3&id=883#883 Muki987 09:22, 23 April 2009 (UTC)

They sometimes take a while to reply. I think it might be a holiday up there too (La Diada de Sant Jordi) - Francis Tyers 09:51, 23 April 2009 (UTC)

A more standard rendering of the sentence below:

$ echo "Why should I have taken a raincoat and an umbrella? My aunt who lives in Georgia said that the weather would be nice." | \
 ./analyzer -f ../../data/config/en.cfg
DEPENDENCIES: NO HEAD Found!!! Check your chunking grammar and your dependency-building rules.
DEPENDENCIES: NO HEAD Found!!! Check your chunking grammar and your dependency-building rules.
WRB(why)/top/(Why why WRB -) [
  MD(should)/modnorule/(should should MD -)
  PRP(i)/modnorule/(I i PRP -)
  verb/modnorule/(taken take VBN -) [
    VB*<have>/modnorule/(have have VBP -)
  ]
  Z(a)/modnorule/(a 1 Z -)
  grup-n/modnorule/(raincoat raincoat NN -)
  CC(and)/modnorule/(and and CC -)
  sn/modnorule/(umbrella umbrella NN -) [
    DT/modnorule/(an an DT -)
  ]
  Fit(?)/modnorule/(? ? Fit -)
]

PRP$(my)/top/(My my PRP$ -) [
  grup-n/modnorule/(aunt aunt NN -)
  WP(who)/modnorule/(who who WP -)
  verb/modnorule/(lives live VBZ -)
  IN(in)/modnorule/(in in IN -)
  grup-n/modnorule/(Georgia georgia NNP -)
  verb/modnorule/(said say VBD -)
  IN(that)/modnorule/(that that IN -)
  sn/modnorule/(weather weather NN -) [
    DT/modnorule/(the the DT -)
  ]
  verb/modnorule/(be be VB -) [
    MD/modnorule/(would will MD -)
  ]
  adj/modnorule/(nice nice JJ -)
  Fp(.)/modnorule/(. . Fp -)
]

Although the analysis is still a bit of a mystery. They both seem to come out fine in the web interface. - Francis Tyers 10:00, 23 April 2009 (UTC)

The SVN gives:


$ echo "Why should I have taken a raincoat and an umbrella, my aunt who lives in Georgia said that the weather would be nice." | \
 ./analyzer -f ~/source/FREELING/local/share/FreeLing/config/en.cfg 
sub-cl/top/(Why why WRB -) [
  mod-chunk/modnomatch/(should should MD -)
  sv/cmod/(taken take VBN -) [
    vb-have/aux/(have have VBP -)
    sn-chunk/ncsubj/(I i PRP -)
    sn-coor/dobj/(and and CC -) [
      sn-chunk/conj/(raincoat raincoat NN -) [
        DT/det/(a a DT -)
      ]
      sn-chunk/conj/(umbrella umbrella NN -) [
        DT/det/(an a DT -)
      ]
    ]
  ]
  sf-brk/modnomatch/(, , Fc -)
  sn-chunk/modnomatch/(aunt aunt NN -) [
    PRP$/ncmod-poss/(my my PRP$ -)
    rel-cl/cmod/(who who WP -) [
      rel/ccomp/(lives live VBZ -) [
        sp-chunk/ncmod/(in in IN -) [
          sv/cmod/(said say VBD -) [
            n-chunk/ncsubj/(Georgia georgia NNP -)
          ]
        ]
      ]
    ]
  ]
  sub-cl/modnomatch/(that that IN -) [
    sv/cmod/(be be VB -) [
      mod-chunk/aux/(would would MD -)
      sn-chunk/ncsubj/(weather weather NN -) [
        DT/det/(the the DT -)
      ]
      attrib/ncmod/(nice nice JJ -)
    ]
  ]
  st-brk/modnomatch/(. . Fp -)
]

Which is much better. - Francis Tyers 11:06, 23 April 2009 (UTC)

The equivalent apertium output (although slightly mangled for translation en→ca) would be:

^Adv<adv><itg>{^why<adv><itg>$}$ 
^inf<SV><inf><PD><ND>{^should<3>$}$ 
^prnsubj<SN><p1><mf><sg>{^prpers<prn><p1><mf><sg>$}$ 
^have_pp<SV><vblex><pri><PD><ND>{^have<vbhaver><3><4><5>$ ^take<vblex><pp><m><sg>$}$ 
^det_nom<SN><DET><GD><sg>{^a<det><ind><3><4>$ ^coat<n><4>$}$ 
^cnj<cnjcoo>{^and<cnjcoo>$}$ 
^det_nom<SN><DET><GD><sg>{^a<det><ind><3><4>$ ^umbrella<n><4>$}$
^coma<cm>{^,<cm>$}$ 
^det_nom<SN><DET><GD><sg>{^my<det><pos><3><4><sp>$ ^aunt<n><4>$}$ 
^reladj<REL><an><mf><sp>{^who<rel><an><3><4>$}$ 
^verbcj<SV><vblex><pri><p3><sg>{^live<vblex><3><4><5>$}$ 
^pr<PREP>{^in<pr>$}$ 
^nom<SN><UNDET><sg>{^Georgia<np><loc><4>$}$ 
^verbcj_perif<SV><reporting><ifip><PD><ND>{^anar<vaux><4><5>$ ^say<vblex><inf>$}$ 
^cnj<cnjsub>{^that<cnjsub>$}$ 
^det_nom<SN><DET><GD><sg>{^the<det><def><3><4><sp>$ ^weather<n><4>$}$ 
^verbcj<SV><vbser><cni><PD><ND>{^be<vbser><3><4><5>$}$ 
^adj<SA><GD><ND>{^nice<adj><2><3>$}$
^punt<sent>{^.<sent>$}$

We can actually collapse det_nom cnj det_nom into e.g. det_nom_cnj_det_nom, but probably collapsing the relatives would be harder. The benefit of the FreeLing output is that 'my aunt who lives in Georgia' is expressed as one chunk that can be moved. Both ways have their benefits, for hu→en I'd go with Apertium and for en→hu probably FreeLing/Matxin or a hybrid of Apertium/Matxin. - Francis Tyers 11:27, 23 April 2009 (UTC)

Versions

I do not understand, why are you using freeling 2.0 or 2.1, when for matxin is clearly 1.5 suggested.

I also do not understand, how can be that fundamental differences between freeeling 2.1 (the svn version) and 1.5, that I use. I get following :

Why should I have taken a raincoat and an umbrella, my aunt who lives in Georgia said that the weather would be nice.
grup-n/top/(Why why NN) [
  MD(should)/modnorule/(should should MD)
  NP(i)/modnorule/(I i NP)
  verb/modnorule/(taken take VBN) [
    VB*<have>/modnorule/(have have VBP)
  ]
  IN(a)/modnorule/(a a IN)
  grup-n/modnorule/(raincoat raincoat NN)
  CC(and)/modnorule/(and and CC)
  sn/modnorule/(umbrella umbrella NN) [
    DT/modnorule/(an an DT)
  ]
  Fc(,)/modnorule/(, , Fc)
  PP$(my)/modnorule/(my my PP$)
  grup-n/modnorule/(aunt aunt NN)
  WP(who)/modnorule/(who who WP)
  verb/modnorule/(lives live VBZ)
  IN(in)/modnorule/(in in IN)
  NP(georgia)/modnorule/(Georgia georgia NP)
  verb/modnorule/(said say VBD)
  IN(that)/modnorule/(that that IN)
  sn/modnorule/(weather weather NN) [
    DT/modnorule/(the the DT)
  ]
  verb/modnorule/(be be VBP) [
    MD/modnorule/(would would MD)
  ]
  adj/modnorule/(nice nice JJ)
  Fp(.)/modnorule/(. . Fp)
]

Which is fundamentally different from your output. If for matxin 1.5 is the valid version, why are you using the svn version of freeling? Muki987 13:00, 23 April 2009 (UTC)

The problem is that the people who develop Matxin are shy of committing to their SVN repository. For internal development they are using freeling 2.0/2.1 and lttoolbox 3.1, just they haven't committed it to their SVN yet (you can see the last commit is from sometime in November!). So, for testing we should try the version of FreeLing that they are using. I've sent them an email asking if they can send us a snapshot of what they have locally. - Francis Tyers 13:19, 23 April 2009 (UTC)

PS. Lluís responded on the FreeLing forum. - Francis Tyers 13:37, 23 April 2009 (UTC)

The pont is, a really good dep. tagger sees, that the sentence has 3 almost independent parts:

1. Why should I have taken a raincoat and an umbrella (where raincoat and umbrella belong together)
2. My aunt lives in Georgia, told
3. It will be nice weather

Muki987 11:54, 24 April 2009 (UTC)

Yep, exactly :) - Francis Tyers 12:22, 24 April 2009 (UTC)

Dog us

http://www.smh.com.au/cgi-bin/common/popupPrintArticle.pl?path=/articles/2008/09/30/1222651083043.html

I got it here: http://sujitpal.blogspot.com/2008/11/ir-math-in-java-hmm-based-pos.html

Do you know the expression: "Failure will dog us"? Does this mean, failure will follow us?

Thanks, Muki987 10:42, 25 April 2009 (UTC)

Yes, this is an expression that can be used. It means that "Failure will follow us and cause us trouble". You can probably disambiguate with a rule that says "choose infinitive if -1 modal and +1 personal pronoun". - Francis Tyers 10:45, 25 April 2009 (UTC)

Thanks. Seems to be used quite seldom in real life. Shows, that dog can be a verb, if used in a verb environment. He dogs a cat, for example. Or the dogs dogged a cat. Or the wolves dogged the rabbit. Or my big mistake dogged me 10 years long. At least the blogspot indicates this. Muki987 11:06, 25 April 2009 (UTC)

Yes, but outside certain fixed semantic environments, it sounds odd.

*He dogs a cat
*The dogs dogged a cat
The wolves dogged the rabbit. 
*My big mistake dogged me for 10 years long. 
My mistake dogged me for ten years.

The ones marked with '*' are not wrong syntactically, but I would say they sound very strange. - Francis Tyers 11:14, 25 April 2009 (UTC)

What is wrong with

The wolves dogged the rabbit. 
My mistake dogged me for ten years.

?

How to write them syntactically correctly? Muki987 12:21, 25 April 2009 (UTC)

No those are correct, I mean they sound good. - Francis Tyers 12:33, 25 April 2009 (UTC)

Thanks, Muki987 12:36, 25 April 2009 (UTC)

Fixes

It looks great, I noticed a couple of issues, although I haven't tested it, try making these edits and seeing how the result turns out. - Francis Tyers 05:36, 30 April 2009 (UTC)

Hi regan,

The expressions semi-crazed rants and you are exactly the wrong sort of person to work with translation in any form are your expressions, you try to apply to others. regan, you think, something entitles you to use that kind of language?

regan, your semi-crazed rants in all subjects are rather primitive, and not amusing at all.

What do you think, who you are, that you would like to decide, who may say what? You believe, you are a soviet commissar? Or a kapo of the barrack? Have you let check your mental state? If you behave like that, you, regan are exactly the wrong sort of person to work with translation in any form.

Of course, present language killers and people idiotizers on the TV screens and in radios, newspapers and magazines written by idiots, foreign advertisers and similar state-supported criminals try to push foreign words, which is not good for any language, and makes tools, like wordnet necessary.

19:00, 17 May 2009 (UTC)

Dep analysis

I try to dependency analyse this sentence, since it is complicated enough:

1. I think that if you have an agenda that you want to push of this kind, then you are exactly the wrong sort of person to work with translation in any form.

Why looking at that, I think, that is grammatically incorrect and erroneous.

2. I think that if you have an agenda that you want to push of this kind, then...

Should not that be:

3. I think that if you have an agenda of this kind, that you want to push, then... ?

Are both 2 and 3 correct, or is 3 wrong? With other words, is "push of" a valid structure?

Both variants sound ok to my ears, although I would say:

"I think that if you have an agenda of this kind that you want to push, ..."

(without extra comma) - Francis Tyers 16:10, 20 May 2009 (UTC)

If the sentence is no good in the first form, is it still understandable?

of this kind is an attribute of the agenda. Let's assume, the attribute is blue.

4. I think that if you have a blue agenda, that you want to push, then...

5. I think that if you have an agenda that you want to push blue, then...

6. I think that if you have an agenda that you want to push, and that is blue, then...

I changed of this kind to blue. Is 5 in that form not completely bad? Is 6 ok? Does "of this kind" implicitely say: and that is of this kind?

Dependency analysis shows very different results, and if version 2 is correct, I report that to freeling- otherwise not.

If you have an agenda of this kind, that you want to push, then you are good. sub-adv/top/(If if IN -) [

 sv/modnorule/(have have VBP -) [
   sn-chunk/ncsubj/(you you PRP -)
   sn-chunk/dobj/(agenda agenda NN -) [
     DT/det/(an a DT -)
   ]
 ]
 sp-chunk/modnorule/(of of IN -) [
   sn-chunk/dobj/(kind kind NN -) [
     DT/det/(this this DT -)
   ]
 ]
 sf-brk/modnorule/(, , Fc -)
 rel-cl/modnorule/(that that WDT -) [
   rel/ccomp/(want want VBP -) [
     sn-chunk/ncsubj/(you you PRP -)
     sp-chunk/ncmod/(to to TO -) [
       sv/cmod/(push push VB -)
     ]
   ]
 ]
 sf-brk/modnorule/(, , Fc -)
 claus/modnorule/(are be VBP -) [                <----------------------------- are
   adv/cmod/(then then RB -)
   sn-chunk/ncsubj/(you you PRP -)
   attrib/ncmod/(good good JJ -)
   st-brk/ta/(. . Fp -)
 ]

]

If you have an agenda that you want to push of this kind, then you are good. sub-adv/top/(If if IN -) [

 sv/modnorule/(have have VBP -) [
   sn-chunk/ncsubj/(you you PRP -)
   sn-chunk/dobj/(agenda agenda NN -) [
     DT/det/(an a DT -)
   ]
 ]
 claus/modnorule/(are be VBP -) [               <---------------------------- are
   sub-cl/modnomatch/(that that IN -) [
     sv/modnorule/(want want VBP -) [
       sn-chunk/ncsubj/(you you PRP -)
       sp-chunk/ncmod/(to to TO -) [
         sv/cmod/(push push VB -) [
           sp-chunk/ncmod/(of of IN -) [
             sn-chunk/dobj/(kind kind NN -) [
               DT/det/(this this DT -)
             ]
           ]
         ]
       ]
     ]
     sf-brk/modnorule/(, , Fc -)
   ]
   adv/cmod/(then then RB -)
   sn-chunk/ncsubj/(you you PRP -)
   attrib/ncmod/(good good JJ -)
   st-brk/ta/(. . Fp -)
 ]

]

In my opinion the second cas is misinterpreted. What do you think?Muki987 19:05, 20 May 2009 (UTC)