Chunking - Revision history

Bech: Link to French page

2014-10-08T06:55:42Z

Link to French page

Bech: Category:Documentation in English

2011-09-03T18:26:46Z

Category:Documentation in English

194.187.151.201: /* External links */

2010-08-01T16:04:12Z

External links

Unhammer at 20:55, 17 April 2009

2009-04-17T20:55:45Z

Jacob Nordfalk at 03:13, 30 September 2008

2008-09-30T03:13:44Z

Jacob Nordfalk: /* See also */

2008-09-28T17:01:55Z

Jacob Nordfalk: /* Headline text */

2008-09-28T15:45:34Z

Headline text

202.63.240.9: /* Example */

2008-09-14T13:51:11Z

Example

Jacob Nordfalk: /* Now, a note. */

2008-09-11T09:14:04Z

Now, a note.

Francis Tyers at 05:46, 11 September 2008

2008-09-11T05:46:17Z

@@ Line 1: / Line 1: @@
+[[Fragmentation|En français]]
 {{TOCD}}
 ==Shallow transfer ==

@@ Line 101: / Line 101: @@
 [[Category:Documentation]]
 [[Category:Writing transfer rules]]
+[[Category:Documentation in English]]

@@ Line 95: / Line 95: @@
 ==External links==
 * [http://en.wikipedia.org/wiki/Chunking_(computational_linguistics) wikipedia]
-* [http://nltk.org/doc/en/chunk.html Chunking] (Natural Language Toolkit)
+* [http://nltk.googlecode.com/svn/trunk/doc/book/ch07.html Chunking] (Natural Language Toolkit)
 * [http://crfchunker.sourceforge.net/ CRFChunker] (Conditional Random Fields English Phrase Chunker)
 * [http://jtextpro.sourceforge.net/ JTextPro] (A Java-based Text Processing Toolkit)

← Older revision		Revision as of 20:55, 17 April 2009
Line 100:		Line 100:

	[[Category:Documentation]]		[[Category:Documentation]]
			[[Category:Writing transfer rules]]

@@ Line 1: / Line 1: @@
 {{TOCD}}
+==Shallow transfer ==
-==Short intro==
+Shallow transfer means there is no parse trees (which are used in "deep transfer").
-<pre>
+But then how is reordering of the phrase then going to happen?
-     jacobn>	But really I have a big problem about all this "shallow transfer".
+By chunking (in three-stages): First we reorder words in the chunk, then we reorder chunks.
-	<spectie>	shallow transfer = no parse trees
+* first, we match phrase patterns, like adj+noun or adj+adj+noun
-	<spectie>	basically
+* from these, we make a 'pseudo lemma', with a tag containing the type - normally 'SN' (noun phrase) or SV (verb phrase)
-	<jimregan2>	yep
+* then, we translate based on these pseudo words breaking the language down to its bare essentials, basically
-	<jacobn>	HOW is reordering of the phrase then going to happen!!
-     jimregan2> we use chunking
+==Chunking explained==
-    <jimregan2> first we reorder words in the chunk, then we reorder chunks
-	<jacobn>	Pls tell me 'bout it or point to a web page
-	<jimregan2>	it's easy enough
-	<jimregan2>	first, we match phrase patterns
-	<jimregan2>	adj+noun
-	<jimregan2>	adj+adj+noun
-	<jimregan2>	from these, we make a 'pseudo lemma', with a tag containing the type - normally 'SN' (noun phrase) or SV (verb phrase)
-	<jimregan2>	then, we translate based on these pseudo words
-	<jimregan2>	breaking the language down to its bare essentials, basically
-	<jimregan2>	at the moment, I'm taking the 'hard wired' parts of the english to spanish chunker, and adapting it for french
-	<jimregan2>	changing 'más' to 'plus' in a macro, etc.
-      <spectie> but the chunks cannot be recursive
-</pre>
-==Longer intro==
 Our rules are based on the source language patterns; we need to use
 chunking for f.ex. English-Esperanto, so the first task is to identify those
 patterns.
+<pre>
 the man sees the girl
+</pre>
 Chunking:
+<pre>
 SN(the man) SV(sees) SN(the girl)
+</pre>
 (Normally, in English those are 'NP' and 'VP' for 'noun phrase' and
@@ Line 48: / Line 32: @@
 tag the chunks with whatever information will be useful later.
+So the chunks are normally given a 'pseudo lemma' that matches the
-So
-The chunks are normally given a 'pseudo lemma' that matches the
 pattern that matched them ('the man', 'my friend' will be put in a
 chunk called 'det_nom', etc.), the first tag added is the phrase type;
@@ Line 58: / Line 40: @@
 So, taking 'big cat', we would get:
+<pre>
 ^adj_nom<SN><sg><CD>{^granda<ad><2><3>$ ^kato<n><2><3>$}$
+</pre>
-(the numbers in the lemma tags mean 'take the information from chunk
+The numbers in the lemma tags (here <2><3>) mean 'take the information from chunk
-tag number #', CD means 'Case to be Determined - it's not fully
+tag number #'. CD means 'Case to be Determined (it's not fully
 established, as GD and ND are, but it's the logical one to use).
-so, with a simple SN SV SN, we can have a rule that outputs the same
+So, with a simple SN SV SN, we can have a rule that outputs the same
 things in the same order, but changes the 'CD' of SN number 1 to
-'nom', and of SN number 2 to 'acc'. All very simple.
+'nom', and of SN number 2 to 'acc'.
-===Now, a note.===
-The next kind of thing we should think about is the type of sentence
-part that goes like this:
-'the man you saw'
-'the man the girl saw'
-I don't know if we have to change word order here - probably not - but
-the nominative and accusative are SNs 2 and 1 respectively.
-But think about this:
-'the man my brother became'
-Adding accusative here is wrong, so what can we do about it? Not much.
-Maybe in this specific instance, sure, but generally, we can only take
-the common cases and hope for the best. There's been plenty of work
-into statistical parsing, subject identification, etc., but it's still
-not much better than picking the common cases, and hoping for the
-best.
-This is why we always tell people to have their translations checked
-by a native speaker :)
 ==Example==
@@ Line 128: / Line 88: @@
 ==See also==
+* [[Chunking: A full example]]
 * [[Apertium stream format#Chunks]]
 * [[Preparing to use apertium-transfer-tools]]
-* [[Chunking: A full example]]
+* [[English and Esperanto]]
 ==External links==

← Older revision		Revision as of 15:45, 28 September 2008
Line 138:		Line 138:

	[[Category:Documentation]]		[[Category:Documentation]]

	== Headline text ==

@@ Line 99: / Line 99: @@
 </pre>
-becomes after disambiguation
+becomes after tagger disambiguation
 <pre>
 ^prpers<prn><subj><p1><mf><sg>$
@@ Line 114: / Line 114: @@
 </pre>
-and transformed by rule SN SV SN -> SV SV SN<acc>
+and transformed by rule SN SV SN<nom> -> SN SV SN<acc>
 <pre>
 ^prnpers<SN><p1><mf><sg>{^prpers<prn><subj><2><3><4>$}$
@@ Line 120: / Line 120: @@
 ^nom<SN><sg><acc>{^signalo<n><2><3><4>$}$.
 </pre>
+Note how the chunk has now tags nom<SN><sg><acc> and therefore ^signalo<n><2><3><4>$ gets these tags when unchunking:
-and then unchunked
 <pre>
 ^prpers<prn><subj><p1><mf><sg>$