User:Khannatanmai/Secondary info apertium stream format - Revision history

Khannatanmai at 09:25, 17 July 2020

2020-07-17T09:25:48Z

Khannatanmai: /* Another one */ Added weights in examples

2020-06-08T05:52:36Z

Another one: Added weights in examples

Khannatanmai: /* Thoughts */

2020-06-08T05:07:11Z

Thoughts

Khannatanmai: /* Another one */

2020-06-08T05:05:29Z

Another one

Khannatanmai: /* Features */

2020-06-07T19:58:26Z

Features

Khannatanmai: /* Thoughts */

2020-06-07T19:52:42Z

Thoughts

Khannatanmai at 19:52, 7 June 2020

2020-06-07T19:52:26Z

Khannatanmai: /* Another one */

2020-06-07T19:37:26Z

Another one

Khannatanmai: /* Another one */

2020-06-07T19:11:09Z

Another one

Khannatanmai: /* Another one */

2020-06-07T18:45:35Z

Another one

← Older revision		Revision as of 09:25, 17 July 2020
Line 1:		Line 1:
			<strong style="color:maroon;font-size:1.5em;>The secondary tags project was shelved as the need for reading specific secondary information wasn't established. This work continued as LU-bound secondary information using [[User:Khannatanmai/Wordbound_blanks \| wordbound blanks]]. </strong>

	Original Proposal: Modifying the apertium stream format and eliminating dictionary trimming: [[User:Khannatanmai/GSoC2020Proposal_Trimming]]		Original Proposal: Modifying the apertium stream format and eliminating dictionary trimming: [[User:Khannatanmai/GSoC2020Proposal_Trimming]]

@@ Line 68: / Line 68: @@
 Special tag in the LU that will be ignored by FSTs. Format: <!..>
+W: weight
 <pre>
-^«/«<lquot><!1>$^Ajawxel/Ajawaxel<n><!2>$^,/,<cm><!3>$ ^ri/ri<det><!4>$ ^in/in<prn><pers><p1><sg><!5>/in<prn><pro><p1><sg><!6>$ ^kinwilo/<impf><o_sg3><s_sg1>il<v><tv><!7>+o<mark><!8>$ ^che/chi<pr><!9>+<px3sg>ech<n><rel><!10>$ ^ri/ri<det><!11>$ ^at/at<prn><pers><p2><sg><!12>/at<prn><pro><p2><sg><!13>$^,/,<cm><!14>$ ^at/at<prn><pers><p2><sg><!15>/at<prn><pro><p2><sg><!16>$ ^jun/jun<adj><!17>/jun<n><!18>/jun<num><!19>$ ^qʼaxal/qʼaxal<adj><!20>$ ^utzij/<px3sg>tzij<n><!21>$ ^ri/ri<det><!22>$ ^Dyos/Dyos<n><!23>$
+^«/«<lquot><!1>$[{W:0.987}]^Ajawxel/Ajawaxel<n><!2>$[{W:0.587}]^,/,<cm><!3>$[{W:0.953}] ^ri/ri<det><!4>$[{W:0.675}] ^in/in<prn><pers><p1><sg><!5>/in<prn><pro><p1><sg><!6>$[{sl_ids:5; W:0.203}{sl_ids:6; W:0.457}] ^kinwilo/<impf><o_sg3><s_sg1>il<v><tv><!7>+o<mark><!8>$[{sl_ids:7; W:0.621}{sl_ids:8; W:0.760}] ^che/chi<pr><!9>+<px3sg>ech<n><rel><!10>$[{sl_ids:9; W:0.832}{sl_ids:10; W:0.562}] ^ri/ri<det><!11>$[{W:0.675}] ^at/at<prn><pers><p2><sg><!12>/at<prn><pro><p2><sg><!13>$[{sl_ids:12; W:0.709}{sl_ids:13; W:0.302}]^,/,<cm><!14>$[{W:0.953}] ^at/at<prn><pers><p2><sg><!15>/at<prn><pro><p2><sg><!16>$[{sl_ids:15; W:0.709}{sl_ids:16; W:0.302}] ^jun/jun<adj><!17>/jun<n><!18>/jun<num><!19>$[{sl_ids:17; W:0.123}{sl_ids:18; W:0.346}{sl_ids:19; W:0.841}] ^qʼaxal/qʼaxal<adj><!20>$[{W:0.301}] ^utzij/<px3sg>tzij<n><!21>$[{W:0.205}] ^ri/ri<det><!22>$[{W:0.675}] ^Dyos/Dyos<n><!23>$[{W:0.442}]
 </pre>
@@ Line 76: / Line 78: @@
 sf-> surface form
 <pre>
-^«<lquot><!1>$[{sf:«}]^Ajawaxel<n><!2>$[{sf:Ajawxel}]^,<cm><!3>$[{sf:,}] ^ri<det><!4>$[{sf:ri}] ^in<prn><pro><p1><sg><!6>$[{sf:in}] ^<impf><o_sg3><s_sg1>il<v><tv><!7>+o<mark><!8>$[{sf:kinwilo}] ^chi<pr><!9>+<px3sg>ech<n><rel><!10>$[{sf:che}] ^ri<det><!11>$[{sf:ri}] ^at<prn><pers><p2><sg><!12>$[{sf:at}]^,<cm><!14>$[{sf:,}] ^at<prn><pro><p2><sg><!16>$[{sf:at}] ^jun<num><!19>$[{sf:jun}] ^qʼaxal<adj><!20>$[{sf:qʼaxal}] ^<px3sg>tzij<n><!21>$[{sf:utzij}] ^ri<det><!22>$[{sf:ri}] ^Dyos<n><!23>$[{sf:Dyos}]
+^«<lquot><!1>$[{W:0.987; sf:«}]^Ajawaxel<n><!2>$[{W:0.587; sf:Ajawxel}]^,<cm><!3>$[{W:0.953; sf:,}] ^ri<det><!4>$[{W:0.675; sf:ri}] ^in<prn><pro><p1><sg><!6>$[{sl_ids:6; W:0.457; sf:in}] ^<impf><o_sg3><s_sg1>il<v><tv><!7>+o<mark><!8>$[{sf:kinwilo}{sl_ids:7; W:0.621}{sl_ids:8; W:0.760}] ^chi<pr><!9>+<px3sg>ech<n><rel><!10>$[{sf:che}{sl_ids:9; W:0.832}{sl_ids:10; W:0.562}] ^ri<det><!11>$[{W:0.675; sf:ri}] ^at<prn><pers><p2><sg><!12>$[{sl_ids:12; W:0.709; sf:at}]^,<cm><!14>$[{W:0.953; sf:,}] ^at<prn><pro><p2><sg><!16>$[{sl_ids:16; W:0.302; sf:at}] ^jun<num><!19>$[{sl_ids:19; W:0.841; sf:jun}] ^qʼaxal<adj><!20>$[{W:0.301; sf:qʼaxal}] ^<px3sg>tzij<n><!21>$[{W:0.205; sf:utzij}] ^ri<det><!22>$[{W:0.675; sf:ri}] ^Dyos<n><!23>$[{W:0.442; sf:Dyos}]
 </pre>
@@ Line 83: / Line 85: @@
 cmp_id: compound id
 <pre>
-^«<lquot><!1>$[{sf:«}]^Ajawaxel<n><!2>$[{sf:Ajawxel}]^,<cm><!3>$[{sf:,}] ^ri<det><!4>$[{sf:ri}] ^in<prn><pro><p1><sg><!6>$[{sf:in}] ^<impf><o_sg3><s_sg1>il<v><tv><!7>$[{cmp_id:1; sf:kinwilo}]^o<mark><!8>$[{cmp_id:1; sf:kinwilo}] ^chi<pr><!9>$[{cmp_id:2; sf:che}] ^<px3sg>ech<n><rel><!10>$[{cmp_id:2; sf:che}] ^ri<det><!11>$[{sf:ri}] ^at<prn><pers><p2><sg><!12>$[{sf:at}]^,<cm><!14>$[{sf:,}] ^at<prn><pro><p2><sg><!16>$[{sf:at}] ^jun<num><!19>$[{sf:jun}] ^qʼaxal<adj><!20>$[{sf:qʼaxal}] ^<px3sg>tzij<n><!21>$[{sf:utzij}] ^ri<det><!22>$[{sf:ri}] ^Dyos<n><!23>$[{sf:Dyos}]
+^«<lquot><!1>$[{W:0.987; sf:«}]^Ajawaxel<n><!2>$[{W:0.587; sf:Ajawxel}]^,<cm><!3>$[{W:0.953; sf:,}] ^ri<det><!4>$[{W:0.675; sf:ri}] ^in<prn><pro><p1><sg><!6>$[{sl_ids:6; W:0.457; sf:in}] ^<impf><o_sg3><s_sg1>il<v><tv><!7>$[{sl_ids:7; W:0.621; cmp_id:1; sf:kinwilo}]^o<mark><!8>$[{sl_ids:8; W:0.760; cmp_id:1; sf:kinwilo}] ^chi<pr><!9>$[{sl_ids:9; W:0.832; cmp_id:2; sf:che}] ^<px3sg>ech<n><rel><!10>$[{sl_ids:10; W:0.562; cmp_id:2; sf:che}] ^ri<det><!11>$[{W:0.675; sf:ri}] ^at<prn><pers><p2><sg><!12>$[{sl_ids:12; W:0.709; sf:at}]^,<cm><!14>$[{W:0.953; sf:,}] ^at<prn><pro><p2><sg><!16>$[{sl_ids:16; W:0.302; sf:at}] ^jun<num><!19>$[{sl_ids:19; W:0.841; sf:jun}] ^qʼaxal<adj><!20>$[{W:0.301; sf:qʼaxal}] ^<px3sg>tzij<n><!21>$[{W:0.205; sf:utzij}] ^ri<det><!22>$[{W:0.675; sf:ri}] ^Dyos<n><!23>$[{W:0.442; sf:Dyos}]
 </pre>
@@ Line 90: / Line 92: @@
 mw_part: parts of a multi-word unit
 <pre>
-^«<lquot><!1>$[{sf:«}]^Ajawaxel<n><!2>$[{sf:Ajawxel}]^,<cm><!3>$[{sf:,}] ^ri<det><!4>$[{sf:ri}] ^in<prn><pro><p1><sg><!6>$[{sf:in}] ^<impf><o_sg3><s_sg1>il<v><tv><!7>$[{cmp_id:1; sf:kinwilo}]^o<mark><!8>$[{cmp_id:1; sf:kinwilo}] ^chi<pr><!9>$[{cmp_id:2; sf:che}] ^<px3sg>ech<n><rel><!10>$[{cmp_id:2; sf:che}] ^ri<det><!11>$[{sf:ri}] ^at<prn><pers><p2><sg><!12>$[{sf:at}]^,<cm><!14>$[{sf:,}] ^at<prn><pro><p2><sg><!16>$[{sf:at}] ^jun<num><!19>$[{sf:jun}] ^qʼaxal tzij ri Dyos<n><!24>$[{sl_ids:20; mw_part:1; sf:qʼaxal}{sl_ids:21; mw_part:2; sf:utzij}{sl_ids:22; mw_part:3; sf:ri}{sl_ids:23; mw_part:4; sf:Dyos}]
+^«<lquot><!1>$[{W:0.987; sf:«}]^Ajawaxel<n><!2>$[{W:0.587; sf:Ajawxel}]^,<cm><!3>$[{W:0.953; sf:,}] ^ri<det><!4>$[{W:0.675; sf:ri}] ^in<prn><pro><p1><sg><!6>$[{sl_ids:6; W:0.457; sf:in}] ^<impf><o_sg3><s_sg1>il<v><tv><!7>$[{sl_ids:7; W:0.621; cmp_id:1; sf:kinwilo}]^o<mark><!8>$[{sl_ids:8; W:0.760; cmp_id:1; sf:kinwilo}] ^chi<pr><!9>$[{sl_ids:9; W:0.832; cmp_id:2; sf:che}] ^<px3sg>ech<n><rel><!10>$[{sl_ids:10; W:0.562; cmp_id:2; sf:che}] ^ri<det><!11>$[{W:0.675; sf:ri}] ^at<prn><pers><p2><sg><!12>$[{sl_ids:12; W:0.709; sf:at}]^,<cm><!14>$[{W:0.953; sf:,}] ^at<prn><pro><p2><sg><!16>$[{sl_ids:16; W:0.302; sf:at}] ^jun<num><!19>$[{sl_ids:19; W:0.841; sf:jun}] ^qʼaxal tzij ri Dyos<n><!24>$[{sl_ids:20; W:0.301; sf:qʼaxal; mw_part:1}{sl_ids:21; W:0.205; sf:utzij; mw_part:2}{sl_ids:22; W:0.675; sf:ri; mw_part:3}{sl_ids:23; W:0.442; sf:Dyos; mw_part:4}]
 </pre>

@@ Line 95: / Line 95: @@
 =Thoughts=
-* compound id stores the id of the compound, but currently doesn't store the order of the parts in the compound (assumed to be ascending order of ids). mw_id however, has a blank block that preserves the order of the original parts that were made into an MWU.
+* compound id stores the id of the compound, but currently doesn't store the order of the parts in the compound (assumed to be ascending order of ids). mw_part does preserve the order.
+* I'm trying to make all blank blocks go with a general philosophy of having sl&tl ids and the other information pertaining to these.
-* There seem to be several kind of blank blocks now. Either I let it be like that - disambiguate the kind of blank block based on its components. Or, blank blocks get names. Such as [{..}mw{..}cmp{..}]. This could work but doesn't seem like a great idea.
 Possibilities:
-<pre>^qʼaxal tzij ri Dyos<n><!24>$[{part_ids:20,21,22,23}{sl_ids:20; sf:qʼaxal}{sl_ids:21; sf:utzij}{sl_ids:22; sf:ri}{sl_ids:23; sf:Dyos}]</pre>
+<pre>^qʼaxal tzij ri Dyos<n><!24>$[{mw_ids:20,21,22,23}{sl_ids:20; sf:qʼaxal}{sl_ids:21; sf:utzij}{sl_ids:22; sf:ri}{sl_ids:23; sf:Dyos}]</pre>
 <pre>^qʼaxal tzij ri Dyos<n><!24>$[{sl_ids:20; sf:qʼaxal}{sl_ids:21; sf:utzij}{sl_ids:22; sf:ri}{sl_ids:23; sf:Dyos}]</pre>
-<pre>^qʼaxal tzij ri Dyos<n><!24>$[{sl_ids:20; mw_part:1; sf:qʼaxal}{sl_ids:21; mw_part:2; sf:utzij}{sl_ids:22; mw_part:3; sf:ri}{sl_ids:23; mw_part:4; sf:Dyos}]</pre>

@@ Line 88: / Line 88: @@
 '''Separable Output:'''
+mw_part: parts of a multi-word unit
-mw_id: multiword id
 <pre>
-^«<lquot><!1>$[{sf:«}]^Ajawaxel<n><!2>$[{sf:Ajawxel}]^,<cm><!3>$[{sf:,}] ^ri<det><!4>$[{sf:ri}] ^in<prn><pro><p1><sg><!6>$[{sf:in}] ^<impf><o_sg3><s_sg1>il<v><tv><!7>$[{cmp_id:1; sf:kinwilo}]^o<mark><!8>$[{cmp_id:1; sf:kinwilo}] ^chi<pr><!9>$[{cmp_id:2; sf:che}] ^<px3sg>ech<n><rel><!10>$[{cmp_id:2; sf:che}] ^ri<det><!11>$[{sf:ri}] ^at<prn><pers><p2><sg><!12>$[{sf:at}]^,<cm><!14>$[{sf:,}] ^at<prn><pro><p2><sg><!16>$[{sf:at}] ^jun<num><!19>$[{sf:jun}] ^qʼaxal tzij ri Dyos<n><!24>$[{mw_id:24; sl_ids:20,21,22,23}{sl_ids:20; sf:qʼaxal}{sl_ids:21; sf:utzij}{sl_ids:22; sf:ri}{sl_ids:23; sf:Dyos}]
+^«<lquot><!1>$[{sf:«}]^Ajawaxel<n><!2>$[{sf:Ajawxel}]^,<cm><!3>$[{sf:,}] ^ri<det><!4>$[{sf:ri}] ^in<prn><pro><p1><sg><!6>$[{sf:in}] ^<impf><o_sg3><s_sg1>il<v><tv><!7>$[{cmp_id:1; sf:kinwilo}]^o<mark><!8>$[{cmp_id:1; sf:kinwilo}] ^chi<pr><!9>$[{cmp_id:2; sf:che}] ^<px3sg>ech<n><rel><!10>$[{cmp_id:2; sf:che}] ^ri<det><!11>$[{sf:ri}] ^at<prn><pers><p2><sg><!12>$[{sf:at}]^,<cm><!14>$[{sf:,}] ^at<prn><pro><p2><sg><!16>$[{sf:at}] ^jun<num><!19>$[{sf:jun}] ^qʼaxal tzij ri Dyos<n><!24>$[{sl_ids:20; mw_part:1; sf:qʼaxal}{sl_ids:21; mw_part:2; sf:utzij}{sl_ids:22; mw_part:3; sf:ri}{sl_ids:23; mw_part:4; sf:Dyos}]
 </pre>

@@ Line 27: / Line 27: @@
 * If a block of information doesn't have <code>sl_ids</code> or <code>tl_ids</code>, it refers to the entire LU (can be changed to <code>{sl_ids:11; tl_ids:67,68}</code> if we want that each block in a word bound blank should have IDs.
 * <code>sl_ids</code> and <code>tl_ids</code> can take multiple ids (from source or from target). This helps deal with many-to-many relationships between the tokens of the two languages.
+* When LUs merge, a new blank block is added that preserves information about the order of the original LUs with their ids. Also, the secondary information from all those LUs is added to the wordbound blank of the new merged LU, with each of their IDs inside the blocks.
+* When LUs break, secondary information is duplicated onto all parts, and each part gets the same compound id. Right now the order of the parts of the compound isn't preserved but it can be, if needed.
 = Rationale =

@@ Line 57: / Line 57: @@
 == Another one ==
-Original Analyser Output:
+'''Original Analyser Output:'''
 <pre>
 ^«/«<lquot>$^Ajawxel/Ajawaxel<n>$^,/,<cm>$ ^ri/ri<det>$ ^in/in<prn><pers><p1><sg>/in<prn><pro><p1><sg>$ ^kinwilo/<impf><o_sg3><s_sg1>il<v><tv>+o<mark>$ ^che/chi<pr>+<px3sg>ech<n><rel>$ ^ri/ri<det>$ ^at/at<prn><pers><p2><sg>/at<prn><pro><p2><sg>$^,/,<cm>$ ^at/at<prn><pers><p2><sg>/at<prn><pro><p2><sg>$ ^jun/jun<adj>/jun<n>/jun<num>$ ^qʼaxal/qʼaxal<adj>$ ^utzij/<px3sg>tzij<n>$ ^ri/ri<det>$
@@ Line 63: / Line 63: @@
 </pre>
-New Analyser Output:
+'''New Analyser Output:'''
+Special tag in the LU that will be ignored by FSTs. Format: <!..>
 <pre>
 ^«/«<lquot><!1>$^Ajawxel/Ajawaxel<n><!2>$^,/,<cm><!3>$ ^ri/ri<det><!4>$ ^in/in<prn><pers><p1><sg><!5>/in<prn><pro><p1><sg><!6>$ ^kinwilo/<impf><o_sg3><s_sg1>il<v><tv><!7>+o<mark><!8>$ ^che/chi<pr><!9>+<px3sg>ech<n><rel><!10>$ ^ri/ri<det><!11>$ ^at/at<prn><pers><p2><sg><!12>/at<prn><pro><p2><sg><!13>$^,/,<cm><!14>$ ^at/at<prn><pers><p2><sg><!15>/at<prn><pro><p2><sg><!16>$ ^jun/jun<adj><!17>/jun<n><!18>/jun<num><!19>$ ^qʼaxal/qʼaxal<adj><!20>$ ^utzij/<px3sg>tzij<n><!21>$ ^ri/ri<det><!22>$ ^Dyos/Dyos<n><!23>$
 </pre>
-Tagger Output:
+'''Tagger Output:'''
+sf-> surface form
 <pre>
 ^«<lquot><!1>$[{sf:«}]^Ajawaxel<n><!2>$[{sf:Ajawxel}]^,<cm><!3>$[{sf:,}] ^ri<det><!4>$[{sf:ri}] ^in<prn><pro><p1><sg><!6>$[{sf:in}] ^<impf><o_sg3><s_sg1>il<v><tv><!7>+o<mark><!8>$[{sf:kinwilo}] ^chi<pr><!9>+<px3sg>ech<n><rel><!10>$[{sf:che}] ^ri<det><!11>$[{sf:ri}] ^at<prn><pers><p2><sg><!12>$[{sf:at}]^,<cm><!14>$[{sf:,}] ^at<prn><pro><p2><sg><!16>$[{sf:at}] ^jun<num><!19>$[{sf:jun}] ^qʼaxal<adj><!20>$[{sf:qʼaxal}] ^<px3sg>tzij<n><!21>$[{sf:utzij}] ^ri<det><!22>$[{sf:ri}] ^Dyos<n><!23>$[{sf:Dyos}]
+</pre>
+'''Pretransfer Output:'''
+cmp_id: compound id
+<pre>
+^«<lquot><!1>$[{sf:«}]^Ajawaxel<n><!2>$[{sf:Ajawxel}]^,<cm><!3>$[{sf:,}] ^ri<det><!4>$[{sf:ri}] ^in<prn><pro><p1><sg><!6>$[{sf:in}] ^<impf><o_sg3><s_sg1>il<v><tv><!7>$[{cmp_id:1; sf:kinwilo}]^o<mark><!8>$[{cmp_id:1; sf:kinwilo}] ^chi<pr><!9>$[{cmp_id:2; sf:che}] ^<px3sg>ech<n><rel><!10>$[{cmp_id:2; sf:che}] ^ri<det><!11>$[{sf:ri}] ^at<prn><pers><p2><sg><!12>$[{sf:at}]^,<cm><!14>$[{sf:,}] ^at<prn><pro><p2><sg><!16>$[{sf:at}] ^jun<num><!19>$[{sf:jun}] ^qʼaxal<adj><!20>$[{sf:qʼaxal}] ^<px3sg>tzij<n><!21>$[{sf:utzij}] ^ri<det><!22>$[{sf:ri}] ^Dyos<n><!23>$[{sf:Dyos}]
+</pre>
+'''Separable Output:'''
+mw_id: multiword id
+<pre>
+^«<lquot><!1>$[{sf:«}]^Ajawaxel<n><!2>$[{sf:Ajawxel}]^,<cm><!3>$[{sf:,}] ^ri<det><!4>$[{sf:ri}] ^in<prn><pro><p1><sg><!6>$[{sf:in}] ^<impf><o_sg3><s_sg1>il<v><tv><!7>$[{cmp_id:1; sf:kinwilo}]^o<mark><!8>$[{cmp_id:1; sf:kinwilo}] ^chi<pr><!9>$[{cmp_id:2; sf:che}] ^<px3sg>ech<n><rel><!10>$[{cmp_id:2; sf:che}] ^ri<det><!11>$[{sf:ri}] ^at<prn><pers><p2><sg><!12>$[{sf:at}]^,<cm><!14>$[{sf:,}] ^at<prn><pro><p2><sg><!16>$[{sf:at}] ^jun<num><!19>$[{sf:jun}] ^qʼaxal tzij ri Dyos<n><!24>$[{mw_id:24; sl_ids:20,21,22,23}{sl_ids:20; sf:qʼaxal}{sl_ids:21; sf:utzij}{sl_ids:22; sf:ri}{sl_ids:23; sf:Dyos}]
 </pre>