User:Khannatanmai/New Apertium stream format - Revision history

Khannatanmai at 09:20, 17 July 2020

2020-07-17T09:20:44Z

Khannatanmai: /* Progress */

2020-05-17T20:45:49Z

Progress

Khannatanmai: /* Generator */

2020-05-17T20:45:21Z

Generator

Khannatanmai: /* Generator */

2020-05-14T20:20:09Z

Generator

Khannatanmai: /* Progress */

2020-05-14T20:19:34Z

Progress

Khannatanmai: /* Chunker (t1x) */ Modifications done

2020-05-12T18:53:56Z

Chunker (t1x): Modifications done

Khannatanmai: /* Transfer */

2020-05-12T13:50:21Z

Transfer

Khannatanmai: /* Transfer */

2020-05-12T13:49:27Z

Transfer

Khannatanmai: /* Transfer */ transfer desired output reached with secondary tags

2020-05-12T13:48:26Z

Transfer: transfer desired output reached with secondary tags

Khannatanmai: /* Transfer */ transfer puts appropriate secondary tags in output

2020-05-11T11:41:40Z

Transfer: transfer puts appropriate secondary tags in output

← Older revision		Revision as of 09:20, 17 July 2020
Line 1:		Line 1:
			<strong style="color:maroon;font-size:1.5em;>The secondary tags project was shelved as the need for reading specific secondary information wasn't established. This work continued as LU-bound secondary information using [[User:Khannatanmai/Wordbound_blanks \| wordbound blanks]]. </strong>

	Here I will provide updates about the development of the new Apertium stream format, which will include an arbitrary amount of optional secondary information.		Here I will provide updates about the development of the new Apertium stream format, which will include an arbitrary amount of optional secondary information.

@@ Line 308: / Line 308: @@
 = Progress =
 The first thing to do is to modify modules such that secondary tags pass through without any problems through the stream.
+'''Further updates can be found [[User:Khannatanmai/Secondary_tags_features|here]].
 == Transfer ==

@@ Line 375: / Line 375: @@
 ]
 </pre>
+'''Further updates can be found [[User:Khannatanmai/Secondary_tags_features|here]].

@@ Line 364: / Line 364: @@
 * Modified the generator such that it removes all trailing secondary tags before giving the input to the FST.
-Earlier Output:
+'''Earlier Output:'''
+<pre>
+ #The #dog #of #the #boy #run #fast#.#.[][
+]
+</pre>
-New Output:
+'''New Output:'''
 <pre>
 The dogs of the boy run fast..[][

@@ Line 359: / Line 359: @@
 * Input without secondary tags shows no regression in transfer output.
 * Need to test more thoroughly with a one-step transfer system, with MLUs, and with more variables.
+== Generator ==
+* Modified the generator such that it removes all trailing secondary tags before giving the input to the FST.
+Earlier Output:
+New Output:
+<pre>
+The dogs of the boy run fast..[][
+]
+</pre>

@@ Line 210: / Line 210: @@
 === Chunker (t1x) ===
+'''Output with original input(without secondary tags):'''
-'''Current Output:'''
 <pre>
@@ Line 217: / Line 217: @@
 </pre>
-'''Output with modified input:'''
+'''Proposed output with modified input(with secondary tags):'''
 <pre>
-^Det_nom<SN><m><pl>{^the<det><def><3>$ ^dog<n><3>$}$ ^default<default>{^of<pr><sf:del>$}$ ^det_nom<SN><m><sg>{^the<det><def><3>$ ^boy<n><3>$}$ ^verbcj<SV><vblex><pri><p3><pl>{^run<vblex><pres>$}$ ^adj<SA><m><sg>{^fast<adj><sint>$}$ ^default<default>{^.<sent><sf:.>$}$ ^default<default>{^.<sent><sf:.>$}$[][
+^Det_nom<SN><m><pl>{^the<det><def><3><sf:Los>$ ^dog<n><3><sf:perros>$}$ ^de<PREP>{^of<pr><sf:del>$}$ ^det_nom<SN><m><sg>{^the<det><def><3><sf:del>$ ^boy<n><3><sf:chico>$}$ ^verbcj<SV><vblex><pri><p3><pl>{^run<vblex><pres><sf:corren>$}$ ^adj<SA><m><sg>{^fast<adj><sint><sf:rápido>$}$^punt<sent>{^.<sent><sf:.>$}$^punt<sent>{^.<sent><sf:.>$}$[][
 ]
 </pre>
+'''Output with modified input (Reached proposed output):'''
-* It stops matching prepositions and delimiters. This is probably because the patterns were defined as "pr" instead of "pr.*".  To deal with this issue we could modify transfer pattern matching such that it ignores secondary tags during pattern matching. This would assume we cannot add secondary tags in t1x, and is consistent with our policy to not add tags in dixes. This was discussed earlier in section outlining why we need secondary tags.
-* We also ignore secondary tags in pattern matching as FSTs are order dependent and secondary tags will not be. Most likely we will let the FSTs ignore secondary tags and make a new pattern matching system just for the secondary tags.
-* The parser will be modified to detect secondary tags and access them.
-* The output system of the transfer will need to be modified such that the secondary tags are output no matter what (since they won't be mentioned in the <out> section). These secondary tags in the TL LU will be taken from wherever the lemma of a tag comes from. If the lemma comes from a variable then naturally it won't have any secondary tags.
-* It will also be given the ability to add secondary tags in the output.
-* A pseudo-attribute will be added which gets a string of all secondary tags. Regex: <code>((?:<[^<>]+:[^<>]*>)+)</code> or <code>((<[^>]+:[^>]+>)+)</code> (courtesy [[User:popcorndude]]).
-* The pseudo-attribute tags will be needed to modified such that it doesn't include secondary tags to ensure backwards compatibility.
-'''Output modified manually (Proposed output):'''
 <pre>
@@ Line 238: / Line 230: @@
 ]
 </pre>
+* <s>It stops matching prepositions and delimiters. This is probably because the patterns were defined as "pr" instead of "pr.*".  To deal with this issue we could modify transfer pattern matching such that it ignores secondary tags during pattern matching.</s>
+* <s>We also ignore secondary tags in pattern matching as FSTs are order dependent and secondary tags will not be. Most likely we will let the FSTs ignore secondary tags and make a new pattern matching system just for the secondary tags.</s>
+* The parser will be modified to detect secondary tags and access them.
+* <s>The output system of the transfer will need to be modified such that the secondary tags are output no matter what (since they won't be mentioned in the <out> section). These secondary tags in the TL LU will be taken from wherever the lemma of a tag comes from. If the lemma comes from a variable then wherever the variable gets the lemma, that's where the secondary tags will come from.</s>
+* <s>It will also be given the ability to add secondary tags in the output.</s>
+* <s>A pseudo-attribute will be added which gets a string of all secondary tags. Regex: <code>((?:<[^<>]+:[^<>]*>)+)</code> or <code>((<[^>]+:[^>]+>)+)</code> (courtesy [[User:popcorndude]]).</s>
+* <s>The pseudo-attribute tags will be needed to modified such that it doesn't include secondary tags to ensure backwards compatibility.</s>
 === Interchunk (t2x) ===

@@ Line 310: / Line 310: @@
 == Transfer ==
+'''9 May 2020'''
 * Modified transfer.cc so that while pattern matching, secondary tags are ignored.

@@ Line 323: / Line 323: @@
 ]
 </pre>
+'''11 May 2020'''
 * Modified regex of tags so that it ignores secondary tags and added regex for stags in transfer_data.cc
@@ Line 340: / Line 342: @@
 Note, the verb didn't get its secondary tag because it uses a variable to print the lemma.
+'''12 May 2020'''
 * Modified transfer so that a map is created between variable_name and secondary tags from the lemma that this variable clips (only if the variable clips a lem/lemh), which is then added in the output if the variable is called in output.

@@ Line 323: / Line 323: @@
 ]
 </pre>
+* Modified regex of tags so that it ignores secondary tags and added regex for stags in transfer_data.cc
+* Transfer gets secondary tags from the LU where the lem or lemh comes from in the <code><lu>..</lu></code>, given that this lu block is in <code><out>..</out></code>. Still need to deal with variables.
+'''Earlier output:'''
+<pre>
+^Det_nom<SN><m><pl>{^the<det><def><3>$ ^dog<n><3>$}$ ^de<PREP>{^of<pr><sf:del>$}$ ^det_nom<SN><m><sg>{^the<det><def><3>$ ^boy<n><3>$}$ ^verbcj<SV><vblex><pri><p3><pl>{^run<vblex><pres>$}$ ^adj<SA><m><sg>{^fast<adj><sint>$}$ ^punt<sent>{^.<sent><sf:.>$}$ ^punt<sent>{^.<sent><sf:.>$}$[][
+]
+</pre>
+'''New output:'''
+<pre>
+^Det_nom<SN><m><pl>{^the<det><def><3><sf:Los>$ ^dog<n><3><sf:perros>$}$ ^de<PREP>{^of<pr><sf:del>$}$ ^det_nom<SN><m><sg>{^the<det><def><3><sf:del>$ ^boy<n><3><sf:chico>$}$ ^verbcj<SV><vblex><pri><p3><pl>{^run<vblex><pres>$}$ ^adj<SA><m><sg>{^fast<adj><sint><sf:rápido>$}$ ^punt<sent>{^.<sent><sf:.>$}$ ^punt<sent>{^.<sent><sf:.>$}$[][
+]
+</pre>
+Note, the verb didn't get its secondary tag because it uses a variable to print the lemma.