MTX format - Revision history

Frankier: /* Arithmetic operators */

2016-08-29T18:42:26Z

Arithmetic operators

Frankier: /* Boolean operators */

2016-08-29T18:41:59Z

Boolean operators

Frankier: /* Macro definition and use */

2016-08-29T18:40:50Z

Macro definition and use

Frankier: /* Macro definition and use */

2016-08-29T18:29:45Z

Macro definition and use

Frankier at 08:43, 23 August 2016

2016-08-23T08:43:53Z

Frankier at 15:39, 22 August 2016

2016-08-22T15:39:46Z

Frankier at 13:35, 22 August 2016

2016-08-22T13:35:40Z

Frankier: Created page with "This page serves a reference to the MTX format. The MTX format describes features to be used by the Perceptron tagger. == Example == Here is an example of the basic outl..."

2016-08-22T13:24:39Z

Created page with "This page serves a reference to the MTX format. The MTX format describes features to be used by the Perceptron tagger. == Example == Here is an example of the basic outl..."

New page

This page serves a reference to the MTX format. The MTX format describes features to be used by the [[Perceptron tagger]].

== Example ==

Here is an example of the basic outline of an MTX file to illustrate the structure and some common constructs:

<nowiki>
<?xml version="1.0" ?>①
<!DOCTYPE metatag [
<!ENTITY commondefns SYSTEM "commondefns.mtx">②
]>
③
<metatag>
<coarse-tags tag="mytsx.tsx" />④
<beam-width val="10" />⑤
<defns>⑥
&commondefns;②
<def-str name="plus" val="+" />
<def-macro name="foo">
...
</def-macro>
...
</defns>
<feats>⑦

<feat>⑧
...
<pred>...</pred>
<out>
<macro name="foo"></macro>
...
</out>
<out-many>...</out-many>
</feat>
</feats>
</metatag>
</nowiki>

# The format is an XML format.
# So files can be included using XML entities as illustrated.
# And XML comments can be used.
# If you want to make use of coarse tags you can reference a TSX file using a relative file path.
# You can change the beam width of used in decoding with this tag.
# The defns section contains constants and macros.
# The feats section contains feature definitions
# Each feature definition can contain many boolean predicates with <pred>, normal output with <out> and generation of many features from an array type with <out-many>

== Tag reference ==

@@ Line 99: / Line 99: @@
 === Arithmetic operators ===
+ <nowiki>
-<add>, <sub>
+<add>, <sub></nowiki>
 === Feature extraction ===

@@ Line 94: / Line 94: @@
 <and>, <not>, <or>
+The commutative operators <and> and <or> can take 2 or more operands. Not can take one operand.
 === Arithmetic operators ===

@@ Line 71: / Line 71: @@
  <nowiki>
 <def-macro as="foo" args="bar baz">
-...
+  ...
-<var name="bar" />
+  <var name="bar" />
-...
+  ...
-<var name="baz" />
+  <var name="baz" />
-...
+  ...
 </def-macro></nowiki>

@@ Line 67: / Line 67: @@
 === Macro definition and use ===
+To define a macro foo with arguments bar and baz:
-<def-macro>
+ <nowiki>
+<def-macro as="foo" args="bar baz">
+...
+<var name="bar" />
+...
+<var name="baz" />
+...
+</def-macro></nowiki>
+To use it in a feature.
+ <nowiki>
+<feat>
+  <macro name="foo">
+    <int val="42" />
+    <str val="ook?" />
+</feat></nowiki>
 === Other definitions and their use ===

@@ Line 64: / Line 64: @@
 == Tag reference ==
+=== Macro definition and use ===
+<def-macro>
+=== Other definitions and their use ===
+<def-set> <def-str>
 === Boolean operators ===
@@ Line 74: / Line 82: @@
 === Feature extraction ===
+<ex-wordoids>
 === Wordoid addressing ===
+Lexical units are addressed with integers, but addressing wordoids must be done by...
+<addr-of-ints>
+<clamp>
+<carry>
 === Sets ===
+<set-has>
 === String operators ===
+<substr>
 === Loops ===
+<for-each>
-=== Macros ===

@@ Line 51: / Line 51: @@
 Each <feat> tag can generate zero, one or many features for each wordoid.
+== Designing new features ==
+One process to design a new feature might go as follows:
+# Start from an existing feature file.
+# Find a mistagged word.
+# Run the tagger in debug mode as detailed on [[Perceptron tagger]].
+# Come up with a possible feature which would end up fire for the correct analysis and end up positive given the training data or come up with a feature which would fire for the incorrect analysis and would end up negative given the training data.
+# Rerun to see if it's fixed your mistagged word.
+# Run cross validation on your whole training corpus to check the overall accuracy hasn't gone down.
 == Tag reference ==