Sentence segmenting - Revision history

Unhammer at 12:51, 26 September 2018

2018-09-26T12:51:37Z

Dtvrij74: /* Sentrick */

2013-01-12T22:47:29Z

Sentrick

Dtvrij74: /* Sentrick */

2013-01-12T22:14:51Z

Sentrick

Dtvrij74: /* MxTerminator */

2013-01-05T00:29:44Z

MxTerminator

Dtvrij74: /* Sentrick */

2013-01-04T21:49:26Z

Sentrick

Dtvrij74: /* Sentrick */

2013-01-04T21:37:42Z

Sentrick

Dtvrij74: /* Sentrick */

2013-01-04T21:32:21Z

Sentrick

Dtvrij74: /* Sentrick */

2013-01-04T21:31:14Z

Sentrick

Dtvrij74 at 14:37, 2 January 2013

2013-01-02T14:37:14Z

Dtvrij74 at 00:56, 2 January 2013

2013-01-02T00:56:36Z

@@ Line 3: / Line 3: @@
 This page gives a review and usage instructions for some sentence-segmenting tools.
 Keywords: Sentence segmentation, sentence tokenization, sentence tokenisation
+{| class=wikitable
+|-
+|Tool
+|Author/article
+|Method
+|Language
+|-
+|-
+|NLTK Punkt
+|[https://www.mitpressjournals.org/doi/abs/10.1162/coli.2006.32.4.485 Kiss & Strunk 2006]
+|Unsupervised
+|Python 2 & 3
+|-
+|splitta
+|[https://dl.acm.org/citation.cfm?id=1620920 Gillick 2009]
+|Supervised, SVM
+|Python 2
+|-
+|MxTerminator
+|[https://arxiv.org/pdf/cmp-lg/9704002.pdf Reynar & Ratnaparkhi 1997]
+|Supervised, maxent
+|Java
+|-
+|Sentrick
+|Patrick Tschorn
+|[https://web.archive.org/web/20120620210553/http://www.denkselbst.de:80/sentrick/index.html seems supervised]
+|Java and Prolog(!)
+|}
 ==Tools==

@@ Line 233: / Line 233: @@
 It will generate a folder called <code>dist</code>. In terminal, type <code>cd dist</code> to get to the dist directory.
-You need some input data with segmented sentences separated by a newline. `editor.sh` and `snippetCollector.sh` (two minumalistic GUI tools) will save the boundary positions (punctuation that ends sentences) in a .bps file:
+You need some input data with segmented sentences separated by a newline. <code>editor.sh</code> and <code>snippetCollector.sh</code> (two minumalistic GUI tools) will save the boundary positions (punctuation that ends sentences) in a .bps file:
 Type <code>sh editor.sh</code> to open the Sentrick Sentence Boundary Editor (aka editor.sh).

@@ Line 211: / Line 211: @@
 ===Sentrick===
-'''WILL BE UPDATED''' as I receive more information!!!
 [http://www.denkselbst.de/sentrick2/ Home]
 <br />
@@ Line 281: / Line 278: @@
 After it builds, there will be a file called <code>learnVetoRules.sh</code> in <code>modules/niffler/scripts</code>.
+The script needs to be passed the four .pl files generated by tdgen.sh. This may take a while.
+There will be some <code>acceptedByTeacher.*</code> files. <code>acceptedByTeacher.html</code> contains the segmented sentences. The <code>.pl</code> version contains all the generated rules as plain text Prolog. The <code>.ser</code> file can be used to bootstrap the learning cycle so you don't have to make Sentrick relearn everything.
 ;Usage

@@ Line 207: / Line 207: @@
 ./mxterminator projectdir < textfile
 </pre>
+(Note: these are instructions from the readme and from developer consultation, not tested yet)
 ===Sentrick===

@@ Line 219: / Line 219: @@
 ;Training
+Good luck :)
 Get the developer's version
@@ Line 273: / Line 275: @@
 swi.dyn=amd64
 </pre>
+If it still does not work, run <code>ant clean dist -Darch=nachtigaller</code> from the root folder of your Sentrick installation.
+After it builds, there will be a file called <code>learnVetoRules.sh</code> in <code>modules/niffler/scripts</code>.
 ;Usage

@@ Line 258: / Line 258: @@
 dir is the directory of the .txt and the .bps files. de.denkselbst.sentrick.sbd.NoSbdProvider is the sbd. ../out is the output directory (where your 4 .pl files will end up).
-This converts the files to a Prolog file. Next, cd to <code>modules/niffler</code>.
+This converts the files to a Prolog file.
+Next, cd to <code>modules/niffler</code>. Build by using:
+<pre>
+ant clean dist -Darch=nachtigaller
+</pre>
+If it gives an error like:
+<pre>
+cannot find symbol : class Term
+</pre>
+That means that it cannot find jpl.jar. Open <code>nachtigaller.properties</code> in <code>modules/niffler</code> and set <code>swi.home</code> to where you installed SWI-Prolog. Set <code>swi.dyn</code> equal to your processor. Here is an example of how <code>nachtigaller.properties</code> may look:
+<pre>
+swi.home=/usr/lib/swi-prolog
+swi.dyn=amd64
+</pre>
 ;Usage

@@ Line 254: / Line 254: @@
 <pre>
 sh tdgen.sh dir utf-8 de.denkselbst.sentrick.sbd.NoSbdProvider ../out
-<pre>
+</pre>
 dir is the directory of the .txt and the .bps files. de.denkselbst.sentrick.sbd.NoSbdProvider is the sbd. ../out is the output directory (where your 4 .pl files will end up).

@@ Line 49: / Line 49: @@
 Add the PunktLanguageVars variable as a second argument for [http://nltk.googlecode.com/svn/trunk/doc/api/nltk.tokenize.punkt.PunktTrainer-class.html PunktTrainer], like this:
 <pre>
-trainer = nltk.tokenize.punkt.PunktTrainer(text, language_punkt_vars)
+trainer = nltk.tokenize.punkt.PunktTrainer(traindata, language_punkt_vars)
 params = trainer.get_params()
 sbd = nltk.tokenize.punkt.PunktSentenceTokenizer(params)
@@ Line 55: / Line 55: @@
 * The first line is to create a [http://nltk.googlecode.com/svn/trunk/doc/api/nltk.tokenize.punkt.PunktTrainer-class.html PunktTrainer] object
-* The variable <code>text</code> is the corpus that you want to train the trainer on.
+* The variable <code>traindata</code> is the corpus that you want to train the trainer on.
 * <code>language_punkt_vars</code> is your PunktLanguageVars variable.
 * [http://nltk.googlecode.com/svn/trunk/doc/api/nltk.tokenize.punkt.PunktSentenceTokenizer-class.html PunktSentenceTokenizer] accepts your parameters (<code>params</code>), which we'll also use later.
@@ Line 62: / Line 62: @@
 If the language uses the same punctuation as English, you create the trainer slightly differently:
 <pre>
-trainer = nltk.tokenize.punkt.PunktTrainer(text)
+trainer = nltk.tokenize.punkt.PunktTrainer(traindata)
 trainer.INCLUDE_ALL_COLLOCS = True
 trainer.INCLUDE_ABBREV_COLLOCS = True
@@ Line 69: / Line 69: @@
 * The first line is to create a [http://nltk.googlecode.com/svn/trunk/doc/api/nltk.tokenize.punkt.PunktTrainer-class.html PunktTrainer] object
-* <code>text</code> is the (string) corpus you want to train the trainer on.
+* <code>traindata</code> is the (string) corpus you want to train the trainer on.
 * <code>INCLUDE_ALL_COLLOCS</code> makes the trainer look for and remember abbreviations and initialisms.
 * <code>INCLUDE_ABBREV_COLLOCS</code> makes the trainer look for and remember word pairs where the first word is an abbreviation. It has to be placed after <code>INCLUDE_ALL_COLLOCS</code>.
@@ Line 75: / Line 75: @@
 * We'll use the <code>params</code> variable later.
-For every language, assuming that the variable <code>text</code> is the corpus you want to train your trainer on, do:
+For every language, assuming that the variable <code>traindata</code> is the corpus you want to train your trainer on, do:
 <pre>
-trainer.train(text)
+trainer.train(traindata)
 </pre>