User:Pankajksharma/Patcher - Revision history

Unhammer at 08:30, 2 June 2016

2016-06-02T08:30:26Z

Pankajksharma at 17:49, 20 August 2014

2014-08-20T17:49:46Z

Pankajksharma: /* The grounding tag */

2014-08-20T17:38:00Z

The grounding tag

Pankajksharma at 17:36, 20 August 2014

2014-08-20T17:36:59Z

Pankajksharma at 11:15, 17 August 2014

2014-08-17T11:15:13Z

Pankajksharma at 11:12, 17 August 2014

2014-08-17T11:12:04Z

Pankajksharma at 11:01, 17 August 2014

2014-08-17T11:01:36Z

Pankajksharma at 10:58, 17 August 2014

2014-08-17T10:58:21Z

Pankajksharma at 10:55, 17 August 2014

2014-08-17T10:55:51Z

Pankajksharma at 10:33, 17 August 2014

2014-08-17T10:33:28Z

@@ Line 3: / Line 3: @@
 Over here a screen shot of original algorithm along with how to use it is provided.
+See the [[User:Pankajksharma/Application|Application]], original idea page: [[Ideas_for_Google_Summer_of_Code/Command-line_translation_memory_fuzzy-match_repair]]
-Application http://wiki.apertium.org/wiki/User:Pankajksharma/Application
 For installation instructions, please visit: http://wiki.apertium.org/wiki/User:Pankajksharma/Patcher_Installation

@@ Line 1: / Line 1: @@
 Wiki page for GSoC work done by [[User:pankajksharma | pankajksharma]] during GSoC 2014.
-Over here a screenshot of original algorithm along with how to use it is provided.
+Over here a screen shot of original algorithm along with how to use it is provided.
 Application http://wiki.apertium.org/wiki/User:Pankajksharma/Application
@@ Line 11: / Line 11: @@
 ==Apertium Python Patcher==
-For a given pair of sentences (S, S') having a Fuzzy match score above a certain threshold (simply put, similar to each other) in one natural language (say s). If T is the translation of S (say in another natural lanuguage t), the job of this patcher is to obtain T' i.e., the corresponding transaltion of S'.
+For a given pair of sentences (S, S') having a Fuzzy match score above a certain threshold (simply put, similar to each other) in one natural language (say s). If T is the translation of S (say in another natural language t), the job of this patcher is to obtain T' i.e., the corresponding translation of S'.
 See http://wiki.apertium.org/wiki/User:Pankajksharma/Application#Proposal for more detail.
@@ Line 60: / Line 60: @@
   -c     C.db        Specify the sqlite3 db to be used for caching
-  -d     D           Specify the lanuguage-pair installation directory
+  -d     D           Specify the language-pair installation directory
   --cam              Only those patches which cover all the mismatches
@@ Line 101: / Line 101: @@
   -c C               Specify the sqlite3 db to be used for caching
-  -d D               Specify the lanuguage-pair installation directory
+  -d D               Specify the language-pair installation directory
   --cam              Only those patches which cover all the mismatches
@@ Line 149: / Line 149: @@
   -h, --help         show this help message and exit
-  -d D               Specify the lanuguage-pair installation directory
+  -d D               Specify the language-pair installation directory
   -c C               Specify the sqlite3 db to be used for caching
@@ Line 172: / Line 172: @@
 --cam             Includes only those sentences which covers all mismatches
---compare         Compares all reults for above two modes (verbose doesn't work in this mode)
+--compare         Compares all results for above two modes (verbose doesn't work in this mode)
         usage: python reg_test.py pairs/en-es.pairs en-es --mode compare
@@ Line 179: / Line 179: @@
 ===preprocess.py===
-Preprocess the corpus for generating input for reg_test
+Pre process the corpus for generating input for reg_test
         usage: preprocess.py [-h] [-v] [--min-fms MIN_FMS] [--max-len MAX_LEN]
@@ Line 229: / Line 229: @@
                 D
-Calulates FMS distribtution for all corpuses pressent in directory D.
+Calculates FMS distribution for all corpses present in directory D.
 positional arguments:
@@ Line 239: / Line 239: @@
   -h, --help         show this help message and exit
-  -d D               Specify the lanuguage-pair installation directory
+  -d D               Specify the language-pair installation directory
   --min-fms MIN_FMS  Minimum value of fuzzy match score of S and S1.
@@ Line 301: / Line 301: @@
   -h, --help         show this help message and exit
-  -d D               Specify the lanuguage-pair installation directory
+  -d D               Specify the language-pair installation directory
   --min-fms MIN_FMS  Minimum value of fuzzy match score of S and S1.

@@ Line 26: / Line 26: @@
 This patcher provides a --go (grounded only) option. This option only patches when the mismatch is covered from both sides. For example, ('he went there', 'he wanted to go there') is a grounded phrase pair but ('he went there', 'she went there') is not.
-Grounding requires min-len (minimum length of phrase) to be atleast more than 1. Otherwise, there's no chance of patching.
+Grounding (--go) requires min-len (minimum length of phrase) to be more than 1. Otherwise, there no patching would be possible.
 ==Scripts and how to run==

@@ Line 22: / Line 22: @@
 ===Heuristics for best match===
 Currently the overall sum of length of phrases used to obtain the patch is used to guess the best possible patch. This sum is expected to represent the degree of coverage each possible patch has obtained.
+===The grounding tag===
+This patcher provides a --go (grounded only) option. This option only patches when the mismatch is covered from both sides. For example, ('he went there', 'he wanted to go there') is a grounded phrase pair but ('he went there', 'she went there') is not.
+Grounding requires min-len (minimum length of phrase) to be atleast more than 1. Otherwise, there's no chance of patching.
 ==Scripts and how to run==

@@ Line 19: / Line 19: @@
 [[File:on_the_fly_patcher.png]]
+===Heuristics for best match===
+Currently the overall sum of length of phrases used to obtain the patch is used to guess the best possible patch. This sum is expected to represent the degree of coverage each possible patch has obtained.
 ==Scripts and how to run==

@@ Line 65: / Line 65: @@
 example: python repair.py "the black dog was barking whole night" "el perro negro ladraba noche entera" "the black cat was barking whole night" en-es -v
+===tmx_patcher.py===
+Reads Translation Memory (TM) and tries to patch given sentence (S) with the help of best matching sentence available in TM
+        usage: tmx_patcher.py [-h] [-v] [-t] [-c C] [-d D] [--cam] [--go]
+                      [--min-fms MIN_FMS] [--min-len MIN_LEN]
+                      [--max-len MAX_LEN]
+                      TM S LP
+positional arguments:
+  TM                 Translation Memory
+  S                  Second Sentence
+  LP                 Language Pair for TM (for example en-eo)
+optional arguments:
+  -h, --help         show this help message and exit
+  -v                 Verbose Mode
+  -t                 Show patching traces
+  -c C               Specify the sqlite3 db to be used for caching
+  -d D               Specify the lanuguage-pair installation directory
+  --cam              Only those patches which cover all the mismatches
+  --go               To patch only grounded mismatches
+  --min-fms MIN_FMS  Minimum value of fuzzy match score of S and S1.
+  --min-len MIN_LEN  Minimum length of sub-segment allowed.
+  --max-len MAX_LEN  Maximum length of sub-segment allowed.
+Example: python tmx_patcher.py /tmp/ca-en-short.tmx "Les úniques respostes útils són les que creen preguntes." ca-en --min-fms 0.8 --cam --min-len 1 --max-len 3 -v
 ===fms.py===

@@ Line 68: / Line 68: @@
 ===fms.py===
-usage: fms.py [-h] S S1
+        usage: fms.py [-h] S S1
 Provides FMS of strings S and S1 using Wagner-Fischer algorithm.
@@ Line 86: / Line 86: @@
 Regression test for our patcher (repair.py). This script takes Out (the output of preprocess.py) and LP (the name of language pair).
-usage: reg_test.py [-h] [-d D] [-c C] [-v] [--mode MODE] [--go]
+        usage: reg_test.py [-h] [-d D] [-c C] [-v] [--mode MODE] [--go]
                    [--min-fms MIN_FMS] [--min-len MIN_LEN] [--max-len MAX_LEN]
                    out LP
@@ Line 125: / Line 125: @@
 --compare         Compares all reults for above two modes (verbose doesn't work in this mode)
-Example usage: python reg_test.py pairs/en-es.pairs en-es --mode compare
+        usage: python reg_test.py pairs/en-es.pairs en-es --mode compare
@@ Line 132: / Line 132: @@
 Preprocess the corpus for generating input for reg_test
-usage: preprocess.py [-h] [-v] [--min-fms MIN_FMS] [--max-len MAX_LEN]
+        usage: preprocess.py [-h] [-v] [--min-fms MIN_FMS] [--max-len MAX_LEN]
                      SLF TLF SLFT TLFT OUT
@@ Line 162: / Line 162: @@
 Calculates and show a histogram of the distribution of FMS between pair of sentences present in corpus F.
-usage: file_stats.py [-h] [--min-fms MIN_FMS] F
+        usage: file_stats.py [-h] [--min-fms MIN_FMS] F
 positional arguments:
@@ Line 176: / Line 176: @@
 ===stats.py===
-usage: stats.py [-h] [-d D] [--min-fms MIN_FMS] [--min-len MIN_LEN]
+        usage: stats.py [-h] [-d D] [--min-fms MIN_FMS] [--min-len MIN_LEN]
                 [--max-len MAX_LEN]
                 D
@@ Line 198: / Line 198: @@
   --max-len MAX_LEN  Maximum length of sub-string allowed.
-===Set A generator===
+===Set A generator (deprecated)===
-usage: A_generator.py [-h] [--min-fms MIN_FMS] [--min-len MIN_LEN]
+        usage: A_generator.py [-h] [--min-fms MIN_FMS] [--min-len MIN_LEN]
                       [--max-len MAX_LEN]
                       S S1
@@ Line 231: / Line 231: @@
-===Set D generator===
+===Set D generator (deprecated)===
-Usage: python D_generator.py --helpusage: D_generator.py [-h] [-d D] [--min-fms MIN_FMS] [--min-len MIN_LEN] [--max-len MAX_LEN]
+        usage: D_generator.py [-h] [-d D] [--min-fms MIN_FMS] [--min-len MIN_LEN] [--max-len MAX_LEN]
-                      S T S1 LP
+        S T S1 LP
 Generates set D.

@@ Line 26: / Line 26: @@
 ===On the fly patching (repair.py)===
-usage: repair.py [-h] [-v] [-t] [-c C] [-d D] [--cam] [--go]
+          usage: repair.py [-h] [-v] [-t] [-c C] [-d D] [--cam] [--go]
                  [--min-fms MIN_FMS] [--min-len MIN_LEN] [--max-len MAX_LEN]
                  S T S1 LP
@@ Line 155: / Line 155: @@
   --max-len MAX_LEN  Maximum length of sentences allowed (default 25)
-  example: python preprocess.py ../ap/mtacat/en.en-es.train ../ap/mtacat/es.en-es.train ../ap/mtacat/en.en-es.testset ../ap/mtacat/es.en-es.test en-es.pairs -v
+  example: python preprocess.py en.en-es.train es.en-es.train en.en-es.testset es.en-es.test en-es.pairs