Tools for TMX - Revision history

Shardulc at 02:43, 10 March 2018

2018-03-10T02:43:26Z

2014-10-02T08:38:07Z

Notes

2010-08-01T10:32:21Z

Notes

2009-03-12T14:42:16Z

Code

2009-03-11T17:34:08Z

Code

2009-03-11T15:43:30Z

2009-03-11T15:15:00Z

Code

2009-03-11T14:59:09Z

Examples

2009-03-11T14:22:19Z

Code

2009-03-11T14:18:16Z

Code

@@ Line 1: / Line 1: @@
+{{Github-unmigrated-tool}}
 {{TOCD}}
 As it is now quite easy to make lots of large TMXes with [[Bitextor]] and other tools, it would be good to have some tools for processing them. There are various things that it would be useful to do. For example, given a TMX file:

@@ Line 173: / Line 173: @@
 ==Notes==
 <references/>
+==See also==
+* [[TMX]]
+* [[Translation memory]]
 [[Category:Tools]]

@@ Line 22: / Line 22: @@
 ** For 1,055,574 TUs (file size: 298M), processed in 12 minutes using max. 1.7Gb RAM resulting in 25,546 TUs
 * <code>tmx-trim</code>
-** For 25,546 (file size: 8.5M), processed in 11 seconds resulting in 13,009 TUs
+** For 25,546 TUs (file size: 8.5M), processed in 11 seconds resulting in 13,009 TUs
+** For 16,373,421 TUs (file size: 1.5G), processed in 57 minutes resulting in 9,492,081 TUs
 ;Pending jobs

@@ Line 18: / Line 18: @@
 * <code>tmx-extract</code>
-** For ? TUs (file size: 16G), processed in 3 hours using max. 4M RAM.
+** For 188,807,877 TUs (file size: 16G), processed in 3 hours using max. 4M RAM.
 * <code>tmx-uniq</code>
 ** For 1,055,574 TUs (file size: 298M), processed in 12 minutes using max. 1.7Gb RAM resulting in 25,546 TUs

@@ Line 22: / Line 22: @@
 * <code>tmx-trim</code>
 ** For 25,546 (file size: 8.5M), processed in 11 seconds resulting in 13,009 TUs
+;Pending jobs
+* Make it work with <code>stdin</code> and <code>stdout</code>
+* Make <code>tmx-uniq</code> work with a previously sorted TMX and not do the sorting itself.
+* Make <code>tmx-sort</code> faster (maybe use a similar strategy to paradigm chopper?)
+* Make better heuristics for <code>tmx-trim</code>
+* Rewrite python in C++ where it makes sense.
 ==Examples==

@@ Line 24: / Line 24: @@
 ==Examples==
+===tmx-trim===
+<code>tmx-trim</code> takes a TMX file and performs some basic discard operations, it checks for and discards the sentences if:
+* the segments are exactly the same
+* one segment is twice as long or twice as short as the other
+* one segment as two more or two fewer punctuation marks than the other
+* either one of the segments contains no alphabetic characters
+It would be quite easy to add other heuristics, and tweak the above heuristics.
 ===tmx-extract===