Talk:Calculating coverage - Revision history

Unhammer: /* wikicat2.sh */ new section

2010-03-11T09:34:22Z

wikicat2.sh: new section

Unhammer: gsed

2009-09-04T09:27:54Z

gsed

Unhammer at 17:41, 26 March 2009

2009-03-26T17:41:29Z

Unhammer at 17:40, 26 March 2009

2009-03-26T17:40:22Z

Francis Tyers: User talk:Unhammer/Coverage moved to Talk:Calculating coverage

2009-03-26T09:43:08Z

User talk:Unhammer/Coverage moved to Talk:Calculating coverage

Francis Tyers: New page: ''(Mac OS X `sed' doesn't allow \n in replacements, so I just use an actual (escaped) newline...)'' :I've come across this before, the best thing to do here is install GNU sed ;) - ~~~~

2009-03-25T22:48:15Z

New page: ''(Mac OS X `sed' doesn't allow \n in replacements, so I just use an actual (escaped) newline...)'' :I've come across this before, the best thing to do here is install GNU sed ;) - ~~~~

New page

''(Mac OS X `sed' doesn't allow \n in replacements, so I just use an actual (escaped) newline...)''

:I've come across this before, the best thing to do here is install GNU sed ;) - [[User:Francis Tyers|Francis Tyers]] 22:48, 25 March 2009 (UTC)

@@ Line 4: / Line 4: @@
 :: <code>sudo port install ssed</code> for Super-Sed (long lost cousin of [http://www.youtube.com/results?search_query=super-ted&search_type=&aq=f Super-Ted]). Felt it important to point out the fact to potential other Mac users though... or maybe I should've written "Mac? Use ssed" instead. (Or <code>sudo port install gsed</code> for GNU sed, which seems faster than ssed.) [[User:Unhammer|Unhammer]]
+== wikicat2.sh ==
+I like to keep the punctuation, this is an alternative wikicat script. Use <code>$ ./wikicat2.sh blah-pages-articles.xml.bz2</code>:
+<pre>
+#!/bin/sh
+# clean up wiki for running through apertium-destxt
+# awk prints full lines, make sure each html element has one
+bzcat "$@" | sed 's/>/>\n/g' | sed 's/</\n</g' |\
+# want only stuff between <text...> and </text>
+awk '
+/<text.*>/,/<\/text>/ { print $0 }
+' |\
+sed 's/\[\[\([a-z]\{2,3\}\):[^]]\+\]\]//g' |\
+# Drop all transwiki links
+sed 's/\[\[[^]|]*|//g' | sed 's/\]\]//g' | sed 's/\[\[//g' |\
+# wiki markup, retain bar and fie from [[foo|bar]] [[fie]]
+sed 's/\[http[^ ]*\([^]]*\)\]/\1/g' |\
+# wiki markup, retain `bar fie' from [http://foo bar fie]
+sed 's/&.*;/ /g' |\
+# remove entities greedily, so as to get rid of hidden html too
+# Keep only lines starting with a capital letter, removing tables with style info etc.
+grep '^[ \t]*[A-ZÆØÅ]' # Your alphabet here
+</pre>

← Older revision		Revision as of 09:27, 4 September 2009
Line 3:		Line 3:
	:I've come across this before, the best thing to do here is install GNU sed ;) - [[User:Francis Tyers\|Francis Tyers]] 22:48, 25 March 2009 (UTC)		:I've come across this before, the best thing to do here is install GNU sed ;) - [[User:Francis Tyers\|Francis Tyers]] 22:48, 25 March 2009 (UTC)

	:: <code>sudo port install ssed</code> for Super-Sed (long lost cousin of [http://www.youtube.com/results?search_query=super-ted&search_type=&aq=f Super-Ted]). Felt it important to point out the fact to potential other Mac users though... or maybe I should've written "Mac? Use ssed" instead. [[User:Unhammer\|Unhammer]]		:: <code>sudo port install ssed</code> for Super-Sed (long lost cousin of [http://www.youtube.com/results?search_query=super-ted&search_type=&aq=f Super-Ted]). Felt it important to point out the fact to potential other Mac users though... or maybe I should've written "Mac? Use ssed" instead. (Or <code>sudo port install gsed</code> for GNU sed, which seems faster than ssed.) [[User:Unhammer\|Unhammer]]

← Older revision		Revision as of 17:41, 26 March 2009
Line 3:		Line 3:
	:I've come across this before, the best thing to do here is install GNU sed ;) - [[User:Francis Tyers\|Francis Tyers]] 22:48, 25 March 2009 (UTC)		:I've come across this before, the best thing to do here is install GNU sed ;) - [[User:Francis Tyers\|Francis Tyers]] 22:48, 25 March 2009 (UTC)

	:: <code>sudo port install ssed</code> for Super-Sed (long lost cousin of [http://www.youtube.com/results?search_query=super-ted&search_type=&aq=f Super-Ted]) [[User:Unhammer\|Unhammer]]		:: <code>sudo port install ssed</code> for Super-Sed (long lost cousin of [http://www.youtube.com/results?search_query=super-ted&search_type=&aq=f Super-Ted]). Felt it important to point out the fact to potential other Mac users though... or maybe I should've written "Mac? Use ssed" instead. [[User:Unhammer\|Unhammer]]

← Older revision		Revision as of 17:40, 26 March 2009
Line 2:		Line 2:

	:I've come across this before, the best thing to do here is install GNU sed ;) - [[User:Francis Tyers\|Francis Tyers]] 22:48, 25 March 2009 (UTC)		:I've come across this before, the best thing to do here is install GNU sed ;) - [[User:Francis Tyers\|Francis Tyers]] 22:48, 25 March 2009 (UTC)

			:: <code>sudo port install ssed</code> for Super-Sed (long lost cousin of [http://www.youtube.com/results?search_query=super-ted&search_type=&aq=f Super-Ted]) [[User:Unhammer\|Unhammer]]

← Older revision	Revision as of 09:43, 26 March 2009
(No difference)