Xml grep - Revision history

Unhammer: /* It's slow on big files */

2017-10-10T09:04:09Z

It's slow on big files

2016-06-20T11:12:28Z

It's slow on big files: minor spelling correction

2016-01-29T08:21:24Z

It's slow on big files

2016-01-29T08:20:31Z

It's slow on big files

2016-01-29T08:19:55Z

I get "Unknown option --xpath"

2015-03-25T12:22:48Z

External links

2015-02-18T14:13:51Z

But I want XML awk/sed/diff/patch/join/etc.!

2015-02-18T14:13:23Z

But I want XML awk/sed/diff/patch/join/etc.!

2014-12-09T20:44:20Z

But I want XML awk/sed/diff/patch/join/etc.!

2014-12-03T12:43:35Z

@@ Line 110: / Line 110: @@
    | xmlstarlet sel -N w='http://www.mediawiki.org/xml/export-0.10/' -t -c '//w:text/text()'
 </code>
+Another alternative is <code>xml_grep<code>. Do
+<pre>
+sudo apt install xml-twig-tools
+</pre>
+to install it. It seems to run fine on huge input without eating memory :-)
+<pre>
+xzcat corpus.xml.xz | xml_grep -p --cond='foo[@id="a"]//[@class="meh"]'
+</pre>
 ==I get "Unknown option --xpath"==

@@ Line 93: / Line 93: @@
 Both xmllint and xmlstarlet seem to sponge the whole file into ram before doing anything.
-What if you just want to run on the first 1000 lines? If you try "head -1000 file | xmlstarlet" it'll complain aince your tags aren't closed. Here's a trick to avoid that:
+What if you just want to run on the first 1000 lines? If you try "head -1000 file | xmlstarlet" it'll complain since your tags aren't closed. Here's a trick to avoid that:
 <pre>

@@ Line 103: / Line 103: @@
 E.g. to get some articles out of a wikipedia dump:
 <code>
-bzcat [https://dumps.wikimedia.org/dawiki/20160111/dawiki-20160111-pages-articles.xml.bz2 dawiki-20160111-pages-articles.xml.bz2] \
+  bzcat [https://dumps.wikimedia.org/dawiki/20160111/dawiki-20160111-pages-articles.xml.bz2 dawiki-20160111-pages-articles.xml.bz2] \
    | head -1000 \
    | xmllint --recover - \
-   | xmlstarlet sel -N w=http://www.mediawiki.org/xml/export-0.10/ -t -c '//w:text/text()'
+   | xmlstarlet sel -N w='http://www.mediawiki.org/xml/export-0.10/' -t -c '//w:text/text()'
 </code>

@@ Line 103: / Line 103: @@
 E.g. to get some articles out of a wikipedia dump:
-<pre>
+<code>
-bzcat [https://dumps.wikimedia.org/dawiki/20160111/dawiki-20160111-pages-articles.xml.bz2 dawiki-20160111-pages-articles.xml.bz2] | head -1000 |xmllint --recover - | xmlstarlet sel -N w=http://www.mediawiki.org/xml/export-0.10/ -t -c '//w:text/text()'
+bzcat [https://dumps.wikimedia.org/dawiki/20160111/dawiki-20160111-pages-articles.xml.bz2 dawiki-20160111-pages-articles.xml.bz2] \
+   | head -1000 \
-</pre>
+   | xmllint --recover - \
+   | xmlstarlet sel -N w=http://www.mediawiki.org/xml/export-0.10/ -t -c '//w:text/text()'
+</code>
 ==I get "Unknown option --xpath"==

@@ Line 89: / Line 89: @@
 Note that on some systems, the command for xmlstarlet is simply "xml".
+==It's slow on big files==
+Both xmllint and xmlstarlet seem to sponge the whole file into ram before doing anything.
+What if you just want to run on the first 1000 lines? If you try "head -1000 file | xmlstarlet" it'll complain aince your tags aren't closed. Here's a trick to avoid that:
+<pre>
+xzcat corpus.xml.xz | head -1000 | xmllint --recover - | xmllint --xpath '//foo/bar'
+</pre>
+The --recover option to xmllint will close end tags for you :-) and prints what it closed to stderr.
+E.g. to get some articles out of a wikipedia dump:
+<pre>
+bzcat [https://dumps.wikimedia.org/dawiki/20160111/dawiki-20160111-pages-articles.xml.bz2 dawiki-20160111-pages-articles.xml.bz2] | head -1000 |xmllint --recover - | xmlstarlet sel -N w=http://www.mediawiki.org/xml/export-0.10/ -t -c '//w:text/text()'
+</pre>
 ==I get "Unknown option --xpath"==
-You need to upgrade your libxml2 (--xpath was added in 2.7.7, back in Mar 15 2010).
+You need to upgrade your libxml2 (--xpath was added in 2.7.7, back in Mar 15 2010). Or use xmlstarlet.
 ==External links==

@@ Line 95: / Line 95: @@
 ==External links==
 * http://oreilly.com/perl/excerpts/system-admin-with-perl/ten-minute-xpath-utorial.html
+==See also==
+* [[ Dixtools: Grep ]]
 [[Category:Documentation in English]]
 [[Category:XML]]

← Older revision		Revision as of 14:13, 18 February 2015
Line 82:		Line 82:


	You can have several -c's per -m, so if you wanted to output all the text from <l>'s and <r>'s under the same <p> in a file, tab-separated, you could do:		You can have several -c's (and -n's and -o's) per -m, so if you wanted to output all the text from <l>'s and <r>'s under the same <p> in a file, tab-separated, you could do:

	$ xmlstarlet sel -t -o $'LEFT\tRIGHT' -n -m '//e/p' -c 'l/text()' -o $'\t' -c 'r/text()' -n apertium-nno-nob.nno-nob.dix		$ xmlstarlet sel -t -o $'LEFT\tRIGHT' -n -m '//e/p' -c 'l/text()' -o $'\t' -c 'r/text()' -n apertium-nno-nob.nno-nob.dix

@@ Line 80: / Line 80: @@
 to make r be LR on the entry with lemma Norge.
+You can have several -c's per -m, so if you wanted to output all the text from &lt;l&gt;'s and &lt;r&gt;'s under the same &lt;p&gt; in a file, tab-separated, you could do:
+  $ xmlstarlet sel -t -o $'LEFT\tRIGHT' -n -m '//e/p' -c 'l/text()' -o $'\t' -c 'r/text()' -n apertium-nno-nob.nno-nob.dix

@@ Line 74: / Line 74: @@
 "sel" means select/query, -t means output as text, -m means "match this XPath expression", "-c ." here means "output what was matched" and -n means add a newline.
+To use xmlstarlet as a sed replacement, try e.g.
+  $ xmlstarlet ed -P -u '//e[@lm="Norge"]/@r' -v "LR" apertium-nno.nno.dix
+to make r be LR on the entry with lemma Norge.
 Note that on some systems, the command for xmlstarlet is simply "xml".

@@ Line 42: / Line 42: @@
 To get all attributes of the e element that has the lm "cake":
 <pre>
-$ xmllint --xpath '//e[lm="cake"]/@*' apertium-eng.eng.dix
+$ xmllint --xpath '//e[@lm="cake"]/@*' apertium-eng.eng.dix
 </pre>
@@ Line 67: / Line 67: @@
 To do more complex xml munging, you might want to install [http://xmlstar.sourceforge.net/ XML Starlet]. The syntax takes a bit of getting used to, but it's quite powerful. If nothing else, it's worth getting because it can append newlines after each selection. E.g.
-    xmllint --xpath '//e[@r="LR"]//text()' apertium-eng.eng.dix
+  $ xmllint --xpath '//e[@r="LR"]//text()' apertium-eng.eng.dix
 would get text content of entries marked LR, but there will be no separator between each match. XML Starlet requires -a -few -more -options, but can output newlines after each match:
-    xmlstarlet sel -t -m '//e[@r="LR"]//text()' -c . -n apertium-eng.eng.dix
+  $ xmlstarlet sel -t -m '//e[@r="LR"]//text()' -c . -n apertium-eng.eng.dix
 "sel" means select/query, -t means output as text, -m means "match this XPath expression", "-c ." here means "output what was matched" and -n means add a newline.
+Note that on some systems, the command for xmlstarlet is simply "xml".
 ==I get "Unknown option --xpath"==