One-liners

Useful (mostly) bash one-liners

• Perl regular-expression for removing all tags after the initial:

perl -pe 's/(\^[^<]+<[^>]+>)(<\w+>)*\$/\1\$/g;'

^Lemma<V><Pres><Sg>$ -> ^Lemma<V>$

• Get unknown words from chunked text and sort by frequency:

sed 's/\$\W*\^/$\n^/g' | grep '@' | sed 's/><.*/>$/g' |  sort -f | uniq -ci  | sort -gr

tr " " "\n" | grep "@" | tr -d "[:punct:]" | sort | uniq -c | sort -r

• Strip newlines:

sed ':a;N;$!ba;s/\n//g'

Alternatively: tr '\n' ' '