Quick and Dirty Data Extraction in AWK

I couldn't live without it. I've done more than you can imagine with this "simple" utility, including munge perl (perl you say?) yes, perl.

Because of a2p (Awk to Perl) you can "think" in awk, then translate to perl. Works like a charm for those quick and dirty assignments where you need something CPAN can do, and have some data munging to do.

I've used perl for quite some time, I still like the "feel" of awk better. The fact that it teaches pattern scanning is huge.

You could write a howto about it. (hint)

I have used awk for many, many years. In 1990, I was responsible for getting an aerospace engineer to generate finite element model (MSC/NASTRAN) input files using awk.

I have written a ".ini" file parser in awk, which consisted of a really weird state machine. It makes my head hurt just to look at that code now, even though I run it daily.

My favorite "awk" scripts (all 3 of these are one-liners, by the way):

1) Calculate mean (arithmetic average): awk 'BEGIN{sum=0;cnt=0;}{++cnt;sum+=$1}END{print sum, " ", cnt," ", sum/cnt;}'

2) Switch two columns, which is surprisingly useful: awk '{print $2, $1}'

3) Calculate median of a single column of numbers: sort -n | awk 'BEGIN{c=0;sum=0;}{a[c++]=$1;sum+=$1;}END{ave=sum/c;if((c%2)==1){median=a[int(c/2)];}else{median=(a[c/2]+a[c/2-1])/2;}print sum," ",c," ",ave," ",median}'

I've migrated to Python over the last year, but I used to do a lot of Perl. I still maintained a working acquaintance with "awk" over the years.

I use {g}awk daily, although I'm getting into the habit of applying Ruby to things instead, via a simple skeleton app. It never ceases to amaze the onlookers when some intractible problem which looks like requiring days of hand-editing turns into fifteen minutes of awk hacking and an instant's runtime.

As per the UNIX® "no app is an island" philosophy, I also have fun doing stuff like this (lowercases a directory full of files):

for x in *[A-Z]*; do mv $x $(echo $x | tr [A-Z] [a-z]); done

Or this (eliminates duplicates, I use it on large TrueType collections):

for x in $(md5 *.png | sort | uniq -w32 -d | cut -f 2); do rm -vf "$x"; done