[CentOS] Optimizing grep, sort, uniq for speed

Thu Jun 28 19:37:46 UTC 2012
m.roth at 5-cent.us <m.roth at 5-cent.us>

Sean Carolan wrote:
> Thank you Mark and Gordon.  Since the hostnames I needed to collect
> are in the same field, at least in the lines of the file that are
> important.  I ended up using suggestions from both of you, the code is
> like this now.  The egrep is there to make sure whatever is in the 9th
> field looks like a domain name.
>
> for host in $(awk '{ print $9 }' ${TMPDIR}/* | egrep
> "[-\.0-9a-z][-\.0-9a-z]*.com" | sort -u); do
>     HOSTS+=("$host")
> done
>
*sigh*
awk is not "cut". What you want is
awk '{if (/[-\.0-9a-z][-\.0-9a-z]*.com/) { print $9;}}' | sort -u

No grep needed; awk looks for what you want *first* this way.

     mark, who learned awk in the very early nineties, writing
              100-200 line awk scripts....