[CentOS] Optimizing grep, sort, uniq for speed

Thu Jun 28 18:57:34 UTC 2012
m.roth at 5-cent.us <m.roth at 5-cent.us>

Sean Carolan wrote:
> This snippet of code pulls an array of hostnames from some log files.
> It has to parse around 3GB of log files, so I'm keen on making it as
> efficient as possible.  Can you think of any way to optimize this to
> run faster?
>
> HOSTS=()
> for host in $(grep -h -o "[-\.0-9a-z][-\.0-9a-z]*.com" ${TMPDIR}/* |
> sort | uniq); do
>     HOSTS+=("$host")
> done

For one, do the sort in one step: sort -u. For another, are the hostnames
always the same field? For example, if they're all /var/log/messages, I'd
do awk '{print $4;}' | sort -u

       mark