Re: [CentOS] Optimizing grep, sort, uniq for speed

Sean Carolan wrote:

Thank you Mark and Gordon. Since the hostnames I needed to collect are in the same field, at least in the lines of the file that are important. I ended up using suggestions from both of you, the code is like this now. The egrep is there to make sure whatever is in the 9th field looks like a domain name.

for host in $(awk '{ print $9 }' ${TMPDIR}/* | egrep "[-.0-9a-z][-.0-9a-z]*.com" | sort -u); do HOSTS+=("$host") done

*sigh* awk is not "cut". What you want is awk '{if (/[-.0-9a-z][-.0-9a-z]*.com/) { print $9;}}' | sort -u

No grep needed; awk looks for what you want *first* this way.

mark, who learned awk in the very early nineties, writing 100-200 line awk scripts....