[CentOS] Optimizing grep, sort, uniq for speed
m.roth at 5-cent.us
m.roth at 5-cent.usThu Jun 28 18:57:34 UTC 2012
- Previous message: [CentOS] Optimizing grep, sort, uniq for speed
- Next message: [CentOS] Optimizing grep, sort, uniq for speed
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Sean Carolan wrote: > This snippet of code pulls an array of hostnames from some log files. > It has to parse around 3GB of log files, so I'm keen on making it as > efficient as possible. Can you think of any way to optimize this to > run faster? > > HOSTS=() > for host in $(grep -h -o "[-\.0-9a-z][-\.0-9a-z]*.com" ${TMPDIR}/* | > sort | uniq); do > HOSTS+=("$host") > done For one, do the sort in one step: sort -u. For another, are the hostnames always the same field? For example, if they're all /var/log/messages, I'd do awk '{print $4;}' | sort -u mark
- Previous message: [CentOS] Optimizing grep, sort, uniq for speed
- Next message: [CentOS] Optimizing grep, sort, uniq for speed
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the CentOS mailing list