General
cancel
Showing results for 
Search instead for 
Did you mean: 

low performance when running 'grep' and 'awk' scripting over large text file

SOLVED
Go to solution

low performance when running 'grep' and 'awk' scripting over large text file

Our application algorithm has to parse 12 times over a large text file (size=148MB).
We ran the same script over the same file on different machines and strangelly the worst performante was produced on our brand new server: an HP Proliant DL380G5 w/ 10krpm SAS disks. I suppose that it must be a question of tunning the kernel. What can i do to enhance the application processing time on this server?

TEST RESULTS (same script, same large file, different servers):
i) Laptop PC w/ Celeron processor runnig GentoOS (BEST PERFORM) - 1'38"
ii) HP Proliant DL380G3 Xeon 3GHz running RHEL3.6 - 6'14"
iii) HP Proliant DL380G5 2xXeon 2-core 3,4GHz running RHEL4.4 - 11'26"
3 REPLIES
Ermin Borovac
Honored Contributor
Solution

Re: low performance when running 'grep' and 'awk' scripting over large text file

This is a long shot by what is environment variable LANG set to?

In the past it's been reported that grep, sort and friends can be slow if LANG is set to one of the UTF variants.

Try

$ LANG=C grep ...
Steven E. Protter
Exalted Contributor

Re: low performance when running 'grep' and 'awk' scripting over large text file

Shalom,

Interesting results.

The Os is different.

The processor is different.

The amount ofmemory is likely different.

The Patch level is different.

The storage(disk) performance is likely different.

Any of the factors above could impact the results.

More memory and a larger buffer cache will normally improve performance in such sitations. There are a lot of variables here.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com

Re: low performance when running 'grep' and 'awk' scripting over large text file

BINGO! That is impressive: setting var env LANG, the execution time has reduced from 11' to 48". Thank you Ermin Borovac.