Performance - please suggest

[GnanaShekar]

Hi,

The user is executing below find command on a NFS/NAS share. This command is executed on a RHEL4 U3 64bit Virtual machine(1CPU, 1GB RAM).

find -name out.\*.gz -mtime -5 -exec zcat {} \; | grep -c "date_posted"

The time required to complete this is 28 minutes.
Please suggest/recommend what can be done to get this done quickly.

Thanks.

[Steven E. Protter]

Shalom,

There probably is no way to speed this up.

Your speed is effected most by NFS read/seek time.

The inquiry itself is complex, if you could simplify it and make it less CPU intensive that would help.

What are you trying to accomplish? check the contents of some zcat'd files for certain content? That is hard on the disk and the CPU.

SEP

[Alan_152]

I generally agree with Steven. However, you might want to make sure that your network connectivity is optimized as well as possible. Check the connection rate, duplexing, and if any errors are being reported.

[Justin_99]

Can this be ran directly on the NFS server? I have seen some performance increases when running find on the NFS server vs. across the NFS mount. If this is not possible maybe two step the processes, generate the list with find then zcat grep the list? Just a thought.

[Ivan Ferreira]

I think that NFSv4 will help.

Also, you could split your command in more than one, if you have enought cpu available and your network/disk bandwidth allows it.

For example:

RES1=`find /path1 -name out.\*.gz -mtime -5 -exec zcat {} \; | grep -c "date_posted"`
RES2=`find /path2 -name out.\*.gz -mtime -5 -exec zcat {} \; | grep -c "date_posted"`
RES3=`find /path3 -name out.\*.gz -mtime -5 -exec zcat {} \; | grep -c "date_posted"`

expr $RES1 + $RES2 + $RES3

Using this approach, ensure that your find command does not overlaps parent/child directories.

[GnanaShekar]

Hi,

Thanks for all your inputs.

We want to run the find / grep commands on the NFS/NAS mounts.

During this operation, I want to measure the network traffic. I want to measure what is the download speed.

I guess there will be a lot of data being downloaded from the NFS/NAS share on to the temporary local filesystems for processing (correct me if I am wrong).

I guess the time that this operation takes will depend on the network speed, local memory and cpu capabilities (correct me if I am wrong).

Please suggest,

[Robin T. Slotten]

You are asking find to read every file on the NFS that matches *.gz. This means every one of those files will be transfered across the NFS mount in order to grep for the string. Running the command locally on the NFS server or via an rsh command would likely sped things up for you.

[Mike Stroyan]

Your performance may be limited by by grep rather than NFS. You could compare the speed of your search to the speed of
find -name out.\*.gz -mtime -5 -exec zcat {} \; > /dev/null

Why should a simple grep be slower than an NFS transfer? First, zcat may expand the files a lot after the NFS transfer. Second, grep in RHEL4U3 has a really bad performance problem with locales that have utf8 character encoding.
See the comment about "Back-ported egf-speedup patch (bug #179636)." in
http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/release-notes/as-x86/RELEASE-NOTES-U4-x86-en.html
and the defect report at
https://bugzilla.redhat.com/show_bug.cgi?id=69900

You can work around the grep performance problem by setting "export LANG=C" before running grep.