High CPU usage in file movements

[kenny chia]

Hi
I have a script that moves a file into a directory that contains 80,000 files. I noticed that this consumes huge amount of CPU resources (about 80%). Will a destination directory with alot of files cause a high CPU usage?

[twang]

I think the problem could be the disks with high utilization during movement.
If the utilization are very high, will be necessary to divide the load between many disks, maybe using disk stripe.

Check for bottlenecks during movement:
# sar -u 5 5
# sar -d 5 5
# vmstat 5 5
# sar -v 5 5
# swapinfo -tam

[Steven E. Protter]

eeek!

Collect some data, but I'll tell you now, thats too many files for one diretory. That's not a technical answer, I don't know what the limit is. I do know an ls or find command will beat the system hard in that directory.

Subdirectories exist to organize files logically.

The table of file information(non technical term) must take a lifetime to load.

There is no need to stop this process. Let it finish and see what you can do to improve it.

SEP

[kenny chia]

Hi SEP
I ran some of your attached scripts. Attached is the zip file of the results.
There are some high usage spikes, caused by the move command. The spikes are also caused by Oracle Log switches every 7 mins :-(

[Jakes Louw]

Have a look at the filesystem fragmentation using "fsadm -D -E". If there is fragmentation, you will need OnlineJFS to fix, or else back up the filesystem, do a NEWFS on the filesystem to wipe out the structure, and then restore.
There are also several kernel params that help with FS performance: vxfs_max_ra_bytes, vx_ncsize are the two main boys here.

[Mark Grant]

I do not think moving one file into a directory with 80,000 entries in it, of itself is going to cause you too much problem (assuming the file you are copying isn't too big). I seem to remember the only performance issues with directories are that maximum perforance is gained when they are exact multiples of disk block size, in size. This might not be true any more though.I agree with SEP that you may well be having other issues with regards to disk IO. You might be able to test if the directory size is causing you a problem by using "mv" within that directory and seeing how long it takes. "mv" when moving a file within the same filesystem only changes the name in the directory where as "cp" and "mv" when "mv" is moving across filesystems essentially copy all the data and change the name in the directory.

[john korterman]

Hi Kenny,

the problem may lie in the size of the directory file itself. What does an ls -ld of the directory show?
If you have 80000 files in the directory there are also 80000 references to these in the directory file itself. You probably have more, as a directory files does not shrink, even when files in the directory are deleted. Instead, deleted files are marked as free/available in the directory file, and when you add a new file to the directory, the system will look for a free/available file reference in the directory file to use, and this activity could consume a substantial amount of your cpu power.

regards,
John K.