dealing with too many files in dir

[Dave Chamberlin]

In running Oracle Apps (10.7) all log files and output files are written to common directories (logPROD and outPROD) on my system. The log dir has between 70000 and 100000 files at a given time, and I think that this is a performance problem on this disk - which always has higher activity than others. An ls -l shows the directory itself is about 3.5M. The files in the dir consume about 2G. I have read earlier posts about performace problems due to large numbers of files, and If I understand correctly - that 3.5M piece is being constantly being rewritten. I can change Oracle to write the files to the default log/out dirs in the APPL_TOP. This spreads out the files by application, but there is not enough space there and I also worry about contention between writing log files and reading APPS excutables etc. I am thinking I can I create symbolic links in those APPS dirs, putting the actual files on a disk with more space - thus solving both problems. My question is what is the performance hit of using links here? What activity will it create on the APPS disk? Any suggestions would be appreciated. Thanks

[John Poff]

Hi Dave,

We ran into the same problem and we created separate filesystems for the log and out directories. You shouldn't have to fool with symlinks as you can mount those filesystems inside of your APPS directory tree.

It won't completely solve your performance problem as you'll still have 70000-100000 files in a single directory and filesystem, but at least they will be separate from your APPS filesystem, and if they fill up the out or log filesystem [which they will!], it won't kill your APPS filesystem.

JP

[John Poff]

Hi again,

I didn't think we were using symlinks in our setup [we didn't used to], but then I got to looking and I remembered that when they upgraded Oracle last year they changed the filesystems and decided to symlink over to the out and log dir/filesystems. It doesn't seem to hurt the performance.

JP

[Manju Kampli]

these log files are generally not used online and kept for history purpose when things go wrong.. we used to run a script which used to move the older files to a subdirectory and keeping the number of files in that directory to small number...the subdir used to contain one month log files

HTH
Manju

[Bill Hassell]

There aren't any performance penalties using the existing files by name, but directory searches and creation of new files will be very slow since the directory structure must be traversed. And of course, using ls or filename matching like echo a* will be very slow or fail completely because the list of files far exceeds the maximum line length for most shells.

Create a hierarchy, ideally completely away from data and index disks, where the more directories you can create, the easier it will be to manage the thousands of files.

[Tim Medford]

Dave - If these are just log files that don't need to be accessed on a regular basis, you may considering tar'ing them into blocks of 1000 files.

I did a similar thing with a directory that contained about 1 million small files. I ended up with only 1000 tar files instead. Also wrote a script that automated the process on a weekly basis.

Just a thought,
Tim

[Dave Chamberlin]

Thanks for the replies. I must keep the files accessable by oracle apps for 4 days - then I can just delete them (I currently use cron for this). I am currently trying the link idea on my dev box and it works ok so far.