files created and deleted quickly

[SILVERSTAR]

Hi,

I am trying to understand which kind of issue may I run into with this scenario:

I have an application which I do not manage and about which I do not know quite nothing apart that it will create on the UX box about 90~100 files on a VXFS in one second.
This application runs 24X7 days.
The files are large 1Mb max, they are created, readed by a process and then deleted.
The creation, the usage and the deletion happens in few seconds maximum one minute.

I would like to know:
-any limitation to the number of files in FS (probably none)
-can the process of creation/deletion become very slow ?
-Will system resources go in crisis(cpu,memory,disk)
-each raccomendation you may suggest.

Thanks
Angelo

[Fred Ruffet]

I see these :
. max_files and max_files_lim. Ensure that these kernel parameters will allow you to open enough files at the same time.
. Have a look at ulimit for the user that will launch the process. He may have limitation on open files.
. As long as files are not so big, you may not run out of memory, but it depends on how much you have. Same for CPU. Just to be sure, check yu have Swap = twice the RAM.
. Disk IO rates. This may be a blocking point. Depending on your hardware, on your raid-level, etc you may have differences in perfs. Just run a test using dd may help you determine if your disk speed will be enough.

Regards,

Fred

[Mobeen_1]

Angelo,
I agree with my earlier post that you need to watch out for the following params

max_files and max_files_lim

Make sure that these are large enough and you don't run into any issues. An issue may arise when the no. of files created on your VXFS equals these values. In such an instance you would not be able to create any files and your file creation process will fail. So make sure on these and keep an eye.

Another advise, when you are talking about the kind of IO you explained, i am sure you would be better off splitting the IO across multiple disks. I cannot comment without knowing the requirements of your application. But i can say for sure that, if you have a maximum IO on a filesystem, then by identifying and moving groups of files across multiple filesystems will defintely give you better performance.

regards
Mobeen

[Volker Borowski]

Hmm,

100 files / sec
60 sec lifetime
1 MB / file

... gives you a maximum utilisation of 6000 files with a total max size of 6 GB within one minute.

This means you need to pull 6GB per Minute through the filesystemcache at max !!!!
So go for a lot of RAM and a big fs-cache.

Strange Application behavior....
How does it ensure consistency when the box crashes while 6000 flat files are processed ??? May be the designers should go for a database approach ?

Volker