need suggestion on likely IO intensive files in /var

[Dan Copeland]

Admins,

We have a large oracle database server (64 way SD, 196GB RAM). During the avg daily activity, I see IO rates btw 70 and 100 for /var. Of these IO's, about 98% are writes. Besides /var/tmp, any other potential high IO write rate files in var for a very intesive oralce OLTP env.

I'm planning on adding an EMC SAN device and mounting it as /var/tmp. I suppose I could move /var to the EMC device to be sure I get all the high IO files, but I wanted to be able to boot even if the SAN device was not available...

Any thoughts/recommendations?

I could also use any white papers about SAN disk in vg00 -- I saw one link in the archives, but the doc was missing/moved.

tia,
Dan

[Steven E. Protter]

/var which in my mind means variable filesystem is where variable, changeable, almost temporary output goes.

If oracle is producing large print files on a system with a fast CPU, I can see that as a place to look.

I've never had i/o issues with log files like /var/adm/syslog log files.

If you are producing PDF reports, it could be even more data that needs to be written. The fact that its a lot of writes says print spooling to me.

SEP

[Brian M Rawlings]

Interesting question, Dan. With so much write activity, I think you should be able to see where it is happening by scripting up a 'du -sk /var/*' or some variant. If yo u run it once a minute (temporarily, not all the time), this should let you see what files and/or directories are changing in size.

Since you aren't necessarily filling /var up, it sounds like print queues, as SEP suggests, since these get flushed once the print completes. There are other things in /var that come and go, however... sockets act like this, I seem to recall, but I don't remember them taking up much space, they're more placeholders than anything else.

Anyway, try the du command for a look at what is changing in /var, maybe it will give you a leg up on this curiosity.

Lastly, don't put /var out on the SAN. All standard root VG file systems are best left on the boot disk, IMHO. You know why... you need a working system to troubleshoot a non-working SAN. Putting /var/tmp out there? I think that could work... if it proved to be the busy directory or FS, if you mount it up as its own FS.

I'm sure there are more sophisticated ways to monitor I/O to /var or any FS, but if you can see the source of your trouble from a quick and dirty du command, life is good.

Regards, --bmr

[Sundar_7]

Just a thought, one more thing I will look in to is the mail activity in the system.

How about /var/spool/mqueue. Do you see lot of mail traffic in the system ?

[Abdul Rahiman]

/var shouldn't see very high write rates on a typical system. Having said that, it all depends on the applications that are runing and various other system daemons/logs that are set to use the variable file system.
Few things to look for,

1. entries in the /etc/syslog.conf and see if any of the common logs are set for high debugging, for eg; if lpr is set to log each debug entry for every printer, that could easily produce a lot fo logs.

2. Check the mail queues that are being updated frequently.

3. Check the cron logs.

4. Check the print spool directories and monitor the spooler activity.

5. Run a find command to look for new files being created by using the -newer option.
eg: find /var -newer 'somefile' -print
and finally, separate any application related log files that may be using the /var filesystem inadvertently by moving them into a different filesystem than relocating /var itself to the SAN.
Else you should think about relocating the entire root volumegroup to the SAN.

just my $.02,

regds,
Abdul.

[Victor Fridyev]

Hi,

As far as I know, Oracle DBs cause high I/O rate in /var/spool/sockets directory.
I'm not sure that put /var on a SAN device is very good solution. If you have a free disk, which you can include in vg00 and move to it /var, IMHO, this is better.

HTH