Informix eating up /var (info on Inodes)

[Mike Rightmire]

We are using an informix database, which is eating up about 10% of our /var each day. Informix should not be touching var at all, however I am finding all the oninit processes as using up raw space.

When I do an fuser on var, it shows little or no activity touching the directory. When I do an fuser to the /dev/vg##/lvol# (which is the /var directory) I get MANY things touching the device.

When I do a bdf command it shows /var at 74% usage, however when I do a du -sk on /var it shows 65 megs of data being used out of 1 GB (this is what leads me to believe informix is using the device as raw space.)

If I kill the informix database, the /var directory again drops down to 4% usage, and begins climbing again until, in a few days, the /var is at 100% usage and the system starts choking.

I am thinking I need to track what inodes are being opened, by whom, and determine if informix is opening raw space for something and not closing it....and why.

Please help...any advice would be great.

[Volker Borowski]

Hello Mike,

this sounds very strange !!
Can you take the system to single-user mode, unmount /var and do an fsck of the filesystem.

I can barely imagine, that Informix is writing anything on this filesystem by directly going to the device without destroying it.

I think the oninit processes should belong to user and group "informix" in a standard installation. This user should not be able to write to the lvols (rlvols) belonging to "bin" or "root", unless you permit it.

So if fsck returns no error, your problem is a controlled, correct (but may bad programmed) access to the filesystem. May be there is an Informix patch available, are you up to date ?

Second option: filesystem-swap configured on /var ? (I think this space does not show up in "du" )

Do not know if this helps
Volker

[Mike Rightmire]

Hey Volker,

Thanks for the response. Unfortunately, I am not able to brin ght e system down to do a fsck at this time as it is a production box. However, some additional information.

I have Informix support on the line...they cannot figure it out but they have not mentioned a patch.

There is no swap swetup for the same device as var.

When I do an fuser against /var, there is nothing using the area...but when I an fuser against its logical volume (as mentioned above) it shows a ton of stuff touching it. MOST of this stuff (besides the oninits) are system processes. Some of these include ...

syslogd -D
swagentd
lpsched
rbootd
rlogind
cron
ntl_reader
rpcd
rpc.locked

...ETC.

This leads me to believe that the oninits are touching the drive in the same way and for the same reason as these system [processes and deamons, however (due to corruption, bad programming, whatever) the oninits are not using it properly.

Why would these processes be touching the device in that way (remember they are not touching /var....but touching /dev/vg##/lvol#, which is the device /var resides on) and why would oninit be touching it in the same way and failing to do it correctly?

Thanks again!
Mike

[Gadura Praveen]

Mike,

I am not very familiar with informix, but like other applications it might be using "/var/tmp" directory as a temp directory.You can use lsof ( available in gnu ) to track down all the processes using "/var". Also , if you have glance installed on the server , you can select a informix process and track down all the files opened by this process.

Hope this helps !!

[Mike Rightmire]

I did an "lsof /var" and came up with some interesting data. All of the Informix oninit processes are listed as VREG type and are using exactly 694756928 as the size....as are a number of other processes running on the box. I am not sure how to interpret this. I do not know what VREG type is, but I DO know there are NO actual physical files open by any of these processes including in the /var/tmp dir...???????

[Mike Rightmire]

One more piece to the puzzle. All the procs which are showing the same size are also showing the same node of 1893....??????

[Patrick Wallek]

Mark,

Have you tried finding out what that inode number is? Do a 'find /var -inum inode# -exec ll -d {} \;' to see what file it is and let us know.

Also, when you do an fuser on /var what syntax are you using? If you are just doing 'fuser /var' then your output is to be expected because it is just looking at the files in /var and not any of the subdirectories. When doing an fuser on a LV mount point I generally do 'fuser -cu /var'. That should show the same results as your 'fuser /dev/vg??/lvol?'.

I agree that this is very strange behavior.

[someone_4]

I dont know much about informix but right now we are having a replication issue.
I did the command
# fuser -cu /var
and here is what i got ..
what does all this stuff mean?
/var: 2100m(root) 391o(root) 1247m(root) 394o(root) 417o(root
) 418o(root) 549o(root) 958co(root) 1041m(root) 980mo(root)
866cm(root) 996mco(root) 1109o(root) 1172o(root) 1170o(root)
1051o(root) 1169o(root) 1160o(root) 1173o(root) 1267m(root) 1487m
o(root) 1183o(root) 1248m(root) 1249m(root) 1236m(root) 1246m(roo
t) 1245m(root) 16164m(root) 2318m(root) 1320m(root) 9840m(root)
4391c(root) 4390m(root) 2028m(root) 2174m(root) 5176cm(root)

[Patrick Wallek]

The output you have is all of the processes (pid) and the user running the process that are accessing have opened a file somewhere in /var.

Lets take 2100m(root) as an example:

The number (2100) is the pid of the process. You could do a 'ps -ef | grep 2100' to see what exactly this is.

The character in the middle tells you what the process is doing with the file. There can be more than one character here. See the character meanings below.

The user name in parentheses is the login name of the user that started the process.

The characters in the middle field, m in this example, can be any of the following (taken from the fuser man page):

c - file is its current directory.
r - file is its root directory, as set up by the chroot command (see chroot(1M)).
o - It has file open.
m - It has file memory mapped.
t - file is its text file.