/var usage over 90% issue

[jack Hu_1]

Dear Sir:
I found my L2000 server's /var usage over 90%.
bdf show that allocated 700MB.
But the du -sk show only 200MB allocate on /var directory.
So I use the lsof command to collect the result.
Can anyone tell me,how to use the lsof log and find the unnormal processes.
please see the attachment file.
thanks.
Jack

[Steven E. Protter]

Couple of rough solutions:

1) boot your system. You have processes holding the files that were deleted which is holding space. Now that the files are gone they will be hard to kill.

2) If you don't mind killing every user on your system you can check and kill processes on the var system as follows:

fuser -cu /var
checks
fuser -cuk /var kills all processes. every telnetd process opens a process on /var so running that second command will be disruptive.

Any time you get results like that eventually you are going to need to boot. I would also consider how much space you allocate to /var I would say you need more. I tend to allocate several Gigabytes more than typically needed so that I can survive my own mistakes without disrupting users.

SEP

[jack Hu_1]

I just found the lsof log have two AP's process.
The result show that allocate 500MB size.
I will try to stop these two processes first.
Jack

[Con O'Kelly]

As Steven has pointed out there is a process(es) holding open a file(s).

As you have said its possible to use lsof to try & identify these files. What you need is all the open files for /var. Then check through these files & identify any large file sizes that are potential culprits. If you find any check if they show up in a ls listing, if not these could be your culprits.

Cheers
Con

[Con O'Kelly · Accepted solution]

Hi Jack

Noticed after I posted you'd already found the open files.
Command that might be useful is
grep "/var" | sort -nr -k 7.
This shows the offending files. I believe killing that PID (if possible) should free up all the space. Note the file size almost exactly matched the discrepancy in space reported by du & bdf.

This is often a much simpler solution than re-booting.

Cheers
Con

[Helen French]

Is bdf and du shows allocated space OR "used" space ? what's the LV size? did you delete any files recently? bdf will not report the correct sizes all the time, especially when processes are still holding files on it. Your du output will be the correct value. You don't need to worry about it since bdf will show the correct values after those processes get killed. If you don't use bdf outputs in scripts, I wouldn't worry about this issue.

Also, /var normally contains the log files and spool files. You may need to trim some of them - syslog.log, /var/adm/crash files, wtmp and utmp, /var/tmp files and other log files. If the file system got full recently, I would check what changed recently? Any big files in the file system:

# find /var -xdev -type f -depth -size +10000 -exec ll {} \; > /tmp/big_files

# find /var -xdev -type f -depth -mtime +3 -exec ll {} \; > /tmp/new_files

[ashan]

Hi Jack,

Some time a go I face the same problem,
what I found is /var/tmp has lots of old
files, once I clean them I came back to my
expectation amount.

Regards

Ashan

[Jason_216]

hi jack,

i also got the same problem with my server last, but just don't be panin, help throught itrc is all around.

in ur case, there r so many solution giving to u, i just remind u that, we should keep housekeeping all the time 4 teh log file, in my case, i purposely create a LV to store all the old log file, i don it every weeked

bye
goodluck

jason

[twang]

You may need to trim logfile, remove dump from /var/adm/crash (if applicable), or increase size of /var:
1. look for directories that are very large...there is likely something causing either large files of a massive number of small files. Likely candidates are /var/adm, /var/mail, var/tmp:
du -k /var | sort -rn
2. to increase /var size, if you have online JFS on your system by using the following command:
#fsadm -F vxfs -b 1300M /var

If you don't have online JFS, need to boot the system in to single user mode:
#lvextend -L 1300
# extendfs -F vxfs

[jack Hu_1]

very thanks for all your help.
I have check all the log files first.=> It's all ok.
After I kill the two processes, /var free 500MB.
Next time,I will re-arrange the /var file system size.
The lsof is a good tool.

thanks
Jack

[rachel_11]

Look at the line:
new_subsc 5980 loader 1w VREG 64,0x8 547242571 2658 /var (/dev/vg
00/lvol8)

In the lsof output file.
It seems like a huge file, and it's name does not appear (already deleted).

So, the process new_subsc is keeping it.

When this process will terminate, the blocks of this file will get free.

[Ian Dennison_1]

FYI,

Any problems with /var filesystem space usually lead me to the /var/mail directory (cron sticks an email in the mailbox of every user for every cron job run).

Null Points pse, Ian