/var partition got filled up immediately.

[govindarajan_1]

Hi experts,

/var partition got filled up immediately to 100%.

error in dmsg was "msgcnt 1439 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 block extent)"

i did fuser -ck /var then i got disconneted from the server, i understand tat all the process have been killed and restarted, i got the connection again,

May i know why the /var got filled up immediately please advice me how to proceed.


Thanks in advance.

[Yogeeraj_1 · Accepted solution]

Hi,

"fuser -ck /var" will normally kill all processes that have open files on /var

"fuser -cu /var" would have been more appropriate.


You can then selectively analyse the processes.

e.g.
SLX1:>fuser -cu /var
/var: 713o(root) 730o(root) 921m(root) 716o(root) 753o(root) 754o(root) 1002o(root) 1308m(root) 1298cm(root) 1471mo(root) 1410mo(root) 1432o(root) 1386co(root) 1618o(root) 2563o(root) 3150m(root) 1445o(root) 2245o(root) 1529m(root) 2146mo(root) 1532o(root) 1505mco(root) 1635mo(root) 2644o(root) 1577o(root) 1600mo(root) 2221mo(root) 1641mo(root) 1675mo(root) 1650co(root) 1653c(root) 1656co(root) 1712mo(root) 1852m(ias) 1808m(ias) 1853m(ias) 1854m(ias) 1855m(ias) 1880m(ias) 2062m(root) 2099mo(daemon) 2082o(daemon) 2107m(www) 2106m(www) 2108m(www) 2109m(www) 2110m(www) 2161mo (daemon) 2160mo(daemon) 2162mo(daemon) 2168mo(root) 2248o(root) 2231o(root) 2247o(root) 3190m(ias) 2299o(root) 2533o(root) 24671m(root) 2783o(root) 3165m(root) 11195m(ias) 9126m(www) 23194m(ias) 3540m(ias) 12942c(root) 3639m(root)

# ps -ef|grep 3165
ias 3166 3165 0 08:08:17 pts/tb 0:00 -sh
root 3165 1022 0 08:08:17 pts/tb 0:00 telnetd
# ps -ef|grep 1808
ias 1852 1808 0 Nov 18 ? 0:00 /u04/product/oracle/1.0.2.2.2/Ap
ache/Apache/bin/httpd -d /u04/p
ias 1808 1 0 Nov 18 ? 0:52 /u04/product/oracle/1.0.2.2.2/Ap
ache/Apache/bin/httpd -d /u04/p
ias 1853 1808 0 Nov 18 ? 11:06 /u04/product/oracle/1.0.2.2.2/Ap
ache/Apache/bin/httpd -d /u04/p
ias 1880 1808 0 Nov 18 ? 0:00 /u04/product/oracle/1.0.2.2.2/Ap
ache/Apache/bin/httpd -d /u04/p
ias 11195 1808 0 Nov 23 ? 0:00 /u04/product/oracle/1.0.2.2.2/Ap
ache/Apache/bin/httpd -d /u04/p
#


kind regards
yogeeraj

[F Verschuren]

some editors keep a tmp file in ./var/tmp/
So if a user eddids a big file it is store in /var/tmp.

Why you where loged out?
if you are standing in a dir and do a -ck on that dir you are killing your own shell.
next time you can better do the comand form / ore use du -ks /var first to see what file is causing the problem, now it is almost imposeble do find the reason

[Yogeeraj_1]

hi again,

To list all open files on device /dev/hd4, you can also use:

lsof /dev/hd4

e.g.
# lsof /tmp
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
snmpdm 1304 root 2w REG 64,0x4 0 920 /tmp (/dev/vg00/lvol4)
snmpdm 1304 root 4u unix 64,0x4 0t0 922 /tmp/.AgentSockets/A (0x
5dd07940)
hp_unixag 1317 root 1r DIR 64,0x4 147456 2 /tmp
hp_unixag 1317 root 2r DIR 64,0x4 147456 2 /tmp
mib2agt 1326 root 1r DIR 64,0x4 147456 2 /tmp
mib2agt 1326 root 2r DIR 64,0x4 147456 2 /tmp
trapdesta 1341 root 1r DIR 64,0x4 147456 2 /tmp
trapdesta 1341 root 2r DIR 64,0x4 147456 2 /tmp
httpd 2479 ias122 mem REG 64,0x4 2105344 6 /tmp/opm_sys_shm.mem
httpd 2479 ias122 7u REG 64,0x4 2105344 6 /tmp/opm_sys_shm.mem
httpd 2479 ias122 13u REG 64,0x4 0 7 /tmp/opm_sys_shm.sem
httpd 2479 ias122 19u REG 64,0x4 0 8 /tmp/opm_sys_mtx.1195415
131
...
#

From this point forward, it is easier to identify the processes...

kind regards
yogeeraj

[govindarajan_1]

du -ks /var will give the /var usage. i did that it was giving around 50% of disk usage . even we have a script will monitor the server every 30 minutes . i saw the last output of that script. it was only 50% usage of /var.

I am asking you if the situation happens again how do i proceed in a proper way.

I am sure that you can help me.

Thanks in advance.

[MarkSyder]

Likely candidates for files which have been allowed to grow too large:

/var/adm/btmp
/var/adm/sulog
/var/adm/wtmp
/var/adm/cron/log and OLDlog
/var/adm/lp/log
/var/adm/syslog/mail.log
/var/adm/syslog/syslog.log and OLDsyslog.log

The files should be monitored frequently to make sure they don't grow too large. Do not delete them as they are needed: null them; e.g. > /var/adm/btmp

Mark Syder (like the drink but spelt different)

[Yogeeraj_1]

hi again,

>>I am asking you if the situation happens again how do i proceed in a proper way.

This situation can happen because of run-away procesess or application that are generating lots of log messages that can make your /tmp reach 100%.

To be able to troubleshoot the problem when it occurs, you will have analyse which processes have open files on /var.

From there you can find the culprit and terminate (kill -15 ) it so that it releases the space used.

You can optionally modify your application level parameters so that /var/ is no longer used to store log files.

hope this helps too!

kind regards
yogeeraj

[Kapil Jha]

>>I am asking you if the situation happens again how do i proceed in a proper way.

I think best proper procedure is
lsof /var
and see which all big files are there , because when u null some file , file is still there in tmp folder and so does not release space.
so in lsof find the process which has opened this files.
Null the files by >filename
restart the process which is holding this file.
Hope this help!!!!
BR,
Kapil

[govindarajan_1]

Hi All,

Thanks for guiding me.

You all are correct.

when i gave bdf it was showing 100%. then i gave du -sk /var it was showing some result. It was 50% only used space.

i understand wat you guys are telling. i will proceed the way you said.

my question is du is showing some result bdf showing some result . why is that happened.even if some of process write more logs du should show max value am i right ?

Please clarify, Please dont mind. i may ask idiot questions but as a beginner i have to ask my experts ( Its you ).

[Kapil Jha]

Just 2 line from man page
man du
du - summarize disk usage
man bdf
bdf - report number of free disk blocks

I hope above discussion + these two line would solve ur confusion.

In case od confusion best way is RTFM....
BR,
Kapil

[Yogeeraj_1]

hi again,

du and df/bdf calculate different things, see corresponding man pages

probably there is a process still using the inode, then the file will be removed from the filesystem, but not from disk itself.

As mentioned previously, you can check with ls -i or with lsof commands.

hope this helps!

kind regards
yogeeraj