/var is full

[Eli Daniel]

HI,
extended FS / var at 15 gb and scored 100% in 5 minutes

before size FS 8 GB
later size FS 15 GB

5 minutes the FS is FULL

SAPL918_RX6600:/var> du -sk *
0 X11
719016 adm
0 asx
0 empty
352 evm
0 home
26032 jail
0 lost+found
1112 mail
0 news
0 nfs
0 nfs4
263736 opt
0 preserve
240 psb
112 run
488 sam
112 spool
8 statmon
76408 stm
0 symapi
40 tdp_r3
5216 tmp
0 tombstones
0 uucp
112 yp


SAPL918_RX6600:/var> bdf .
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol8 15728640 15728640 0 100% /var

[Bill Hassell]

The directory /var/jail is very suspicious (ie, why is it so large?). And /var/opt is massive. That is likely the problem. There is a runaway program creating/using a lot of space. Since the directory itself is very large, it means there are a very large number of files. Use this command to find them:

du -kx /var/opt | sort -rn | head -20

[Earl_Crowder]

Eli,

As TTr asked, was a logfile or other large file deleted? Perhaps /var/adm/syslog/syslog.log or something in /var/tmp?

If the file was deleted while it was being used, the only way i know of to find it is using lsof.

For example, this command will list files in /var filesystem larger than 1GB that are opened but have been removed (unlinked):
#lsof /var | awk '/REG/&&/lvol/{if ($7>1073741824){print $0}}'

Earl

[Bill Hassell]

I missed /var/stm in your list (that's why sort is so important). A large amount of space in /var/stm (especially when it grows) indicates you have a hardware problem. Look at syslog.log for hardware errors.

[TTr]

/var is 15GB but if you look at the du listing it is nowhere near that. So there is unaccounted space somewhere, most likely is a deleted file.

If there is a hardware error, syslog would do a lot of logging and since /var/adm is only 719MB, the syslog file was deleted.

If there is no hardware error, some other logfile was probably deleted while open.

Run the lsof command as indicated above and show us the output of "ll /var/adm/syslog" and while at it the output of "ll /var/tmp".

[Eli Daniel]

Thanks for information

[TTr]

So what was the cause of all this. Somebody else could have the same problem and can benefit from your resolution.