/var is full

[Eli Daniel]

Hi, i have the problem
the system /var is full
anex the file large
SAPL918_RX6600:/tmp> cat var_files.txt | sort +4 -5nr | head -n 30
-rw-r--r-- 1 root root 26670362 Oct 21 16:57 ./opt/perf/datafiles/logproc
-r--r--r-- 1 bin bin 22364160 Dec 4 2008 ./opt/wbem/repository.tar
-r--r--r-- 1 bin bin 20974032 Jul 9 2007 ./adm/sw/save/PHKL_40640/VXFS-BASE-KRN/usr/conf/mod/vxfs
-rw-r--r-- 1 root root 18361996 Oct 21 16:55 ./opt/perf/datafiles/logdev
-rw------- 1 sfmdb users 16777216 Oct 6 01:05 ./opt/sfmdb/pgsql/pg_xlog/0000000000000008
-rw------- 1 sfmdb users 16777216 Oct 21 01:05 ./opt/sfmdb/pgsql/pg_xlog/0000000000000006
-rw------- 1 sfmdb users 16777216 Oct 21 01:07 ./opt/sfmdb/pgsql/pg_xlog/0000000000000007
-r-xr-xr-x 1 bin bin 16009352 Jan 12 2007 ./adm/sw/save/PHSS_38487/KWDB-RUN/usr/contrib/kwdb/lib/kwdb.ia64.so
-r-xr-xr-x 1 bin bin 15830852 Jan 12 2007 ./adm/sw/save/PHSS_38487/KWDB-RUN/usr/contrib/kwdb/lib/kwdb32.so
-r-xr-xr-x 1 bin bin 15732840 Jan 12 2007 ./adm/sw/save/PHSS_38487/KWDB-RUN/usr/contrib/kwdb/bin/kwdb.ia64
-r-xr-xr-x 1 bin bin 15577252 Jan 12 2007 ./adm/sw/save/PHSS_38487/KWDB-RUN/usr/contrib/kwdb/lib/kwdb64.so
-r-xr-xr-x 1 bin bin 15517184 Jan 12 2007 ./adm/sw/save/PHSS_38487/KWDB-RUN/usr/contrib/kwdb/bin/kwdb32
-r-xr-xr-x 1 bin bin 15285828 Jan 12 2007 ./adm/sw/save/PHSS_38487/KWDB-RUN/usr/contrib/kwdb/bin/kwdb64
-r-xr-xr-x 1 bin bin 14102952 Aug 13 2008 ./adm/sw/save/PHSS_40538/C-KRN/usr/ccs/bin/ld
-r-xr-xr-x 1 bin bin 13179112 Jan 12 2007 ./adm/sw/save/PHSS_37004/DCE-IA64-SHLIB/usr/lib/hpux64/libdcekt.so.1
-r-xr-xr-x 1 bin bin 13064296 Jan 12 2007 ./adm/sw/save/PHSS_37004/DCE-IA64-SHLIB/usr/lib/hpux32/libdcekt.so.1
-r-xr--r-- 1 root sys 11151872 Jul 13 2009 ./adm/sw/products/SFM-CORE/EMT_COREIA/cerdump-min
-r--r--r-- 1 bin bin 9928518 Feb 15 2007 ./adm/sw/save/PHSS_37501/LANG-64ALIB/usr/lib/hpux64/libstd_v2.a
-rw-r--r-- 1 root root 9065480 Oct 21 16:55 ./opt/perf/datafiles/logtran
-r--r--r-- 1 bin bin 8580110 Feb 15 2007 ./adm/sw/save/PHSS_37501/LANG-MIN/usr/lib/hpux32/libstd_v2.a
-r-xr-xr-x 1 bin bin 6487848 Aug 13 2008 ./adm/sw/save/PHSS_40538/C-MIN/usr/ccs/bin/elfdump
-rw-r--r-- 1 root root 6456064 Oct 21 16:55 ./opt/perf/datafiles/logglob
-rw------- 1 sfmdb users 6365184 Oct 21 01:05 ./opt/sfmdb/pgsql/base/17142/17261
-r-xr-xr-x 1 bin bin 6277848 Aug 13 2008 ./adm/sw/save/PHSS_40538/CMDS2-AUX/usr/ccs/bin/fastbind
-r--r--r-- 1 bin bin 6209240 Jan 11 2009 ./adm/sw/save/PHKL_40160/CORE2-KRN/usr/conf/mod/kwdb
-r-xr-xr-x 1 bin bin 5977560 Jan 12 2007 ./adm/sw/save/PHSS_37004/DCE-COR-64SLIB/usr/lib/pa20_64/libdcekt.1
-r-xr-xr-x 1 bin bin 5915656 Aug 13 2008 ./adm/sw/save/PHSS_40538/CMDS2-AUX/usr/ccs/bin/chatr
-r-xr-xr-x 1 bin bin 5402624 Jan 12 2007 ./adm/sw/save/PHSS_37004/DCE-CORE-SHLIB/usr/lib/libdcekt.1
-r--r--r-- 1 bin bin 5344504 Jan 11 2009 ./adm/sw/save/PHKL_40207/CORE2-KRN/usr/conf/mod/pm_sched
-r-xr-xr-x 1 bin bin 5316980 Jul 6 2008 ./adm/sw/save/PHCO_38658/PROG2-AUX/usr/lib/hpux32/libp/libc.so.1

The directory

SAPL918_RX6600:/var/opt/perf/datafiles> du -sk *
8 RUN
8 agdb
0 agdb.lk
8 classinfo.db
5152 logappl
17976 logdev
6320 logglob
16 logindx
8 logpcmd0
26104 logproc
8872 logtran
1296 mikslp.db

[Eli Daniel]

SAPL918_RX6600:/var/opt/perf/datafiles> ll
total 131632
-rw-r--r-- 1 root bin 472 Sep 6 10:24 .glancerc
-rw-r--r-- 1 root bin 7540 Sep 6 10:24 .gpm
-rw-r--r-- 1 root root 31 Oct 16 23:59 RUN
-rw-r--r-- 1 root root 105 Sep 6 10:40 agdb
-rw-r--r-- 1 root root 0 Sep 6 10:40 agdb.lk
-rw-rw-rw- 1 root root 168 Oct 21 17:11 classinfo.db
-rw-r--r-- 1 root root 5264464 Oct 21 17:10 logappl
-rw-r--r-- 1 root root 18381024 Oct 21 17:10 logdev
-rw-r--r-- 1 root root 6457576 Oct 21 17:10 logglob
-rw-r--r-- 1 root root 15856 Oct 16 23:59 logindx
-rw-r--r-- 1 root root 15 Sep 6 10:40 logpcmd0
-rw-r--r-- 1 root root 26731366 Oct 21 17:11 logproc
-rw-r--r-- 1 root root 9068864 Oct 21 17:10 logtran
-rw-r--r-- 1 root sys 1321635 Sep 6 10:28 mikslp.db

[James R. Ferguson]

Hi:

Whatever you do, do *not* manage (manually remove) anything in '/var/adm/sw'. This is the IPD (Installed Product Database) that drives the SD utilities ('swlist', 'swinstall', 'swremove', etc.). If you capriciously chose objects to remove, you will find yourself with a system that you can no longer patch.

You can generally reclaim considerable space in '/var' by running:

# cleanup -c 1

This will remove rollback images of patches that have been superseded at least once. This is quite safe to do.

Regards!

...JRF...

[Eli Daniel]

SAPL918_RX6600:/var> cleanup -c 1
### Cleanup program started at 10/21/10 17:32:25
Commit patches superseded at least 1 time(s) on 'SAPL918'.
Obtaining superseded patch information...done.
No non-committed patches superseded at least 1 time(s) are present.
All information has been logged to /var/adm/cleanup.log.
### Cleanup program completed at 10/21/10 17:32:25

SAPL918_RX6600:/var> bdf .
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol8 15728640 15728640 0 100% /var
SAPL918_RX6600:/var>

[TTr]

Did you remove any large logfiles when /var was grwoing? If the logfile was open (ie. the process was still writing to it), the filename was removed but the filespace is still there and the process is still writing to it.

So what file did you remove?

Also did you look at the subdirectories that Bill listed in the previous posting?

Another place to look for is /var/tmp. It can have a huge number of small files that may be left behind by running processes. A good rule is to delete anything that is older from the server uptime but not open (not being used by any process).

[Bill Hassell]

Analyze the largest directories:

du -kx /var | sort -rnk | head -20

The largest directories is where you would look for problems. The /var/opt/perf/datafiles will have large files but that is normal. Post the output of the above command.

[ani007]

#cd /var
#find . -xdev -size +100000 -exec ls -lrt {} \;
It will find the large files so that u can choose & del. you can del old files from /var/tmp also.

Regards,
Ani

[singh sanjeev]

If nothing can be done to housekeep the /var.

you can check the space in VG00 and increase the /var size to 2GB.

before doing so you have to create space in /var as it is 100% full it will not allow u to run the fsadm command on it.

[Hakki Aydin Ucar]

/var is full , it will make you cannot able to use most of commands, and need to freeing under /var by deleting old unnecessary files OR move them another directory as others stated.
Once I had a situation like this and nothing seemed logical but some cleanup execs in cron got stuck and finally I have to find the process under /var and killed it solved problem. The command is useful to find excessive size used there :

# lsof /var

[rariasn]

Hi Eli:

/dev/vg01/lvswsave 1536000 942531 556872 63% /var/adm/sw/save

/dev/vg02/lvperflog 2048000 1035749 949049 52% /var/opt/perf/datafiles

Create new fs and move data to new fs.

rgs,