topic Re: /var is full in Operating System - HP-UX

/var is full

Eli Daniel — Thu, 21 Oct 2010 20:36:52 GMT

Hi, i have the problem
the system /var is full
anex the file large
SAPL918_RX6600:/tmp> cat var_files.txt | sort +4 -5nr | head -n 30
-rw-r--r-- 1 root root 26670362 Oct 21 16:57 ./opt/perf/datafiles/logproc
-r--r--r-- 1 bin bin 22364160 Dec 4 2008 ./opt/wbem/repository.tar
-r--r--r-- 1 bin bin 20974032 Jul 9 2007 ./adm/sw/save/PHKL_40640/VXFS-BASE-KRN/usr/conf/mod/vxfs
-rw-r--r-- 1 root root 18361996 Oct 21 16:55 ./opt/perf/datafiles/logdev
-rw------- 1 sfmdb users 16777216 Oct 6 01:05 ./opt/sfmdb/pgsql/pg_xlog/0000000000000008
-rw------- 1 sfmdb users 16777216 Oct 21 01:05 ./opt/sfmdb/pgsql/pg_xlog/0000000000000006
-rw------- 1 sfmdb users 16777216 Oct 21 01:07 ./opt/sfmdb/pgsql/pg_xlog/0000000000000007
-r-xr-xr-x 1 bin bin 16009352 Jan 12 2007 ./adm/sw/save/PHSS_38487/KWDB-RUN/usr/contrib/kwdb/lib/kwdb.ia64.so
-r-xr-xr-x 1 bin bin 15830852 Jan 12 2007 ./adm/sw/save/PHSS_38487/KWDB-RUN/usr/contrib/kwdb/lib/kwdb32.so
-r-xr-xr-x 1 bin bin 15732840 Jan 12 2007 ./adm/sw/save/PHSS_38487/KWDB-RUN/usr/contrib/kwdb/bin/kwdb.ia64
-r-xr-xr-x 1 bin bin 15577252 Jan 12 2007 ./adm/sw/save/PHSS_38487/KWDB-RUN/usr/contrib/kwdb/lib/kwdb64.so
-r-xr-xr-x 1 bin bin 15517184 Jan 12 2007 ./adm/sw/save/PHSS_38487/KWDB-RUN/usr/contrib/kwdb/bin/kwdb32
-r-xr-xr-x 1 bin bin 15285828 Jan 12 2007 ./adm/sw/save/PHSS_38487/KWDB-RUN/usr/contrib/kwdb/bin/kwdb64
-r-xr-xr-x 1 bin bin 14102952 Aug 13 2008 ./adm/sw/save/PHSS_40538/C-KRN/usr/ccs/bin/ld
-r-xr-xr-x 1 bin bin 13179112 Jan 12 2007 ./adm/sw/save/PHSS_37004/DCE-IA64-SHLIB/usr/lib/hpux64/libdcekt.so.1
-r-xr-xr-x 1 bin bin 13064296 Jan 12 2007 ./adm/sw/save/PHSS_37004/DCE-IA64-SHLIB/usr/lib/hpux32/libdcekt.so.1
-r-xr--r-- 1 root sys 11151872 Jul 13 2009 ./adm/sw/products/SFM-CORE/EMT_COREIA/cerdump-min
-r--r--r-- 1 bin bin 9928518 Feb 15 2007 ./adm/sw/save/PHSS_37501/LANG-64ALIB/usr/lib/hpux64/libstd_v2.a
-rw-r--r-- 1 root root 9065480 Oct 21 16:55 ./opt/perf/datafiles/logtran
-r--r--r-- 1 bin bin 8580110 Feb 15 2007 ./adm/sw/save/PHSS_37501/LANG-MIN/usr/lib/hpux32/libstd_v2.a
-r-xr-xr-x 1 bin bin 6487848 Aug 13 2008 ./adm/sw/save/PHSS_40538/C-MIN/usr/ccs/bin/elfdump
-rw-r--r-- 1 root root 6456064 Oct 21 16:55 ./opt/perf/datafiles/logglob
-rw------- 1 sfmdb users 6365184 Oct 21 01:05 ./opt/sfmdb/pgsql/base/17142/17261
-r-xr-xr-x 1 bin bin 6277848 Aug 13 2008 ./adm/sw/save/PHSS_40538/CMDS2-AUX/usr/ccs/bin/fastbind
-r--r--r-- 1 bin bin 6209240 Jan 11 2009 ./adm/sw/save/PHKL_40160/CORE2-KRN/usr/conf/mod/kwdb
-r-xr-xr-x 1 bin bin 5977560 Jan 12 2007 ./adm/sw/save/PHSS_37004/DCE-COR-64SLIB/usr/lib/pa20_64/libdcekt.1
-r-xr-xr-x 1 bin bin 5915656 Aug 13 2008 ./adm/sw/save/PHSS_40538/CMDS2-AUX/usr/ccs/bin/chatr
-r-xr-xr-x 1 bin bin 5402624 Jan 12 2007 ./adm/sw/save/PHSS_37004/DCE-CORE-SHLIB/usr/lib/libdcekt.1
-r--r--r-- 1 bin bin 5344504 Jan 11 2009 ./adm/sw/save/PHKL_40207/CORE2-KRN/usr/conf/mod/pm_sched
-r-xr-xr-x 1 bin bin 5316980 Jul 6 2008 ./adm/sw/save/PHCO_38658/PROG2-AUX/usr/lib/hpux32/libp/libc.so.1

The directory

SAPL918_RX6600:/var/opt/perf/datafiles> du -sk *
8 RUN
8 agdb
0 agdb.lk
8 classinfo.db
5152 logappl
17976 logdev
6320 logglob
16 logindx
8 logpcmd0
26104 logproc
8872 logtran
1296 mikslp.db

Re: /var is full

Eli Daniel — Thu, 21 Oct 2010 20:43:21 GMT

SAPL918_RX6600:/var/opt/perf/datafiles> ll
total 131632
-rw-r--r-- 1 root bin 472 Sep 6 10:24 .glancerc
-rw-r--r-- 1 root bin 7540 Sep 6 10:24 .gpm
-rw-r--r-- 1 root root 31 Oct 16 23:59 RUN
-rw-r--r-- 1 root root 105 Sep 6 10:40 agdb
-rw-r--r-- 1 root root 0 Sep 6 10:40 agdb.lk
-rw-rw-rw- 1 root root 168 Oct 21 17:11 classinfo.db
-rw-r--r-- 1 root root 5264464 Oct 21 17:10 logappl
-rw-r--r-- 1 root root 18381024 Oct 21 17:10 logdev
-rw-r--r-- 1 root root 6457576 Oct 21 17:10 logglob
-rw-r--r-- 1 root root 15856 Oct 16 23:59 logindx
-rw-r--r-- 1 root root 15 Sep 6 10:40 logpcmd0
-rw-r--r-- 1 root root 26731366 Oct 21 17:11 logproc
-rw-r--r-- 1 root root 9068864 Oct 21 17:10 logtran
-rw-r--r-- 1 root sys 1321635 Sep 6 10:28 mikslp.db

Re: /var is full

James R. Ferguson — Thu, 21 Oct 2010 20:50:41 GMT

Hi:

Whatever you do, do *not* manage (manually remove) anything in '/var/adm/sw'. This is the IPD (Installed Product Database) that drives the SD utilities ('swlist', 'swinstall', 'swremove', etc.). If you capriciously chose objects to remove, you will find yourself with a system that you can no longer patch.

You can generally reclaim considerable space in '/var' by running:

# cleanup -c 1

This will remove rollback images of patches that have been superseded at least once. This is quite safe to do.

Regards!

...JRF...

Re: /var is full

Eli Daniel — Thu, 21 Oct 2010 21:03:08 GMT

SAPL918_RX6600:/var> cleanup -c 1
### Cleanup program started at 10/21/10 17:32:25
Commit patches superseded at least 1 time(s) on 'SAPL918'.
Obtaining superseded patch information...done.
No non-committed patches superseded at least 1 time(s) are present.
All information has been logged to /var/adm/cleanup.log.
### Cleanup program completed at 10/21/10 17:32:25

SAPL918_RX6600:/var> bdf .
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol8 15728640 15728640 0 100% /var
SAPL918_RX6600:/var>

Re: /var is full

TTr — Thu, 21 Oct 2010 21:53:29 GMT

Did you remove any large logfiles when /var was grwoing? If the logfile was open (ie. the process was still writing to it), the filename was removed but the filespace is still there and the process is still writing to it.

So what file did you remove?

Also did you look at the subdirectories that Bill listed in the previous posting?

Another place to look for is /var/tmp. It can have a huge number of small files that may be left behind by running processes. A good rule is to delete anything that is older from the server uptime but not open (not being used by any process).

Re: /var is full

Bill Hassell — Fri, 22 Oct 2010 03:09:51 GMT

Analyze the largest directories:

du -kx /var | sort -rnk | head -20

The largest directories is where you would look for problems. The /var/opt/perf/datafiles will have large files but that is normal. Post the output of the above command.

Re: /var is full

ani007 — Fri, 22 Oct 2010 05:19:26 GMT

#cd /var
#find . -xdev -size +100000 -exec ls -lrt {} \;
It will find the large files so that u can choose & del. you can del old files from /var/tmp also.

Regards,
Ani

Re: /var is full

singh sanjeev — Fri, 22 Oct 2010 06:08:53 GMT

If nothing can be done to housekeep the /var.

you can check the space in VG00 and increase the /var size to 2GB.

before doing so you have to create space in /var as it is 100% full it will not allow u to run the fsadm command on it.

Re: /var is full

Hakki Aydin Ucar — Fri, 22 Oct 2010 07:08:32 GMT

/var is full , it will make you cannot able to use most of commands, and need to freeing under /var by deleting old unnecessary files OR move them another directory as others stated.
Once I had a situation like this and nothing seemed logical but some cleanup execs in cron got stuck and finally I have to find the process under /var and killed it solved problem. The command is useful to find excessive size used there :

# lsof /var

Re: /var is full

rariasn — Fri, 22 Oct 2010 08:48:24 GMT

Hi Eli:

/dev/vg01/lvswsave 1536000 942531 556872 63% /var/adm/sw/save

/dev/vg02/lvperflog 2048000 1035749 949049 52% /var/opt/perf/datafiles

Create new fs and move data to new fs.

rgs,

Re: /var is full

Eli Daniel — Fri, 22 Oct 2010 12:05:29 GMT

HI,
extended FS / var at 15 gb and scored 100% in 5 minutes

before size FS 8 GB
later size FS 15 GB

5 minutes the FS is FULL

SAPL918_RX6600:/var> du -sk *
0 X11
719016 adm
0 asx
0 empty
352 evm
0 home
26032 jail
0 lost+found
1112 mail
0 news
0 nfs
0 nfs4
263736 opt
0 preserve
240 psb
112 run
488 sam
112 spool
8 statmon
76408 stm
0 symapi
40 tdp_r3
5216 tmp
0 tombstones
0 uucp
112 yp

SAPL918_RX6600:/var> bdf .
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol8 15728640 15728640 0 100% /var

Re: /var is full

Bill Hassell — Fri, 22 Oct 2010 13:40:41 GMT

The directory /var/jail is very suspicious (ie, why is it so large?). And /var/opt is massive. That is likely the problem. There is a runaway program creating/using a lot of space. Since the directory itself is very large, it means there are a very large number of files. Use this command to find them:

du -kx /var/opt | sort -rn | head -20

Re: /var is full

Earl_Crowder — Fri, 22 Oct 2010 13:50:03 GMT

Eli,

As TTr asked, was a logfile or other large file deleted? Perhaps /var/adm/syslog/syslog.log or something in /var/tmp?

If the file was deleted while it was being used, the only way i know of to find it is using lsof.

For example, this command will list files in /var filesystem larger than 1GB that are opened but have been removed (unlinked):
#lsof /var | awk '/REG/&&/lvol/{if ($7>1073741824){print $0}}'

Earl

Re: /var is full

Bill Hassell — Fri, 22 Oct 2010 13:50:25 GMT

I missed /var/stm in your list (that's why sort is so important). A large amount of space in /var/stm (especially when it grows) indicates you have a hardware problem. Look at syslog.log for hardware errors.

Re: /var is full

TTr — Fri, 22 Oct 2010 14:16:03 GMT

/var is 15GB but if you look at the du listing it is nowhere near that. So there is unaccounted space somewhere, most likely is a deleted file.

If there is a hardware error, syslog would do a lot of logging and since /var/adm is only 719MB, the syslog file was deleted.

If there is no hardware error, some other logfile was probably deleted while open.

Run the lsof command as indicated above and show us the output of "ll /var/adm/syslog" and while at it the output of "ll /var/tmp".

Re: /var is full

Eli Daniel — Fri, 22 Oct 2010 18:15:20 GMT

Thanks for information

Re: /var is full

TTr — Sat, 23 Oct 2010 10:57:58 GMT

So what was the cause of all this. Somebody else could have the same problem and can benefit from your resolution.