topic Re: /var full in Operating System - HP-UX

/var full

Fabrizio Sinopoli — Fri, 16 Jun 2006 05:28:11 GMT

Good mornig
I have a problem. If I run the command "bdf /var", it outputs:

Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol8 5111808 5052688 58712 99% /var

Bur du -sk is: 1794504

I rebooted a lot of time, but it doesn't solve the problem.

Can someone help me?
Fabrizio

Re: /var full

Arunvijai_4 — Fri, 16 Jun 2006 05:43:11 GMT

Hi,

Find out what is occupying more.

# du -skx /var/* | sort -nr | head -10

-Arun

Re: /var full

melvyn burnard — Fri, 16 Jun 2006 05:47:16 GMT

check for unwanted dumps under /var/adm/crash.
Also, if you have installed patches and no longer wish to keep copies for the ability to remove them, you can check how much space is in use under /var/adm/sw/save, and if ti is large then use swmodify to commit these patches or just remove the subdirectories under this directory

Re: /var full

john korterman — Fri, 16 Jun 2006 05:54:56 GMT

Hi,

try
sam-->Routine Tasks-->System log files
and reduce the size of some of the logfiles.

regards,
John K.

Re: /var full

rariasn — Fri, 16 Jun 2006 06:00:08 GMT

Hi Fabrizio,

# find /var -xdev -type f -exec ll {} \;| awk '{ printf "%9d %3s %2d %5s %s\n" , $5, $6 , $7, $8, $9 }'|sort -r -n > /tmp/var.txt &

# more /tmp/var.txt

Others tasks,

Modifify /sbin/clean_tmps and /etc/rc.config.d/clean_tmps, for delete files in directory /var/tmp afther reboot.

Verify file /sbin/clean_adm and /etc/rc.config.d/clean for others clean files.

rgs,

ran

Re: /var full

Ninad_1 — Fri, 16 Jun 2006 06:11:43 GMT

Hi,

Usually the difference in sizes shown by bdf/df and du (Usually du shows lesser than df in such cases) is because some process opens a file and then terminates before closing the file leaving the space unreleased.
But once you reboor these should more or less tally.
Next thing is how you check /var using du -
it should be du -skx /var (Because usually there are other filesystems mounted in /var like /var/tmp , /var/adm ...), if you just use du -sk it will show disk usage by all files/directories under /var DIRECTORY and will not show the disk usage of files/directories in /var FILESYSTEM.

Now solving the problem as already suggested above, check for crash files, core files
find /var -xdev -name core
Also
du -sk /var/* | sort -nr | more
to check highest disk using directories - but then you need to check that they are not seperate filesystems and are infact files/directories in /var filesystem .
And cleanup unwanted stuff.

Regards,
Ninad

Re: /var full

Ninad_1 — Fri, 16 Jun 2006 06:17:47 GMT

Hi,

Heres a script which will show highest disk using directories in /var FILESYSTEM (as /var DIRECTORY may contain various filesystems which you need not see as /var FILESYSTEM is full )

du -sk /var/* | sort -nr| while read dirsize dirname
do
if [[ $(bdf $dirname | grep -c '/var$') != 0 ]]
then
echo $dirname $dirsize
fi
done

Regards,
Ninad

Re: /var full

spex — Fri, 16 Jun 2006 06:25:18 GMT

Fabrizio,

You might also find it helpful to look at files which have been recently created/modified:

# find /var -xdev -type f -mtime -3 -exec ll {} \; |sort -k5

Also have a good long look at the contents of /var/tmp.

PCS

Re: /var full

Doug Burton — Fri, 16 Jun 2006 06:27:48 GMT

Some places to check that haven't been mentioned...

/var/adm/syslog
/var/adm/lp
/var/mail
/var/adm/wtmp (to big?)
/var/adm/lp/log
/var/spool/lp/request
/var/spool/cron/tmp
/var/sam/log
/var/spool/mqueue
lost+found (found in any mounted file system)
Did you mount a filesystem *ON* another filesystem in /var (thus hiding what's underneath)?
Remove core files in /var?

Re: /var full

Fabrizio Sinopoli — Fri, 16 Jun 2006 06:44:26 GMT

No core files, no file with a great size...
The system (HP-UX 11.23 Itanium) is new, only few weeks of life...
No software installed, only OS.

Re: /var full

rariasn — Fri, 16 Jun 2006 06:58:16 GMT

More,

get space in /var/adm/sw/save,

# cleanup -p -c 2 (previous)

# cleanup -c 2 (commit)

rgs,

ran

Re: /var full

Torsten. — Fri, 16 Jun 2006 07:03:44 GMT

If you have available space in your vg00, consider to increase lvol8. Really no large files found?

What gives

cleanup -p -c 2

(this is preview mode only)

Re: /var full

Fabrizio Sinopoli — Fri, 16 Jun 2006 07:03:47 GMT

I have 4 identical server. The used space under /var is the same, but on only one server the bdf command says /var at 99%.
The used space in /var/adm/sw/save is the same for all servers. So this is not the problem...

Re: /var full

spex — Fri, 16 Jun 2006 07:05:41 GMT

Fabrizio,

According to your 'bdf' output, you have 58,712 kbytes free. What happens if you create a 60,000 kbyte file? Does the /var filesystem fill up?

prealloc /var/tmp/foo 60000000

PCS

Re: /var full

Fabrizio Sinopoli — Fri, 16 Jun 2006 07:37:49 GMT

File system full...

No space left on device.

Re: /var full

Ninad_1 — Fri, 16 Jun 2006 07:56:40 GMT

Fabrizio,

Did you try running the script I gave you to show max space occupying directories ?
Any clue from that on where is the space going ?
I also suggest you to run lsof to see list of open files to check which files in /var are open to get a clue if there is a big file not being closed by a dead process.
You can install lsof from
http://hpux.connect.org.uk/hppd/hpux/Sysadmin/lsof-4.77/

Regards,
Ninad

Re: /var full

spex — Fri, 16 Jun 2006 08:10:35 GMT

Fabrizio,

Please post output of:

du -skx /var/* | sort -nr | head -10

find /var -type f -xdev -size +5000000c -exec ll {} \;

PCS

Re: /var full

Fabrizio Sinopoli — Fri, 16 Jun 2006 08:20:38 GMT

First output:
du -skx /var/* | sort -nr | head -10
1670904 /var/adm
82480 /var/opt
22768 /var/jail
17128 /var/stm
616 /var/vx
264 /var/sam
128 /var/tmp
120 /var/spool
104 /var/tombstones
96 /var/yp

Second output in attachment.

Re: /var full

James R. Ferguson — Fri, 16 Jun 2006 08:34:25 GMT

Hi Fabrizio:

This would seem to fit either the case of (1) a process that has created a file and unlinked (removed); and the process is still running [a common technique]; or (2) an over-mounted filesystem.

Using 'lsof' or even a combination of 'fuser' (to find processes using '/var') and 'glance' to view open files by the processes using '/var', might prove fruitful for case-1.

For case-2, run 'ncheck' and look for paths that should not appear beneath '/var':

# ncheck -F vxfs /dev/vg00/lvol8

Regards!

...JRF...