/var is full

[Cheung_2]

I have run frecover and terminate it abnormally previously , now the /var is full, but I can't find any large file in our system, how can I reduce the size of /var except reboot the system? thx.

[sycncs]

Hi there

There are a few things you could do.
Check for core files or large log files residing in /var. You could remove them or move them to another location or compress them.
Also, go to /var, do a du -sk ./ and it will give you the actual space that it is taking up. Example: If it shows 400MB and bdf shows 600MB used, it means you have a lot of lost broken links which is still open and taking up space.
What you could do next is to fuser /var and kill those sessions which is no longer active. If the problem persists, you will have to reboot the system as /var cannot be unmounted unless you are single user mode.

Hope this helps.

[Kelli Ward]

Hi,
If you're running savcore (HP-UX 10.2)or savecrash (HP-UX 11x)
There is a directory in /var where the crash files are saved.
It is /var/adm/crash

cd to it, if there is a bounds file and one or a number of directories called core.0, core.1 ... delete them by executing the command, inside the /var/adm/crash directory rm -r *
This is a very dangerous command, if your not familiar with it. It recusively removes all files and folders from your current directory, so be sure to be in the /var/adm/crash directory, you can check this by using the command pwd (print working directory)

If this is the case you may just see a dramatic improvement in your /var/directory.

Good luck,
Kel

[Cheung_2]

hi kelli,

nth in the directory of "/var/adm/crash" .

[Cheung_2]

hi sycncs,
the bdf result is not equal to su result , i kill all dead process , what can i do now except reboot ? thx.

[Bill Hassell]

There might be any large files in /var at all. Instead, a runaway process may have created thousands of small files in some directory. So don't look for big files, look for big directories:

du -kx /var | sort -rn | more

Look at the first few directories under /var. cd into the directory and tpye: ls to see what's there (a few files? thousands of files?) If there are just a few, use:

ll | sort -rnk5 | head

to see the largest files.

[sycncs]

Hi Cheung

Good. At least we are heading in the correct direction. Bill is correct. We can kill those lost files that we can find, but that is very limited if the number stretches to thousands.
It seemed like the machine is quite critical to you now as you are opting out reboot.
But /var is a system directory, unless you can umount and mount it back, it will not return the free space that it is occupying now.

[Ravi_8]

Hi, chung

# ll |sort +4rn in /var will list all files and directories, remove the unwanted files

[Cheung_2]

I didn't any large file in the system , what can i do? thx.

[Jochen Heuer]

Maybe there are unlinked files which are still open (they don't show up in du output since they are already removed but they still use up disk space since they are in use). Use a tool called lsof to check for this:

$ lsof +aL1 /var

Output might look like this:

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
tail 2790 root 0r VREG 64,0x4 13394720 97 /home (/dev/vg00/lvol4)

In this example the process still having the file open is tail. After killing tail the disk space is recovered.

[Jochen Heuer]

Two useful commands to find big files / directories:

to find the big directories in /var run

$ find /var -type d -exec du -sk {} \; | sort -n

then have a look at the biggest dirs (which are printed last)

If you want to check for biggest files in /var just run

$ find /var -type f -exec du -sk {} \; | sort -n

[Jerome Baron]

Hi,

When you install patches the system save old file in /var/adm/sw/save. You maybe have more files in this directory ? You can meke space by using cleanup.

Regards,
Jerome

[Cheung_2]

Thx all response ,
Bolow are my response,
1. no lsof in my system , i am now searching,
2. the largest file is only a 4M file , so I don't think the real largest file is hidden.
3. If I want to remove the patch , can i just remove the file?
eg.
# rm /var/adm/sw/save/PHSS_20444/DCE-CORE-SHLIB/usr/lib/libdce.2

Thx all.

[Jochen Heuer]

lsof can be found on the porting site:

http://hpux.asknet.de/hppd/hpux/Sysadmin/lsof-4.64/

[Jerome Baron]

It's better to use cleanup.
with this you can choose the level of supersed patch remove.

[Jochen Heuer]

If the directory with the biggest usage is /var/adm/sw/save use the program 'cleanup' to get rid of some save-areas ...

[Michael Tully]

Use the cleanup command as suggested:

# cleanup -i

Do not just remove the file

You can get lsof from here:

http://hpux.connect.org.uk/hppd/hpux/Sysadmin/lsof-4.64/

or for 64 bit servers here:
http://www.wiretapped.net/security/host-security/lsof/binaries/hpux/B.11.11/

How large is /var ? perhaps provide the output of these

# bdf

and

# cd /var
# du -sk *

or how Jochen has asked, perhaps then we could help more.

[Trond Haugen]

You should ALWAYS use cleanup to remove backup copies of patches.
In your initial post you state that you ran frecover. WHAT did you frecover, and to where?
Could you also post the output of 'bdf -i /var' so we can get an idea of the size and amount of files.

Regards,
Trond

[Cheung_2]

#bdf
/dev/vg00/lvol8 1105920 1102367 3553 100% /var
#cd /var
#du -sk *
# du -sk *
0 X11
256505 adm
1 dt
0 lost+found
454 mail
0 news
4688 opt
144 preserve
0 rbootd
2 run
521 sam
74 spool
2 statmon
10559 stm
578 tmp
77 tombstones
0 uucp
39 yp

# bdf -i /var
Filesystem kbytes used avail %used iused ifree %iuse Mounted on
/dev/vg00/lvol8 1105920 1102367 3553 100% 18002 1966 90% /var

[Cheung_2]

Hi Trond Haugen,

The file is recovered to /home/ACC/temp , i found a file there , it is "core", i have already removed it , but the /var is still not reduced.

[Jochen Heuer]

This output looks to me as if less then 300 MB is used in /var. So you have either unlinked files which are still open or you have a directory in /var which starts with a dot.

So please check with lsof for unlinked files

lsof +aL1 /var

and run du like this

du -sk `ls -A`

(ls -A shows .file and .dir also).

[Jerome Baron]

can you show if you have message in syslog ?

Regards,
Jerome

[Trond Haugen]

Semms the frecover is unrelated to the lost diskpace. But verry likely Jochen is persuing the real problem (give him points).

Regards,
Trond

[Cheung_2]

Jan 8 17:42:25 GPAPHK vmunix: vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file
system full (1 block extent)^M
Jan 8 17:48:53 EDP vmunix: ^M
Jan 8 17:48:54 EDP above message repeats 97 times
Jan 8 17:48:53 EDP vmunix: vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file
system full (1 block extent)^M
Jan 8 17:48:54 EDP above message repeats 97 times
Jan 8 17:48:57 EDP vmunix: ^M
Jan 8 17:48:57 EDP vmunix: vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file
system full (1 block extent)^M
Jan 8 18:08:53 EDP vmunix: ^M
Jan 8 18:08:54 EDP above message repeats 299 times
Jan 8 18:08:53 EDP vmunix: vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file
system full (1 block extent)^M
Jan 8 18:08:54 EDP above message repeats 299 times
Jan 8 18:08:57 EDP vmunix: ^M
Jan 8 18:08:57 EDP vmunix: vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file
system full (1 block extent)^M

[Jochen Heuer]

Btw. after installing the lsof depot the binary is in

/opt/lsof/bin/lsof