topic Re: /var is full in Operating System - HP-UX

/var is full

Cheung_2 — Thu, 09 Jan 2003 02:40:51 GMT

I have run frecover and terminate it abnormally previously , now the /var is full, but I can't find any large file in our system, how can I reduce the size of /var except reboot the system? thx.

Re: /var is full

sycncs — Thu, 09 Jan 2003 02:44:48 GMT

Hi there

There are a few things you could do.
Check for core files or large log files residing in /var. You could remove them or move them to another location or compress them.
Also, go to /var, do a du -sk ./ and it will give you the actual space that it is taking up. Example: If it shows 400MB and bdf shows 600MB used, it means you have a lot of lost broken links which is still open and taking up space.
What you could do next is to fuser /var and kill those sessions which is no longer active. If the problem persists, you will have to reboot the system as /var cannot be unmounted unless you are single user mode.

Hope this helps.

Re: /var is full

Kelli Ward — Thu, 09 Jan 2003 03:29:35 GMT

Hi,
If you're running savcore (HP-UX 10.2)or savecrash (HP-UX 11x)
There is a directory in /var where the crash files are saved.
It is /var/adm/crash

cd to it, if there is a bounds file and one or a number of directories called core.0, core.1 ... delete them by executing the command, inside the /var/adm/crash directory rm -r *
This is a very dangerous command, if your not familiar with it. It recusively removes all files and folders from your current directory, so be sure to be in the /var/adm/crash directory, you can check this by using the command pwd (print working directory)

If this is the case you may just see a dramatic improvement in your /var/directory.

Good luck,
Kel

Re: /var is full

Cheung_2 — Thu, 09 Jan 2003 03:33:37 GMT

hi kelli,

nth in the directory of "/var/adm/crash" .

Re: /var is full

Cheung_2 — Thu, 09 Jan 2003 03:36:34 GMT

hi sycncs,
the bdf result is not equal to su result , i kill all dead process , what can i do now except reboot ? thx.

Re: /var is full

Bill Hassell — Thu, 09 Jan 2003 03:43:29 GMT

There might be any large files in /var at all. Instead, a runaway process may have created thousands of small files in some directory. So don't look for big files, look for big directories:

du -kx /var | sort -rn | more

Look at the first few directories under /var. cd into the directory and tpye: ls to see what's there (a few files? thousands of files?) If there are just a few, use:

ll | sort -rnk5 | head

to see the largest files.

Re: /var is full

sycncs — Thu, 09 Jan 2003 03:58:45 GMT

Hi Cheung

Good. At least we are heading in the correct direction. Bill is correct. We can kill those lost files that we can find, but that is very limited if the number stretches to thousands.
It seemed like the machine is quite critical to you now as you are opting out reboot.
But /var is a system directory, unless you can umount and mount it back, it will not return the free space that it is occupying now.

Re: /var is full

Ravi_8 — Thu, 09 Jan 2003 05:07:57 GMT

Hi, chung

# ll |sort +4rn in /var will list all files and directories, remove the unwanted files

Re: /var is full

Cheung_2 — Thu, 09 Jan 2003 07:31:46 GMT

I didn't any large file in the system , what can i do? thx.

Re: /var is full

Jochen Heuer — Thu, 09 Jan 2003 08:01:36 GMT

Maybe there are unlinked files which are still open (they don't show up in du output since they are already removed but they still use up disk space since they are in use). Use a tool called lsof to check for this:

$ lsof +aL1 /var

Output might look like this:

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
tail 2790 root 0r VREG 64,0x4 13394720 97 /home (/dev/vg00/lvol4)

In this example the process still having the file open is tail. After killing tail the disk space is recovered.

Re: /var is full

Jochen Heuer — Thu, 09 Jan 2003 08:04:10 GMT

Two useful commands to find big files / directories:

to find the big directories in /var run

$ find /var -type d -exec du -sk {} \; | sort -n

then have a look at the biggest dirs (which are printed last)

If you want to check for biggest files in /var just run

$ find /var -type f -exec du -sk {} \; | sort -n

Re: /var is full

Jerome Baron — Thu, 09 Jan 2003 08:07:11 GMT

Hi,

When you install patches the system save old file in /var/adm/sw/save. You maybe have more files in this directory ? You can meke space by using cleanup.

Regards,
Jerome

Re: /var is full

Cheung_2 — Thu, 09 Jan 2003 08:40:39 GMT

Thx all response ,
Bolow are my response,
1. no lsof in my system , i am now searching,
2. the largest file is only a 4M file , so I don't think the real largest file is hidden.
3. If I want to remove the patch , can i just remove the file?
eg.
# rm /var/adm/sw/save/PHSS_20444/DCE-CORE-SHLIB/usr/lib/libdce.2

Thx all.

Re: /var is full

Jochen Heuer — Thu, 09 Jan 2003 08:44:54 GMT

lsof can be found on the porting site:

http://hpux.asknet.de/hppd/hpux/Sysadmin/lsof-4.64/

Re: /var is full

Jerome Baron — Thu, 09 Jan 2003 08:47:26 GMT

It's better to use cleanup.
with this you can choose the level of supersed patch remove.

Re: /var is full

Jochen Heuer — Thu, 09 Jan 2003 08:47:38 GMT

If the directory with the biggest usage is /var/adm/sw/save use the program 'cleanup' to get rid of some save-areas ...

Re: /var is full

Michael Tully — Thu, 09 Jan 2003 08:50:23 GMT

Use the cleanup command as suggested:

# cleanup -i

Do not just remove the file

You can get lsof from here:

http://hpux.connect.org.uk/hppd/hpux/Sysadmin/lsof-4.64/

or for 64 bit servers here:
http://www.wiretapped.net/security/host-security/lsof/binaries/hpux/B.11.11/

How large is /var ? perhaps provide the output of these

# bdf

and

# cd /var
# du -sk *

or how Jochen has asked, perhaps then we could help more.

Re: /var is full

Trond Haugen — Thu, 09 Jan 2003 08:53:25 GMT

You should ALWAYS use cleanup to remove backup copies of patches.
In your initial post you state that you ran frecover. WHAT did you frecover, and to where?
Could you also post the output of 'bdf -i /var' so we can get an idea of the size and amount of files.

Regards,
Trond

Re: /var is full

Cheung_2 — Thu, 09 Jan 2003 09:56:41 GMT

#bdf
/dev/vg00/lvol8 1105920 1102367 3553 100% /var
#cd /var
#du -sk *
# du -sk *
0 X11
256505 adm
1 dt
0 lost+found
454 mail
0 news
4688 opt
144 preserve
0 rbootd
2 run
521 sam
74 spool
2 statmon
10559 stm
578 tmp
77 tombstones
0 uucp
39 yp

# bdf -i /var
Filesystem kbytes used avail %used iused ifree %iuse Mounted on
/dev/vg00/lvol8 1105920 1102367 3553 100% 18002 1966 90% /var

Re: /var is full

Cheung_2 — Thu, 09 Jan 2003 10:03:39 GMT

Hi Trond Haugen,

The file is recovered to /home/ACC/temp , i found a file there , it is "core", i have already removed it , but the /var is still not reduced.