topic Re: root filesystem getting full in Operating System - HP-UX

root filesystem getting full

Bejoy C Alias — Thu, 24 Mar 2005 07:01:25 GMT

Hi ..
the pblm is our production server's / volume is 95 % . Size of / is 1 GB . In bdf it is showing 95 % use ( used is 900 MB ) . we are having seperate /var /usr /tmp /opt /home . It was only 27 % on last month . If i do a "du -xk /" it is showing only 98 MB in / . I coudn't get any large files in / . I couldn't restart the server also as it is production server .
waiting for your suggestions....

Re: root filesystem getting full

Robert-Jan Goossens_1 — Thu, 24 Mar 2005 07:04:22 GMT

Hi Bejoy,

The first thing you could check is if someone made an error using a tar command.

# ll /dev/rmt

check if there is a large file omn

check for other large files in /

# find / -xdev -size +1000 -exec ll {} \;

Best regards,
Robert-Jan

Re: root filesystem getting full

Pete Randall — Thu, 24 Mar 2005 07:07:16 GMT

When you see such a difference between bdf and du as this, it almost always is caused by an open file that has been removed. In other words the file is still being written to. The only way to fix this without re-booting is to find the process holding the file open and kill it. Tools like fuser and lsof can help but this is far from easy and you may end up having to reboot anyway.

Pete

Re: root filesystem getting full

harry d brown jr — Thu, 24 Mar 2005 07:21:50 GMT

do you have "lsof" (if not get it from here: http://hpux.ee.ualberta.ca/hppd/hpux/Sysadmin/lsof-4.74/).

Look in /dev:

[root@vpart1 /etc/bkup_mail]# du -sk /dev/*|sort -rn | head

Nothing should be BIG here!!

Also using lsof look for files that are open but actually have been deleted. A deleted file that is OPEN will keep the space until the process htat has it open closes the file.

especially /etc or /dev

Also try this:

ls -1d /* | grep -v `mount|cut -d" " -f1|grep -v "^/$" | sed -e "s/^/-e /" -e "s_\\$_ _"` | xargs -i du -sk {}

get non mounted filesystems

live free or die
harry d brown jr

Re: root filesystem getting full

Bill Hassell — Thu, 24 Mar 2005 09:26:31 GMT

Something is badly misplaced. Your / mountpoint only needs to be about 200-300 megs in size. The / directory is only for the OS and is relatively static (doesn't change in size), so something has been stored in / that does not belong there. Find out with du (don't look for big files):

du -kx / | sort -rn | head -20

It should look something like this:

89232 /
41080 /etc
38448 /sbin
26680 /etc/vx
21656 /etc/vx/type
10160 /etc/opt

(du -k is in Kbytes) In this case, an large 11i system is using a total of 89megs with /etc and /sbin the biggest directories. If /dev is more than 30-80K (NEVER megabytes) then you have regular files in /dev that must be removed:

find /dev -type f -exec ll {} \;

Usually these are spelling errors like /dev/rmt/om or /dev/null3 or /dev/nul.

Re: root filesystem getting full

Bejoy C Alias — Fri, 25 Mar 2005 00:51:43 GMT

Dear All.
There is no big files in /dev/rmt or in / or in /dev
and i dont have lsof installed . so let me install the same and will get
back to you...

du -xk / |sort -rn shows the following
98057 /
59757 /etc
34331 /sbin
20473 /etc/iscan
15539 /etc/vx
13135 /etc/vx/type
11395 /etc/opt
------truncated----
254 /dev
34 /dev/pts
18 /dev/ptym
18 /dev/pty

Re: root filesystem getting full

Bejoy C Alias — Fri, 25 Mar 2005 04:20:38 GMT

Finally i restarted the server...the same pblm exists......

Re: root filesystem getting full

Kent Ostby — Fri, 25 Mar 2005 04:45:03 GMT

Bejoy --

Two possible scenarios that jump out at me:

1) Large file right in root :

ll -a

(although I assume you'd find this one)

2) Large file created UNDER a mount point. It wont show with du because du is looking at the mounted filesystems.

TO check this, you need to reboot and boot up into single user mode. From the ISL prompt:

hpux -is

This will boot you with ONLY / mounted. First thing to check is /var's mount point:

ls -l /var

Then run your du -xk / and see what you come up with.

Best regards,

Oz

Re: root filesystem getting full

john korterman — Fri, 25 Mar 2005 05:18:51 GMT

Hi,
try this:
# find / -type f -xdev -exec ls -l {} \;| sort -kn5| tail -5

It will probably take a while to execute, but it should show the five biggest files in the / partition, only. Please show us the output.

regards,
John K.

Re: root filesystem getting full

harry d brown jr — Fri, 25 Mar 2005 07:35:59 GMT

ls -1d /* | grep -v `mount|cut -d" " -f1|grep -v "^/$" | sed -e "s/^/-e /" -e "s_\\$_ _"` | xargs -i find {} -type f -size +1000000c -exec ls -l \{\}\;

it will find files > 1000000 bytes

live free or die
harry d brown jr

Re: root filesystem getting full

harry d brown jr — Fri, 25 Mar 2005 09:33:56 GMT

sorry my last post won't work, try this:

ls -1d /* | grep -v `mount|cut -d" " -f1|grep -v "^/$" | sed -e "s/^/-e /" -e "s_\\$_ _"` | xargs -i echo find {} -type f -size +1000000c -exe
c ls -l "\{\}" \\\; | sh

live free or die
harry d brown jr

Re: root filesystem getting full

Bejoy C Alias — Fri, 08 Apr 2005 04:41:06 GMT

I have done all the type searches u all had given ..but able to find only a 7 files which are more than 10 mb ..
any more options...

Re: root filesystem getting full

Bill Hassell — Fri, 08 Apr 2005 07:41:46 GMT

You wrote:

> du -xk / |sort -rn
> 98057 /
> 59757 /etc
> 34331 /sbin
> ...

These quite normal values (see my previous listing for the same command). Now you said your / filesystem is 1000Megs (1Gb)? This would indicate that a program has a hidden temporary file open on the / filesystem. This is a very nasty programming technique, where a file is created, opened, then removed while it is still open. The directory has the entry removed but the inode(s) are still open to this process. Once the process terminates (or the system is rebooted) then / should return to about 10% usage (which is what du is reporting at 98megs).

If you cannot reboot, download a copy of lsof and run it against the / filesystem. It will report the process ID's of all programs currently using space in the / filesystem. Examine each program that has files open on / and especially look for non-HP-UX programs. There will be a lot of normal HP-UX processes using device files in /dev. Since this is such a large space, sort the output from lsof like this:

lsof / | sort -rnk7 | head -20

Re: root filesystem getting full

Bejoy C Alias — Tue, 12 Jul 2005 03:44:15 GMT

The problem was that we unmounted some volumes temporarily for backup purposes . Some files were kept on that mount points after umounting , which was not seeing after remounting the volumes. We rebooted the server in single user mode and checked the sizes of mount points , one of the mount point was holding 800 mb of files.
Thanks to all...