root filesystem getting full

[Bejoy C Alias]

Hi ..
the pblm is our production server's / volume is 95 % . Size of / is 1 GB . In bdf it is showing 95 % use ( used is 900 MB ) . we are having seperate /var /usr /tmp /opt /home . It was only 27 % on last month . If i do a "du -xk /" it is showing only 98 MB in / . I coudn't get any large files in / . I couldn't restart the server also as it is production server .
waiting for your suggestions....

[Robert-Jan Goossens_1]

Hi Bejoy,

The first thing you could check is if someone made an error using a tar command.

# ll /dev/rmt

check if there is a large file omn

check for other large files in /

# find / -xdev -size +1000 -exec ll {} \;

Best regards,
Robert-Jan

[Pete Randall]

When you see such a difference between bdf and du as this, it almost always is caused by an open file that has been removed. In other words the file is still being written to. The only way to fix this without re-booting is to find the process holding the file open and kill it. Tools like fuser and lsof can help but this is far from easy and you may end up having to reboot anyway.


Pete

[harry d brown jr]

do you have "lsof" (if not get it from here: http://hpux.ee.ualberta.ca/hppd/hpux/Sysadmin/lsof-4.74/).

Look in /dev:

[root@vpart1 /etc/bkup_mail]# du -sk /dev/*|sort -rn | head

Nothing should be BIG here!!

Also using lsof look for files that are open but actually have been deleted. A deleted file that is OPEN will keep the space until the process htat has it open closes the file.

especially /etc or /dev


Also try this:

ls -1d /* | grep -v `mount|cut -d" " -f1|grep -v "^/$" | sed -e "s/^/-e /" -e "s_\\$_ _"` | xargs -i du -sk {}

get non mounted filesystems

live free or die
harry d brown jr

[Bill Hassell]

Something is badly misplaced. Your / mountpoint only needs to be about 200-300 megs in size. The / directory is only for the OS and is relatively static (doesn't change in size), so something has been stored in / that does not belong there. Find out with du (don't look for big files):

du -kx / | sort -rn | head -20

It should look something like this:

89232 /
41080 /etc
38448 /sbin
26680 /etc/vx
21656 /etc/vx/type
10160 /etc/opt

(du -k is in Kbytes) In this case, an large 11i system is using a total of 89megs with /etc and /sbin the biggest directories. If /dev is more than 30-80K (NEVER megabytes) then you have regular files in /dev that must be removed:

find /dev -type f -exec ll {} \;

Usually these are spelling errors like /dev/rmt/om or /dev/null3 or /dev/nul.

[Bejoy C Alias]

Dear All.
There is no big files in /dev/rmt or in / or in /dev
and i dont have lsof installed . so let me install the same and will get
back to you...

du -xk / |sort -rn shows the following
98057 /
59757 /etc
34331 /sbin
20473 /etc/iscan
15539 /etc/vx
13135 /etc/vx/type
11395 /etc/opt
------truncated----
254 /dev
34 /dev/pts
18 /dev/ptym
18 /dev/pty

[Bejoy C Alias]

Finally i restarted the server...the same pblm exists......

[Kent Ostby]

Bejoy --

Two possible scenarios that jump out at me:

1) Large file right in root :

ll -a

(although I assume you'd find this one)

2) Large file created UNDER a mount point. It wont show with du because du is looking at the mounted filesystems.

TO check this, you need to reboot and boot up into single user mode. From the ISL prompt:

hpux -is

This will boot you with ONLY / mounted. First thing to check is /var's mount point:

ls -l /var

Then run your du -xk / and see what you come up with.

Best regards,

Oz

[john korterman]

Hi,
try this:
# find / -type f -xdev -exec ls -l {} \;| sort -kn5| tail -5

It will probably take a while to execute, but it should show the five biggest files in the / partition, only. Please show us the output.

regards,
John K.

[harry d brown jr]

ls -1d /* | grep -v `mount|cut -d" " -f1|grep -v "^/$" | sed -e "s/^/-e /" -e "s_\\$_ _"` | xargs -i find {} -type f -size +1000000c -exec ls -l \{\}\;

it will find files > 1000000 bytes

live free or die
harry d brown jr

[harry d brown jr]

sorry my last post won't work, try this:

ls -1d /* | grep -v `mount|cut -d" " -f1|grep -v "^/$" | sed -e "s/^/-e /" -e "s_\\$_ _"` | xargs -i echo find {} -type f -size +1000000c -exe
c ls -l "\{\}" \\\; | sh

live free or die
harry d brown jr

[Bejoy C Alias]

I have done all the type searches u all had given ..but able to find only a 7 files which are more than 10 mb ..
any more options...

[Bill Hassell]

You wrote:

> du -xk / |sort -rn
> 98057 /
> 59757 /etc
> 34331 /sbin
> ...

These quite normal values (see my previous listing for the same command). Now you said your / filesystem is 1000Megs (1Gb)? This would indicate that a program has a hidden temporary file open on the / filesystem. This is a very nasty programming technique, where a file is created, opened, then removed while it is still open. The directory has the entry removed but the inode(s) are still open to this process. Once the process terminates (or the system is rebooted) then / should return to about 10% usage (which is what du is reporting at 98megs).

If you cannot reboot, download a copy of lsof and run it against the / filesystem. It will report the process ID's of all programs currently using space in the / filesystem. Examine each program that has files open on / and especially look for non-HP-UX programs. There will be a lot of normal HP-UX processes using device files in /dev. Since this is such a large space, sort the output from lsof like this:

lsof / | sort -rnk7 | head -20

[Bejoy C Alias]

The problem was that we unmounted some volumes temporarily for backup purposes . Some files were kept on that mount points after umounting , which was not seeing after remounting the volumes. We rebooted the server in single user mode and checked the sizes of mount points , one of the mount point was holding 800 mb of files.
Thanks to all...