topic Re: filesystem / full in Operating System - HP-UX

filesystem / full

tom quach_1 — Mon, 25 May 2009 18:15:56 GMT

Dear all,

There are many times that i have disk full, but this time, i could not figure it out. so please help!
Thanks in advance,
Tom

HPUX 11.31 on rx3600

prod:[root]:[TWO_TASK=ora]:/
#du -xk |sort -rnk1|head -20 <--compare with my good identical server this number is lower
408248 .
215360 ./sbin
191480 ./etc
121600 ./etc/vx
103752 ./etc/vx/type
93672 ./sbin/fs
53712 ./sbin/fs/vxfs5.0
45792 ./etc/vx/type/static
42704 ./etc/opt
38584 ./etc/opt/resmon
29584 ./sbin/fs/vxfs
25896 ./etc/vx/type/gen
23992 ./etc/opt/resmon/lib
20616 ./etc/lp
19648 ./etc/vx/type/raid5
18576 ./etc/lp/interface
14984 ./etc/vx/static.d
14520 ./etc/lp/interface/model.orig
14168 ./etc/vx/static.d/build
12408 ./etc/vx/type/fsgen

#bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1048576 1038208 10368 99% /
/dev/vg00/lvol1 1835008 330720 1492600 18% /stand
/dev/vg00/lvol8 20480000 12099496 8316816 59% /var
/dev/vg00/lvol10 1032192 143279 833394 15% /util
/dev/vg00/lvol7 4571136 3090312 1469288 68% /usr
/dev/vg00/lvol12 3080192 456612 2460100 16% /users
/dev/vg02/lvol1 10240000 4908215 4999181 50% /u10
/dev/vg00/lvol4 5120000 642784 4447448 13% /tmp
/dev/vg00/lvol6 10584064 7637792 2923272 72% /opt
/dev/vx/dsk/datadg/vol01
157133824 109429806 14% /home
/dev/vx/dsk/datadg/vol03
10333184 2268897 7561495 23% /tmp/IM

also attached the return of this command

prod:[root]:[TWO_TASK=ora]:/
#find . -xdev -print -exec ls -ld {} \;|sort -rnk5 >/tmp/rootlist

Re: filesystem / full

Steven E. Protter — Mon, 25 May 2009 18:22:39 GMT

Shalom,

215360 ./sbin

Way to much stuff in there.

SEP

Re: filesystem / full

Kenan Erdey — Mon, 25 May 2009 18:32:10 GMT

Hi,

check with lsof +L1 under root for unlinked files. These files can fill filesystems.

Re: filesystem / full

James R. Ferguson — Mon, 25 May 2009 18:33:52 GMT

Hi Tom:

It is possible that you have a process has opened a temporary file(s) and then immediately removed it. This is a very common technique that offers security and programming ease. As long as the process runs, it can use the file (and its allocated disk blocks).

The easiest way to identify a process doing this is to use 'lsof' (available from the Porting Center):

http://hpux.connect.org.uk/

You can use 'lsof' to find files like this by doing:

# lsof +D / +L1

Look for files with a 'NLINK' count of zero (0). These represent open files that will vanish as soon as the last process using them terminates. Once the last process using the file exits, the (now) temporarily allocated disk blocks will be returned to the system.

Regards!

...JRF...

Re: filesystem / full

tom quach_1 — Mon, 25 May 2009 18:38:14 GMT

Thanks Steven,

i think because we have VXVM cluster file system.
the total space of the root +400Mb
but i have 1000Mb at /
where are these spaces?
Thanks for your reply.

The good server

#du -sk /dev
168 /dev

#du -sk /etc
253512 /etc

#du -sk /sbin
215352 /sbin
#bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1048576 493168 551112 47% /

The bad server
#du -sk /dev
168 /dev
#du -sk /etc
191480 /etc
#du -sk /sbin
215360 /sbin
#bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1048576 1038208 10368 99% /

Re: filesystem / full

James R. Ferguson — Mon, 25 May 2009 19:02:30 GMT

Hi (again) Tom:

Another tip-off that open, but unlinked files are present (and therefore hidden to 'ls') is a large descrepency between what 'bdf' and 'du' report.

Compare:

# bdf /

# du -xks /

On one of my servers 'bdf' reports a used value of 134,816 versus 129,528 --- not a significant difference.

Regards!

...JRF...

Re: filesystem / full

tom quach_1 — Mon, 25 May 2009 19:10:37 GMT

Thank you all for your replies.

James,
Mine is a big different.
were can i find the hidden files here?
i did use this command
#lsof +D / +L1
and terminate those processes but the number is still the same.

Thanks again for your helps.
Tom

#bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1048576 1038208 10368 99% /

#du -xks /
408248 /

Re: filesystem / full

Kenan Erdey — Mon, 25 May 2009 19:26:01 GMT

Hi again,

lastly if you can't shrink and if you can reboot, there is nothing like fresh rebooted server :)

Kenan.

Re: filesystem / full

tom quach_1 — Mon, 25 May 2009 19:38:27 GMT

Thanks Kenan,
Unfortunately, i can not reboot this server, we run 24/7 and very small window for a reboot. I have to move files that is not important out of the "/" and schedule a reboot some other time.

questions:
the return of this command goes to lvol3 (it's a "/") . can i see the file that it writes to lvol3 ?

#lsof +D / +L1
PH8SRVN 6489 camillil 1u REG 64,0x3 620039132 0 2228 / (/dev
/vg00/lvol3)

Thanks for your helps.
Tom

Re: filesystem / full

Kenan Erdey — Mon, 25 May 2009 20:08:22 GMT

this is the process that fills root. but you can't see what file.. if you'd see, you can see it when you do ls.

if it's possible stop the program/daemon or kill the process with id 6489. this may relase the space back.

Re: filesystem / full

tom quach_1 — Mon, 25 May 2009 20:29:26 GMT

Thanks Kenan,

i did restart the program and saw it write to "lvol3" need to talk to application support on Tuesday but now i need to move files temporarily to different location to save space.
Would you advice in /sbin and /etc which DIR, i can move safely.
Thanks,
Tom

Re: filesystem / full

Dennis Handly — Mon, 25 May 2009 23:34:47 GMT

>I did restart the program and saw it write to "lvol3"

Did it free up any space? You may have to judge whether this application is more important than rebooting your system. Also, can you use TMPDIR to redirect the temp file elsewhere?

>Would you advice in /sbin and /etc which DIR, I can move safely.

None, is the safest answer.
Though there may be some directory in /etc/vx/ that is only used for backup and one for rebuilding the kernel.

Re: filesystem / full

tom quach_1 — Wed, 27 May 2009 21:40:05 GMT

Thank you all for your helps.
i did schedule a reboot and change the programs to write error to a logfiles.

Thanks again for all your helps.
Regards,
Tom