filesystem / full

[tom quach_1]

Dear all,

There are many times that i have disk full, but this time, i could not figure it out. so please help!
Thanks in advance,
Tom


HPUX 11.31 on rx3600

prod:[root]:[TWO_TASK=ora]:/
#du -xk |sort -rnk1|head -20 <--compare with my good identical server this number is lower
408248 .
215360 ./sbin
191480 ./etc
121600 ./etc/vx
103752 ./etc/vx/type
93672 ./sbin/fs
53712 ./sbin/fs/vxfs5.0
45792 ./etc/vx/type/static
42704 ./etc/opt
38584 ./etc/opt/resmon
29584 ./sbin/fs/vxfs
25896 ./etc/vx/type/gen
23992 ./etc/opt/resmon/lib
20616 ./etc/lp
19648 ./etc/vx/type/raid5
18576 ./etc/lp/interface
14984 ./etc/vx/static.d
14520 ./etc/lp/interface/model.orig
14168 ./etc/vx/static.d/build
12408 ./etc/vx/type/fsgen


#bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1048576 1038208 10368 99% /
/dev/vg00/lvol1 1835008 330720 1492600 18% /stand
/dev/vg00/lvol8 20480000 12099496 8316816 59% /var
/dev/vg00/lvol10 1032192 143279 833394 15% /util
/dev/vg00/lvol7 4571136 3090312 1469288 68% /usr
/dev/vg00/lvol12 3080192 456612 2460100 16% /users
/dev/vg02/lvol1 10240000 4908215 4999181 50% /u10
/dev/vg00/lvol4 5120000 642784 4447448 13% /tmp
/dev/vg00/lvol6 10584064 7637792 2923272 72% /opt
/dev/vx/dsk/datadg/vol01
157133824 109429806 14% /home
/dev/vx/dsk/datadg/vol03
10333184 2268897 7561495 23% /tmp/IM

also attached the return of this command

prod:[root]:[TWO_TASK=ora]:/
#find . -xdev -print -exec ls -ld {} \;|sort -rnk5 >/tmp/rootlist

[Steven E. Protter]

Shalom,

215360 ./sbin

Way to much stuff in there.

SEP

[Kenan Erdey]

Hi,

check with lsof +L1 under root for unlinked files. These files can fill filesystems.

[James R. Ferguson]

Hi Tom:

It is possible that you have a process has opened a temporary file(s) and then immediately removed it. This is a very common technique that offers security and programming ease. As long as the process runs, it can use the file (and its allocated disk blocks).

The easiest way to identify a process doing this is to use 'lsof' (available from the Porting Center):

http://hpux.connect.org.uk/

You can use 'lsof' to find files like this by doing:

# lsof +D / +L1

Look for files with a 'NLINK' count of zero (0). These represent open files that will vanish as soon as the last process using them terminates. Once the last process using the file exits, the (now) temporarily allocated disk blocks will be returned to the system.

Regards!

...JRF...

[tom quach_1]

Thanks Steven,

i think because we have VXVM cluster file system.
the total space of the root +400Mb
but i have 1000Mb at /
where are these spaces?
Thanks for your reply.


The good server

#du -sk /dev
168 /dev

#du -sk /etc
253512 /etc

#du -sk /sbin
215352 /sbin
#bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1048576 493168 551112 47% /

The bad server
#du -sk /dev
168 /dev
#du -sk /etc
191480 /etc
#du -sk /sbin
215360 /sbin
#bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1048576 1038208 10368 99% /

[James R. Ferguson]

Hi (again) Tom:

Another tip-off that open, but unlinked files are present (and therefore hidden to 'ls') is a large descrepency between what 'bdf' and 'du' report.

Compare:

# bdf /

# du -xks /

On one of my servers 'bdf' reports a used value of 134,816 versus 129,528 --- not a significant difference.

Regards!

...JRF...

[tom quach_1]

Thank you all for your replies.

James,
Mine is a big different.
were can i find the hidden files here?
i did use this command
#lsof +D / +L1
and terminate those processes but the number is still the same.

Thanks again for your helps.
Tom

#bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1048576 1038208 10368 99% /

#du -xks /
408248 /

[Kenan Erdey]

Hi again,

lastly if you can't shrink and if you can reboot, there is nothing like fresh rebooted server :)

Kenan.

[tom quach_1]

Thanks Kenan,
Unfortunately, i can not reboot this server, we run 24/7 and very small window for a reboot. I have to move files that is not important out of the "/" and schedule a reboot some other time.

questions:
the return of this command goes to lvol3 (it's a "/") . can i see the file that it writes to lvol3 ?

#lsof +D / +L1
PH8SRVN 6489 camillil 1u REG 64,0x3 620039132 0 2228 / (/dev
/vg00/lvol3)

Thanks for your helps.
Tom

[Kenan Erdey]

this is the process that fills root. but you can't see what file.. if you'd see, you can see it when you do ls.

if it's possible stop the program/daemon or kill the process with id 6489. this may relase the space back.

[tom quach_1]

Thanks Kenan,

i did restart the program and saw it write to "lvol3" need to talk to application support on Tuesday but now i need to move files temporarily to different location to save space.
Would you advice in /sbin and /etc which DIR, i can move safely.
Thanks,
Tom

[Dennis Handly]

>I did restart the program and saw it write to "lvol3"

Did it free up any space? You may have to judge whether this application is more important than rebooting your system. Also, can you use TMPDIR to redirect the temp file elsewhere?

>Would you advice in /sbin and /etc which DIR, I can move safely.

None, is the safest answer.
Though there may be some directory in /etc/vx/ that is only used for backup and one for rebuilding the kernel.

[tom quach_1]

Thank you all for your helps.
i did schedule a reboot and change the programs to write error to a logfiles.

Thanks again for all your helps.
Regards,
Tom