topic Re: /usr is 100% in HP-UX, even after cleanup in Operating System - HP-UX

/usr is 100% in hpux even after cleanup

prashantha — Tue, 21 Feb 2012 11:30:49 GMT

/usr filesystem is 100% in one of our HPUX11.0 server , but after cleaning(moving) some license log file which caused this space issue , bdf still shows it is 100%

i tired to sync the fs but still shows 100%

Any idea without rebooting this server can we solve this issue ? Currently it has only 1 GB data where as filesystem is 4GB

root@host1:/usr# bdf .
Filesystem          kbytes    used   avail %used Mounted on
/dev/vg00/lvol7    4096000 4096000       0 100% /usr
root@host1:/usr# du -kx|sort -rn| more
1094991 .
415569 ./local
373096 ./local/etc/os
373096 ./local/etc
370656 ./lib
147014 ./lib/X11
125682 ./lib/X11/fonts
112234 ./lib/X11/fonts/TrueType
59166   ./lib/pa20_64

Br,

Re: /usr is 100% in HP-UX, even after cleanup

Dennis Handly — Tue, 21 Feb 2012 11:47:48 GMT

>Currently it has only 1 GB data where as filesystem is 4GB

You'll need to install lsof to search for the still opened file that was removed.

Re: /usr is 100% in HP-UX, even after cleanup

prashantha — Tue, 21 Feb 2012 12:53:25 GMT

since lsof was not installed in this box , i tried to install this sw it but it shows error for installing since /usr is already full :(

i think i have left with only reboot this box to make that /usr show correct space .

or do you think we have any other option left out ?

        * Reading source for file information.                              │
│ │        * Executing preDSA command.                                        ││
│ │ ERROR:   The used disk space on filesystem "/usr" is estimated to         ││
│ │          increase by 370 Kbytes.

Re: /usr is 100% in HP-UX, even after cleanup

Dennis Handly — Tue, 21 Feb 2012 13:05:57 GMT

>or do you think we have any other option left out?

Install it on another machine and copy the executable over to some other filesystem, then run.

(And then once you get the space, install it properly.)

Re: /usr is 100% in hpux even after cleanup

Matti_Kurkela — Tue, 21 Feb 2012 14:53:31 GMT

You said the issue was caused by some license log file? Then restart the process that normally writes to that file.

The likely reason for this behavior is that the process is still holding the log file open, so the disk space allocated to the log file cannot be freed until the process closes the file. This is standard Unix behavior.

With the 'lsof' utility, it is easy to find all the deleted-but-still-open files and see which processes are holding them open: just run "lsof +L1".

If an application is well-designed, it should either allow its logs to be truncated (not removed!) while the application is running, or have some way to tell the application that it should close the old log file and start a new one. Often this is done by sending the application process a "kill -HUP" or some other signal; see the documentation of the application.

Re: /usr is 100% in hpux even after cleanup

prashantha — Wed, 22 Feb 2012 13:50:09 GMT

Some how I managed to clear some 370K space and install also utility , and able to findout the problem , after stopping and restarting our license server process disk space show normal

Thanks Dennis & MK for your guidance .

root@host1:/usr# bdf .
Filesystem          kbytes    used   avail %used Mounted on
/dev/vg00/lvol7    4096000 4092767    3233 100% /usr
root@host1:/usr#
root@host1:/usr# lsof +L1
COMMAND     PID USER   FD   TYPE     DEVICE   SIZE/OFF NLINK    NODE NAME
sh          261 root    0r PIPE 0x4e741e08          0     0 676361
ypxfrd      597 root    2r VREG     64,0x8          0     0    1921 /var (/dev/vg00/lvol8)
sh          799 root    0r PIPE 0x43b03408          0     0 571598
snmpdm      989 root    2w VREG     64,0x6          0     0    2736 /tmp (/dev/vg00/lvol6)
testwelld 16186 root    1w VREG     64,0x7 3065233408     0   18863 /usr (/dev/vg00/lvol7)
testwelld 16186 root    2w VREG     64,0x7 3065233408     0   18863 /usr (/dev/vg00/lvol7)
lmgrd      1397 root    1w VREG     64,0x7 3065233408     0   18863 /usr (/dev/vg00/lvol7)
lmgrd      1397 root    2w VREG     64,0x7 3065233408     0   18863 /usr (/dev/vg00/lvol7)

after stoping and restarting those liscence server process it went Ok.

root@host1:/usr/local/flexlm/log# bdf .
Filesystem          kbytes    used   avail %used Mounted on
/dev/vg00/lvol7    4096000 1099323 2809972   28% /usr
root@host1:/usr/local/flexlm/log#

Re: /usr is 100% in HP-UX, even after cleanup

Dennis Handly — Wed, 22 Feb 2012 17:31:39 GMT

>after stopping and restarting our license server process disk space show normal

Ah, you did say "license log file" and that may have given a clue which process.

>Thanks Dennis & MK for your guidance.

If you are happy with the answers, please click on the kudos star on each post that was helpful.