How to release a null filename space in filesystem

[lbseraph]

Today I found /var directory already full, then used command ff to check which file size too big. Then find a file (no name) have more than 4GB. I don't know how it generated(This machine no any app., like Oracle. Just installed HP-UX B.11.23.0712, for testing). I tried to use command find to delete it, but failed. Anyone have good method to release the space? Thank you~

root@:[/]#bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1310720 691928 614024 53% /
/dev/vg00/lvol1 1835008 230256 1592304 13% /stand
/dev/vg00/lvol8 7340032 7340032 0 100% /var
/dev/vg00/lvol7 8617984 2764392 5807896 32% /usr
/dev/vg00/lvol4 1048576 72368 968760 7% /tmp
/dev/vg00/lvol6 10575872 5155472 5378112 49% /opt
/dev/vg00/lvol5 131072 11912 118264 9% /home

root@wudai:[/]#ff -s /dev/vg00/lvol8 | sort -rnk 3 |more
./(null) 10819 4793262080 <===
./adm/sw/save/PHCO_33081/VEA-FILESET/opt/VRTSob/jre_ia.tar 29010 15202560
0
./adm/sw/save/PHCO_33081/VEA-FILESET/opt/VRTSob/jre.tar 28924 106736128
./adm/crash/crash.1/vmunix 4479 68657792
./adm/crash/crash.1/image.2.2 15102 67100672
./adm/crash/crash.1/image.2.4 15106 67084288
./adm/crash/crash.1/image.2.3 15105 67084288
./adm/crash/crash.1/image.2.1 15101 67076096
./adm/crash/crash.1/image.2.5 15111 64851968
./adm/sw/save/PHSS_35055/LANG-64ALIB/usr/lib/hpux64/librwtool_v2.a 20731
54990224
./adm/sw/save/PHSS_35055/LANG-MIN/usr/lib/hpux32/librwtool_v2.a 35271 43529790
./adm/sw/save/PHSS_35055/LANG-64ALIB/usr/lib/hpux64/libstd_v2.a 20718 20476142
./opt/wbem/repository.tar 22315 20254720
./adm/sw/save/PHSS_34860/C-MIN/usr/ccs/bin/fdp 21810 18702416
./adm/sw/save/PHSS_35045/CORE2-SHLIBS/usr/lib/hpux32/aries32.so 34105 18287868
./adm/sw/save/PHSS_35055/LANG-MIN/usr/lib/hpux32/libstd_v2.a 35273 17219438

root@wudai:[/]#find . -inum 10819 -exec rm -i {} \; // not output
root@wudai:[/]#bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1310720 691928 614024 53% /
/dev/vg00/lvol1 1835008 230256 1592304 13% /stand
/dev/vg00/lvol8 7340032 7340032 0 100% /var
/dev/vg00/lvol7 8617984 2764392 5807896 32% /usr
/dev/vg00/lvol4 1048576 72368 968760 7% /tmp
/dev/vg00/lvol6 10575872 5155472 5378112 49% /opt
/dev/vg00/lvol5 131072 11912 118264 9% /home

[Dennis Handly]

Do you see anything with:
ll -ia /var

[Venkatesh BL]

> root@wudai:[/]#ff -s /dev/vg00/lvol8 | sort -rnk 3 |more

you are running the command on the volume directly. So, it must be showing the 'relative' path.

> /dev/vg00/lvol8 7340032 7340032 0 100% /var

As /var is the mount point, I'd expect the 'null' file to be in /var directory. Can you try running your 'find -inum' command inside /var dir?

[lbseraph]

1. I can't find any file's inode number is 10819 in /var directory.

root@wudai:[/]#ll -ia /var
total 176
2 dr-xr-xr-x 23 bin bin 8192 Dec 4 14:11 .
2 drwxr-xr-x 21 root root 8192 Dec 1 11:54 ..
5471 drwxr-xr-x 3 bin bin 96 Aug 3 10:30 X11
8 drwxr-xr-x 17 adm adm 8192 Dec 2 13:52 adm
11236 -rw------- 1 root sys 0 Sep 29 15:55 bigfile1
4637 drwxr-xr-x 4 root sys 8192 Nov 19 09:43 dt
2361 drwxr-xr-x 2 root sys 96 Aug 3 10:43 empty
33 drwxrwxrwt 2 bin bin 96 Aug 3 07:20 home
6879 drwxr-xr-x 4 root sys 96 Aug 3 10:37 jail
3 drwxr-xr-x 2 root root 96 Aug 3 07:19 lost+found
187 drwxrwxr-x 2 bin mail 96 Nov 28 23:34 mail
179 drwxrwxrwt 2 bin bin 96 Aug 3 10:19 news
4 dr-xr-xr-x 35 bin bin 8192 Aug 3 10:40 opt
5424 drwxrwxrwt 2 bin bin 96 Aug 3 10:29 preserve
199 dr-xr-xr-x 2 bin bin 8192 Dec 1 11:44 run
37174 drwxrwxr-x 12 bin bin 8192 Dec 5 18:07 sam
37486 dr-xr-xr-x 13 bin bin 8192 Nov 12 22:39 spool
10238 drwxr-xr-x 4 root root 96 Aug 3 11:00 statmon
36665 drwxr-xr-x 6 root other 96 Aug 3 10:31 stm
7 drwxrwxrwt 12 root root 16384 Dec 5 20:50 tmp
10905 drwxr-xr-x 2 root sys 96 Dec 1 11:56 tombstones
5344 dr-xr-xr-x 6 bin bin 96 Aug 3 10:27 uucp
12372 dr-xr-xr-x 5 bin bin 96 Aug 3 11:02 vx
438 drwxr-xr-x 2 bin bin 8192 Aug 3 10:19 yp

2. After running command 'find -inum' in /var, no any output.

root@wudai:[/] # find /var -inum 10819
root@wudai:[/] # bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1310720 691928 614024 53% /
/dev/vg00/lvol1 1835008 230256 1592304 13% /stand
/dev/vg00/lvol8 7340032 7340032 0 100% /var
/dev/vg00/lvol7 8617984 2764392 5807896 32% /usr
/dev/vg00/lvol4 1048576 72376 968752 7% /tmp
/dev/vg00/lvol6 10575872 5155472 5378112 49% /opt
/dev/vg00/lvol5 131072 11912 118264 9% /home
root@wudai:[/] # find /var -inum 10819 -exec rm -i {} \;
root@wudai:[/] # bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1310720 691928 614024 53% /
/dev/vg00/lvol1 1835008 230256 1592304 13% /stand
/dev/vg00/lvol8 7340032 7340032 0 100% /var
/dev/vg00/lvol7 8617984 2764392 5807896 32% /usr
/dev/vg00/lvol4 1048576 72376 968752 7% /tmp
/dev/vg00/lvol6 10575872 5155472 5378112 49% /opt
/dev/vg00/lvol5 131072 11912 118264 9% /home

[Dennis Handly]

>then used command ff to check which file size too big.

Perhaps you should use lsof to look for unlinked files. I'm not sure if ff(1) could find those?

[James R. Ferguson]

Hi:

You used 'ff -s /dev/vg00/lvol8' which returns the file _SIZE_ after the name. This is why you can't find the inode that you think you should!

I'd use 'lsof' as suggested too by Dennis to find unlinked (but open files holding space):

# lsof +D /var +L1

This will expose not only the unlinked files and their sizes, but also the process ID that must terminate before the file's space is released back.

Regards!

...JRF...

[James R. Ferguson]

Hi (again):

> ME: You used 'ff -s /dev/vg00/lvol8' which returns the file _SIZE_ after the name. This is why you can't find the inode that you think you should!

No, that's wrong. ENOCOFFEE. Disregard that, the first field following the name is the inode.

However, I would look to 'lsof' in the manner I cited.

Regards!

...JRF...

[lbseraph]

>Perhaps you should use lsof to look for unlinked files.
>I'd use 'lsof' as suggested too by Dennis to find unlinked (but open files holding space):
> # lsof +D /var +L1
>This will expose not only the unlinked files and their sizes, but also the process ID that must >terminate before the file's space is released back.

-->Thank you for all reply, I have found the root cause, that caused by samx process.
root@wudai:[/]#lsof +D /var +L1 | grep -i 10819
run.cgis 15597 root 1u REG 64,0x8 4794703872 0 10819 /var (/dev/vg00/lvol8)
sh 15600 root 1u REG 64,0x8 4794703872 0 10819 /var (/dev/vg00/lvol8)
samx 15610 root 1u REG 64,0x8 4794703872 0 10819 /var (/dev/vg00/lvol8)

root@wudai:[/]#ps -ef | grep 15597
root 15600 15597 0 Dec 2 ? 0:00 sh -c /usr/sam/lbin/samx -C -s serverConfiguration /usr/lib/X11/Xserver/misc/sam/C/sam.ui
root 15597 15596 0 Dec 2 ? 0:00 /opt/hpsmh/data/cgi-bin/xlaunch/run.cgis
root@wudai:[/]#ps -ef | grep 15610
root 15610 15600 218 Dec 2 ? 5541:21 /usr/sam/lbin/samx -C -s serverConfiguration /usr/lib/X11/Xserver/misc/sam/C/sam.ui

--> From top output, I found samx process take up too many CPU resource.
root@wudai:[/]#top
System: wudai Sun Dec 6 14:50:09 2009
Load averages: 2.46, 2.48, 2.50
168 processes: 138 sleeping, 29 running, 1 zombie
Cpu states:
LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
2.46 90.8% 0.0% 9.2% 0.0% 0.0% 0.0% 0.0% 0.0%

Memory: 378840K (163120K) real, 748852K (331496K) virtual, 94368K free Page# 1/
12

TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
? 15610 root 232 20 26256K 5212K run 5557:43 95.88 95.71 samx
? 48 root 152 20 12960K 11520K run 33:46 0.65 0.65 vxfsd

-->Then use tusc trace it and found PID 15610(samx) in an error loop?
root@wudai:[/]#tusc 15596 | more
( Attached to process 15596 ("/opt/hpsmh/data/cgi-bin/xlaunch/run.cgis") [64-bit
] )
waitpid(15597, 0x9fffffffffd6b998, 0) .................... [sleeping]
Standard input( Detaching from process 15596 ("/opt/hpsmh/data/cgi-bin/xlaunch/r
un.cgis") )
root@wudai:[/] # tusc 15597 | more
( Attached to process 15597 ("/opt/hpsmh/data/cgi-bin/xlaunch/run.cgis") [64-bit
] )
waitpid(15600, 0x9fffffffffd6b880, 0) .................... [sleeping]
Standard input( Detaching from process 15597 ("/opt/hpsmh/data/cgi-bin/xlaunch/r
un.cgis") )
root@wudai:[/]#tusc 15600 | more
( Attached to process 15600 ("sh -c /usr/sam/lbin/samx -C -s serverConfiguration
/usr/lib/X11") [32-bit] )
waitpid(-1, 0x7fffda60, WUNTRACED) ....................... [sleeping]
Standard input( Detaching from process 15600 ("sh -c /usr/sam/lbin/samx -C -s se
rverConfiguration /usr/lib/X11") )
root@wudai:[/] # tusc 15610 | more
( Attached to process 15610 ("/usr/sam/lbin/samx -C -s serverConfiguration /usr/
lib/X11/Xserv") [32-bit] )
In user-mode ............................................. [running]
sigprocmask(SIG_SETMASK, 0x7fffe0d0, NULL) ............... = 0
sigprocmask(SIG_SETMASK, 0x7fffe0d0, NULL) ............... = 0
read(0, 0x400cc0e8, 8192) ................................ ERR#9 EBADF
sigprocmask(SIG_SETMASK, 0x7fffe0d0, NULL) ............... = 0
sigprocmask(SIG_SETMASK, 0x7fffe0d0, NULL) ............... = 0
sigprocmask(SIG_SETMASK, 0x7fffe0d0, NULL) ............... = 0
sigprocmask(SIG_SETMASK, 0x7fffe0d0, NULL) ............... = 0
read(0, 0x400cc0e8, 8192) ................................ ERR#9 EBADF
sigprocmask(SIG_SETMASK, 0x7fffe0d0, NULL) ............... = 0
sigprocmask(SIG_SETMASK, 0x7fffe0d0, NULL) ............... = 0
sigprocmask(SIG_SETMASK, 0x7fffe0d0, NULL) ............... = 0
sigprocmask(SIG_SETMASK, 0x7fffe0d0, NULL) ............... = 0
read(0, 0x400cc0e8, 8192) ................................ ERR#9 EBADF
sigprocmask(SIG_SETMASK, 0x7fffe0d0, NULL) ............... = 0
sigprocmask(SIG_SETMASK, 0x7fffe0d0, NULL) ............... = 0
sigprocmask(SIG_SETMASK, 0x7fffe0d0, NULL) ............... = 0
sigprocmask(SIG_SETMASK, 0x7fffe0d0, NULL) ............... = 0
read(0, 0x400cc0e8, 8192) ................................ ERR#9 EBADF
sigprocmask(SIG_SETMASK, 0x7fffe0d0, NULL) ............... = 0
sigprocmask(SIG_SETMASK, 0x7fffe0d0, NULL) ............... = 0
sigprocmask(SIG_SETMASK, 0x7fffe0d0, NULL) ............... = 0

-->After I used command kill -9 to kill it(straight kill and kill -15 failed), /var show normal.
root@wudai:[/]#bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1310720 692120 613840 53% /
/dev/vg00/lvol1 1835008 230256 1592304 13% /stand
/dev/vg00/lvol8 7340032 2657864 4647816 36% /var
/dev/vg00/lvol7 8617984 2764392 5807896 32% /usr
/dev/vg00/lvol4 1048576 72392 968736 7% /tmp
/dev/vg00/lvol6 10575872 5155472 5378112 49% /opt
/dev/vg00/lvol5 131072 11912 118264 9% /home

-->I don't know why would cause like that(SAM didn't exit correctly?), the system patches about SAM as below, the newest is PHCO_39360, I will install it.
root@wudai:[/] # swlist -l product | grep -i sam
PHCO_34974 1.0 cumulative SAM patch

[lbseraph]

Pls. check the content I posted above, this case caused by samx process error loop.

Thank you for lsof usage. Have assigned the points.

I will close the thread.

Regards,
lbseraph