Disk full but no files..

[Arno Haverlach]

Hello,

We have got a HP-UX 11.11 system that has some problems on a partition. /oracle/admin/zipprod1 is full according to bdf but there are just a few megs used..

# bdf
/dev/vgprod1/lvol4 524288 522234 1989 100% /oracle/admin/zipprod1

#du -ka
0 ./lost+found
6 ./scripts/cr_control_zipprod1.sql
2 ./scripts/create_db/CreateDB.sql
1 ./scripts/create_db/CreateDBCatalog.sql
2 ./scripts/create_db/CreateDBFiles.sql
3 ./scripts/create_db/init.ora.422003142720
1 ./scripts/create_db/postDBCreation.sql
1 ./scripts/create_db/zipprod1.sh
11 ./scripts/create_db
17 ./scripts
1 ./bdump/alert_zipprod1.log
4 ./bdump
0 ./cdump
1 ./create
2 ./pfile/init.ora
1 ./pfile/initzipprod1.ora
3 ./pfile/orapwzipprod1
5 ./pfile/spfilezipprod1.ora
3 ./pfile/spfilezipprod1.ora.09022004.lv
5 ./pfile/spfilezipprod1.ora.02082004.lv
4 ./pfile/init.ora.org.07042005.lv
5 ./pfile/spfilezipprod1.ora.07042005.lv
29 ./pfile
1517 ./udump/sbtio.log
1 ./udump/zipprod1_ora_12964.trc
2 ./udump/zipprod1_ora_26629.trc
1 ./udump/zipprod1_ora_23449.trc
1 ./udump/zipprod1_ora_29993.trc
1525 ./udump
1577 .

df -i shows that there are enough free i-nodes.

# df -i
/oracle/admin/zipprod1 (/dev/vgprod1/lvol4 ) : 548 total i-nodes
510 free i-nodes
38 used i-nodes
6 % i-nodes used

lsof does't show anything usefull

#lsof | grep -i "/oracle/admin/zipprod1"
sh 4577 root cwd DIR 64,0x50004 1024 2 /oracle/admin/zipprod1
lsof 8357 root cwd DIR 64,0x50004 1024 2 /oracle/admin/zipprod1
grep 8358 root cwd DIR 64,0x50004 1024 2 /oracle/admin/zipprod1
oracle 29341 oracle 2u REG 64,0x50004 2344960 27 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29341 oracle 6u REG 64,0x50004 284977376 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29341 oracle 8u REG 64,0x50004 68695799 24 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29343 oracle 6u REG 64,0x50004 284977376 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29343 oracle 8u REG 64,0x50004 284977376 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29345 oracle 6u REG 64,0x50004 284977376 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29345 oracle 8u REG 64,0x50004 68695799 24 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29347 oracle 6u REG 64,0x50004 284977376 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29347 oracle 8u REG 64,0x50004 284977376 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29349 oracle 2u REG 64,0x50004 1346 62 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29349 oracle 6u REG 64,0x50004 284977376 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29349 oracle 8u REG 64,0x50004 68695799 24 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29351 oracle 6u REG 64,0x50004 284977376 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29351 oracle 8u REG 64,0x50004 284977376 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29353 oracle 6u REG 64,0x50004 284977376 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29353 oracle 8u REG 64,0x50004 284977376 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29355 oracle 6u REG 64,0x50004 284977376 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29355 oracle 8u REG 64,0x50004 284977376 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29357 oracle 2u REG 64,0x50004 79638528 17 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29357 oracle 6u REG 64,0x50004 284977376 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29357 oracle 8u REG 64,0x50004 68695799 24 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29359 oracle 2u REG 64,0x50004 80399360 16 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29359 oracle 6u REG 64,0x50004 284977376 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29359 oracle 8u REG 64,0x50004 68695799 24 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)


Any idea's on this one..?
also is it possible to do a fsck on a live filesystem as i'm unable to umount the partition right now..

Thanks..

[Arunvijai_4]

Hello,

Try running "fsck" to find out any inconsistency in your filesystem.

-Arun

[Mark Nieuwboer]

Hi Arno,

probebly a file is removed which is still being used.
do an fuser /oracle/admin/zipprod1
to see which process uses the filesystem and restart that process.

grtz. Mark

[Cem Tugrul]

Arno,
du -kx / | sort -rn

[Cem Tugrul]

du -kx oracle/admin/zipprod1|sort -rn

[Steven E. Protter]

Shalom,

If a file has been deleted but the process still has a lock on it, the space will not be available.

fuser -cu /filesystem
# to display the processes.

You may kill them individually or:

fuser -cuk /filesystem
# very dangerous can halt a system if run incorrectly.

SEP

[Hemmetter]

Hi Arno


did you try
# lsof +L1 /oracle/admin/zipprod1 | grep REG

This gives you the "unlinked" files.


rgds
HGH

[Rashid Hamid]

Try to umount and mount back the file system

[Arno Haverlach]

fuser -cu /oracle/admin/zipprod1
/oracle/admin/zipprod1: 4577c(root) 29345o(oracle) 29357o(oracle) 29359o(oracle) 11030o(oracle) 29355o(oracle) 29343o(oracle) 29349o(oracle) 29341o(oracle) 29347o(oracle) 29351o(oracle) 11655c(root) 29353o(oracle)

# lsof +L1 /oracle/admin/zipprod1 | grep REG

tnslsnr 29262 oracle 3u REG 64,0xa 621036 0 35059 /oracle (/dev/vg00/lvol10)
oracle 29341 oracle 2u REG 64,0x50004 2344960 0 27 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29341 oracle 6u REG 64,0x50004 284977376 0 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29341 oracle 8u REG 64,0x50004 68695799 0 24 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29343 oracle 6u REG 64,0x50004 284977376 0 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29343 oracle 8u REG 64,0x50004 284977376 0 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29345 oracle 6u REG 64,0x50004 284977376 0 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29345 oracle 8u REG 64,0x50004 68695799 0 24 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29347 oracle 6u REG 64,0x50004 284977376 0 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29347 oracle 8u REG 64,0x50004 284977376 0 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29349 oracle 2u REG 64,0x50004 1346 0 62 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29349 oracle 6u REG 64,0x50004 284977376 0 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29349 oracle 8u REG 64,0x50004 68695799 0 24 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29351 oracle 6u REG 64,0x50004 284977376 0 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29351 oracle 8u REG 64,0x50004 284977376 0 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29353 oracle 6u REG 64,0x50004 284977376 0 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29353 oracle 8u REG 64,0x50004 284977376 0 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29355 oracle 6u REG 64,0x50004 284977376 0 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29355 oracle 8u REG 64,0x50004 284977376 0 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29357 oracle 2u REG 64,0x50004 79638528 0 17 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29357 oracle 6u REG 64,0x50004 284977376 0 14 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)
oracle 29357 oracle 8u REG 64,0x50004 68695799 0 24 /oracle/admin/zipprod1 (/dev/vgprod1/lvol4)

I will try to restart oracle tonight as i'm unable to it right now..

See if that helps.

[F Verschuren]

My first gess is alsow a file that was removd when it was in used.

If this the above solutions does not resolve the probelem it can be that if there is a other filesystem mounted under this filesystems that you have files in the mountpoint you can find this "hidden"files whit:
ff /dev/vgprod1/lvol4
where you can see all the files that are on the lvol.

Kind regarts, Freek Verschuren