/ (root) 100% Full - But It's Not ???

[Lee Harris_5]

Hi,

I'm having a weird problem on a RedHat AS 3 update 3 server. df is reporting that the root filesystem is 100% full...

Filesystem Size Used Avail Use% Mounted on
/dev/vg00/lvol1 496M 496M 0 100% /

However, it's not... a du -x -h / shows that it only contains 235MB of data, not 496MB as per df...

Obviously you can't remount the root filesystem, so we first off just rebooted the server. This made no difference, then we set the box to perform an fsck of the filesystem on the next boot, and rebooted the server again...stil no difference.

I'm kind of running out of ideas, and wondered if anyone else had seen this problem or something similar?

Thanks - Lee

[RAC_1]

what does following report?
du -kx / | sort -n
Check big dirs/files.

[Lee Harris_5]

Here's the details from the command, I've omitted some of the output, and only included files mentioned which are of any considerable size. However, as I mentioned before, du reports the contents of / (root) to be only 235MB whereas df reports it as 496MB...strange....

# du -kx / |sort -n

4609 /root
5118 /lib/modules/2.4.21-20.ELsmp/kernel/drivers/scsi
5166 /etc/gconf/gconf.xml.defaults/schemas
5230 /lib/modules/2.4.21-20.ELcustom/kernel/drivers/scsi
5503 /etc/gconf/gconf.xml.defaults
8662 /etc/gconf
9913 /etc/lvmconf
10198 /bin/bin/admincmd
10695 /lib/modules/2.4.21-20.EL/kernel/drivers
11202 /sbin
13928 /lib/modules/2.4.21-20.EL/kernel
14158 /lib/modules/2.4.21-20.EL
14407 /bin/bin
14584 /lib/modules/2.4.21-20.ELsmp/kernel/drivers
17974 /lib/modules/2.4.21-20.ELsmp/kernel
18208 /lib/modules/2.4.21-20.ELsmp
19590 /bin
26495 /etc
28029 /lib/modules/2.4.21-20.ELcustom/kernel/drivers
33355 /lib/modules/2.4.21-20.ELcustom/kernel
33710 /lib/modules/2.4.21-20.ELcustom
35651 /lib/ssa/gcc-lib/i386-redhat-linux-gnu/3.5-tree-ssa/include/c++/i386-red
hat-linux-gnu/bits/stdc++.h.gch
35775 /lib/ssa/gcc-lib/i386-redhat-linux-gnu/3.5-tree-ssa/include/c++/i386-red
hat-linux-gnu/bits
35776 /lib/ssa/gcc-lib/i386-redhat-linux-gnu/3.5-tree-ssa/include/c++/i386-red
hat-linux-gnu
37990 /lib/ssa/gcc-lib/i386-redhat-linux-gnu/3.5-tree-ssa/include/c++
41557 /lib/ssa/gcc-lib/i386-redhat-linux-gnu/3.5-tree-ssa/include
66077 /lib/modules
85502 /lib/ssa/gcc-lib/i386-redhat-linux-gnu/3.5-tree-ssa
85503 /lib/ssa/gcc-lib/i386-redhat-linux-gnu
85504 /lib/ssa/gcc-lib
95231 /lib/ssa
181472 /lib
245192 /

[Stuart Browne]

What type of filesystem is it (ext2? reiserfs?), and what's the block size?

Also try using 'lsof' to see if any large files are opened, but no longer on the filesystem (i.e. someone rm'd a file, but something still had a file-handle open to it).

[Vipulinux]

Hi Lee

Stuart's point is valid but if you have rebooted the case that should not happen..

Can you however create a file or a dir in the / filesystem?

Cheers

[Lee Harris_5]

Hi, yes exactly, the box has been rebooted, so if someone had rm'd a large file, it would no longer be being locked in memory following the reboot. I can create a file in / which is further proof that it's not actually full. I don't know if this helps, but I believe this is a NetBackup Media server, so I'm not sure if its that software that's having some effect???

Thanks - Lee

[Stuart Browne]

It certainly does seem odd.

The reason why I mentioned it is because it's recorruing between boots.

So it's still a viable possiblity (i.e. a startup routine is doing something odd).

Is the /tmp structure on the root filesystem, or on it's own (or using a ramfs) ?

[Vipulinux]

Hi
Could be bcoz its the backup media server...once the backup is complete does it show 100% full...


Cheers

[Vitaly Karasik_1]

Let's reboot this box into single user mode and check what we will see.

[Ivan Ferreira]

What is the complete file system structure reported by df?

Maybe, the directory that uses the disk space is hidden because is a mount point. I would try umounting all file systems in single user to check if some mount point has files inside.

[Lee Harris_5]

Hi,

/tmp is a separate filesystem, and its not using a ramfs or anything like that. The backups run overnight, but the filesystem is full all of the time, it doesn't go up and down.

And i would boot it into Single user mode, but there's no iLo and I haven't got the energy to walk to the server room...what with it being Friday afternoon and all :-) I'll leave this one until Monday I think.

Thanks

[Steven E. Protter]

Shalom Lee,

Few possibilities:

1) the old hidden file trick. you make a folder, copy 50 MB into it and then mount a filesystem on it. The files are there, taking up space on root and are absolutly invisible. To find such things, you boot to single user mode or you umount your filesystems one at a time and look for hidden files.

2) What you think is a folder is not a folder. I commonly make mistakes testing batch jobs, define an incorrect path and the file copy goes into the /root/ folder. Take a look there.

SEP

[Marius Pana_1]

Hi Harris,

Strange! Try an lvdisplay -v /dev/vg00/lvol1 and see what the size of the lv is. I once extended an lv and df reported the old size because I had forgotten to extend the filesystem as well. Maybe this is something that happend for you as well?

//Marius

[Andrew Young_2]

Hi Lee

Since you have rebooted the server I would imagine its not a case of df cacheing information. Run
df -k /
to get the logical volume name for the root filesystem and then run

tune2fs -l

That will give you the superblock info for that filesystem and help with the diagnosis.