root size discrepancy between bdf and backup

[Damien Lass]

Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 409600 330052 74663 82% /

Here is a bdf of our root filesystem. A discrepancy exists between a bdf and our backup statistics. Whenever running an Omniback backup of root, its statistics report backing up 129MB. As you can see, our bdf reports 330MB used in root. Can someone help point me in the right direction about this discrepancy?

We have rebooted the server thinking that a file was deleted that bdf thought still existed, but this was not the case. After the reboot, the filesystem still reported 82% (330MB used).

[harry d brown jr]

Are you doing an incremential backup? Also, is the amount of data backed up compressed?

live free or die
harry

[Patrick Wallek]

Any way to get a list of files and compare what is on your backup to what is in /?

300+ MB seems awfully large for /. My largest is 126 MB and it is on a system that still has / and /stand in the same LV.

Check for files that don't belong, like /dev/rmt/om (letter o instead of zero) and see if that makes a difference.

[Craig Rants]

I would bet that Patrick is right on. I would look at all you backup definitions to make sure someone has not entered a tape device in error.

Look for a non character device in /dev

Good Luck,
C

[Roger Baptiste]

Damien,

The possible reasons could be:

It is an incremental backup
or

Some directories have been excluded from the backup in the datalist definition

or

The usage of root increased dramatically (!!?) after you took the backup.

The best way to check is, take the backup again, this time interactively and see what the size is. Since it is just a few hundred MB, it should zip through quickly.

HTH
raj

[Damien Lass]

Thank you all for you input but I still have not figured out the problem... Your suggestions have eliminated some potential issues though!

Thus far I have...
1.) Compared the last several (full) backups of / and all report approx. 120MB backed up. This includes ALL files and directories under the root. The data is not compressed nor the backup incremental. The recent backups also report 120MB (while 'bdf' still reports 330MB).

2.)I have looked in /dev/rmt for non-character devices and have found none.

If there is anything else I might be missing please let me know. Is there a command out there similar to 'bdf' that I can run to compare the results? Any ideas are welcome.

[Craig Rants]

What device are you copying your backup to? This device could be performing hardware compression. Just a thought.

C

[Uday_S_Ankolekar]

Hi,

Note that a part of total kbytes are used for the minfree area ( this area is reserved to mantain performance ) To see what is the total allocated space use:
df -kt /FS
the difference between the kbites you see from bdf /FS and the total allocated Kb is the minfree.
To calculate the percentage do:
used allocated space / total allocated space * 100

-USA

[Damien Lass]

Craig... My backup does compress the files, but it reports the actual size of the file systemsin its log. All other files backed up match their respective file sizes when compared to a 'bdf'. Root does not. I was unclear on this in my previous message. Thanks.

[Craig Rants]

Are there any database type files in / . Database files are read differently by different apps, i.e., one app may read the database file as the actual size whereas another app reads the database file as the allocated size.

We found this as a problem between fbackup and cpio, when copying a dir with cpio it would be much larger than actual size. Of course this does not fit your situation but it could provide some insight. Just a thought.
C

[A. Clay Stephenson]

Hi Damien:

Before we go any further, let's do a few things:

1) How big is the actual logical volume?
lvdisplay /dev/vg00/lvol3

I'm having a hard time understanding why anyone would create as big a root
filesystem as you have.

2) do - df -k / and see if it reports similar numbers to those of bdf

3) do a du -x / and see if you see enormous directories.

If indeed your filesystem is that large, I suspect two possible answers: a) enormous directories i.e. they once had many files in them but the files have been deleted. b) sparse files. Either one of these would appear large but would in fact not be.

[Craig Rants]

The problem I descriped between cpio and fbackup is a sparse file problem. We may be on to something for your situation since a couple of us have mentioned it now.

C

[Damien Lass]

Here is some more information based on Clay's response...

Background information:
400MB - space allocated for root
337MB - reported space used (df -k /)

I ran a bdf and then a du -skx on a couple logical volumes and these are the results:

/var - bdf: 965MB du -skx /var: 955MB
/usr - bdf: 620MB du -skx /usr: 613MB
/ - bdf: 337MB du -skx / : 120MB

These are approximately the same results reported when an Omniback backup reports its results. As you can see root is the only one that is not similar. Is this an accurate test? Does du -skx / capture all files/directories under root?

Any more ideas?

[Damien Lass]

There are 8 other servers set up in my environment. They are all set up very similar to each other. I am not having problems with any of the other servers reporting large root filesystems.

When I run and compare 'bdf', 'df -k', and 'du -skx' they all come back the same. Root is about 100-120MB on all.

I have looked around for any database files that might have been dropped into root accidentally. I can find no files (or combination of files) that would make root so abnormally large. The serve has been rebooted and the same results are reported.

[A. Clay Stephenson]

Hi Damian:

I never meant to imply that a large root filesystem would not function perfectly; it's just that there is very little reason to ever have a root file system bigger than about 200GB (and that is very, very generous). The reason for that is that once the OS is loaded along with patches the size of the root filesystem should be all but static.

I really think your problem is sparse files. I just need to think of a way to find them.

Here is a very common way for a sparse file to get created:
1) Open a file, write 1K of data at block 0;
2) lseek to Block 1024
3) Write 1K of data.
4) Close the file.

You just wrote a 1MB file that really has 2K of data.

[A. Clay Stephenson]

Hi Damian:

Let's not overlook one of the most common sparse files - core files.

cd /

find . -xdev -name 'core'

A core file could do this very easily.

Clay

[Darrell Allen]

Hi Damien,

I'd just about bet you have a large file held open by a running process that was deleted by hand. Until that process "closes" the file, bdf will reflect the space used as if the file is there. du only looks at the space used by directories and files that it sees. Since your backup is reporting basically the same amount of space backed up as du shows, I believe your backup is fine.

Now you've just got to find the "open but deleted" file.

Darrell

[Darrell Allen]

Here's a couple of threads that may help you find the "deleted while open" file:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x19248ffa98a2d5118ff10090279cd0f9,00.html

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xfecdf715edc6d5118ff10090279cd0f9,00.html

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xa89f0cb17a32d5118fef0090279cd0f9,00.html

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xc0621012aa92d5118ff10090279cd0f9,00.html

Darrell

[Damien Lass]

Clay (or anybody else)... Would a system reboot change the size of a sparsed file? In other words would a file that reports being 2MB in size using 'bdf' (that is actually only 3k) report only 3k after a system reboot? I assume it would not but I'm not sure. Would du -skx report the file differently?

Does anybody know of a way to find sparsed files if this is the case?

[John Payne_2]

What about having a filesystem mounted on top of data? Ie. If you have a /data filesystem, and at some point it was unmounted while users were on the box. A large file or files were created. Since the directory hangs off of / when the filesystem is unmounted, it uses /'s space. When the filesystem is mounted, / is still filled, but you do not see the files. Your backup problibly also does not see the files. If you suspect this, you could boot to single user mode and look at all your mount points hanging off / for files that shouldn't be there since the filesystems aren't unmounted.

If you suspect open files, (which I would not expect, since you rebooted.) do a du -r and find folders that look too big. Then use lsof to see if there are any open files...

How it helps.

John

[Darrell Allen]

Hi Damien,

Sorry for not reading more closely. I missed the part where you said you had rebooted.

John's theory is a good one.

Darrell

[Bill Hassell]

If a bdf or du of a filesystem changes significantly after a reboot, there was very likely large open files that were marked as temporary and once the process(es) were terminated by shutdown, the space would be returned.

Neither HP-UX nor the filesystem management code will change any file in a filesystem, so a sparse file remains unchanged during a reboot.

A not-so common, but annoying technique by some programmers is to create a temp file that occupies space (sometimes lots of space) but has no visible entry in the directory.

A similar situation can be seen when removing an open file. Unless steps are taken to lock the file, the file can be removed (as far as the directory is concerned) but the space remains until the process that owns the file is terminated--at which point the space comes back for du and bdf.

[Damien Lass]

Bill - How would I identify a temp file that has no visible entry in the directory? We have rebooted the server thus restarting all processes that might have held open any deleted files.

[Mark Vollmers]

Damien-

One way would be to just go and start killing off processes that you think might be contributing. Since it always comes back after a reboot, it would be something that always starts back up and repeats itself. Look in /sbin/rc#.d (mine are in rc2.d, but might your's might not be) and see what programs have startup scripts in there. Anything in S9** should be fair game, since that is an open number. Just a thought. Good luck!

Mark