topic Re: root size discrepancy between bdf and backup in Operating System - HP-UX

root size discrepancy between bdf and backup

Damien Lass — Tue, 27 Nov 2001 15:30:45 GMT

Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 409600 330052 74663 82% /

Here is a bdf of our root filesystem. A discrepancy exists between a bdf and our backup statistics. Whenever running an Omniback backup of root, its statistics report backing up 129MB. As you can see, our bdf reports 330MB used in root. Can someone help point me in the right direction about this discrepancy?

We have rebooted the server thinking that a file was deleted that bdf thought still existed, but this was not the case. After the reboot, the filesystem still reported 82% (330MB used).

Re: root size discrepancy between bdf and backup

harry d brown jr — Tue, 27 Nov 2001 15:34:38 GMT

Are you doing an incremential backup? Also, is the amount of data backed up compressed?

live free or die
harry

Re: root size discrepancy between bdf and backup

Patrick Wallek — Tue, 27 Nov 2001 15:37:05 GMT

Any way to get a list of files and compare what is on your backup to what is in /?

300+ MB seems awfully large for /. My largest is 126 MB and it is on a system that still has / and /stand in the same LV.

Check for files that don't belong, like /dev/rmt/om (letter o instead of zero) and see if that makes a difference.

Re: root size discrepancy between bdf and backup

Craig Rants — Tue, 27 Nov 2001 15:40:48 GMT

I would bet that Patrick is right on. I would look at all you backup definitions to make sure someone has not entered a tape device in error.

Look for a non character device in /dev

Good Luck,
C

Re: root size discrepancy between bdf and backup

Roger Baptiste — Tue, 27 Nov 2001 15:55:11 GMT

Damien,

The possible reasons could be:

It is an incremental backup
or

Some directories have been excluded from the backup in the datalist definition

or

The usage of root increased dramatically (!!?) after you took the backup.

The best way to check is, take the backup again, this time interactively and see what the size is. Since it is just a few hundred MB, it should zip through quickly.

HTH
raj

Re: root size discrepancy between bdf and backup

Damien Lass — Wed, 28 Nov 2001 19:41:30 GMT

Thank you all for you input but I still have not figured out the problem... Your suggestions have eliminated some potential issues though!

Thus far I have...
1.) Compared the last several (full) backups of / and all report approx. 120MB backed up. This includes ALL files and directories under the root. The data is not compressed nor the backup incremental. The recent backups also report 120MB (while 'bdf' still reports 330MB).

2.)I have looked in /dev/rmt for non-character devices and have found none.

If there is anything else I might be missing please let me know. Is there a command out there similar to 'bdf' that I can run to compare the results? Any ideas are welcome.

Re: root size discrepancy between bdf and backup

Craig Rants — Wed, 28 Nov 2001 19:51:13 GMT

What device are you copying your backup to? This device could be performing hardware compression. Just a thought.

C

Re: root size discrepancy between bdf and backup

Uday_S_Ankolekar — Wed, 28 Nov 2001 19:51:52 GMT

Hi,

Note that a part of total kbytes are used for the minfree area ( this area is reserved to mantain performance ) To see what is the total allocated space use:
df -kt /FS
the difference between the kbites you see from bdf /FS and the total allocated Kb is the minfree.
To calculate the percentage do:
used allocated space / total allocated space * 100

-USA

Re: root size discrepancy between bdf and backup

Damien Lass — Wed, 28 Nov 2001 20:04:14 GMT

Craig... My backup does compress the files, but it reports the actual size of the file systemsin its log. All other files backed up match their respective file sizes when compared to a 'bdf'. Root does not. I was unclear on this in my previous message. Thanks.

Re: root size discrepancy between bdf and backup

Craig Rants — Wed, 28 Nov 2001 20:14:04 GMT

Are there any database type files in / . Database files are read differently by different apps, i.e., one app may read the database file as the actual size whereas another app reads the database file as the allocated size.

We found this as a problem between fbackup and cpio, when copying a dir with cpio it would be much larger than actual size. Of course this does not fit your situation but it could provide some insight. Just a thought.
C

Re: root size discrepancy between bdf and backup

A. Clay Stephenson — Wed, 28 Nov 2001 20:17:01 GMT

Hi Damien:

Before we go any further, let's do a few things:

1) How big is the actual logical volume?
lvdisplay /dev/vg00/lvol3

I'm having a hard time understanding why anyone would create as big a root
filesystem as you have.

2) do - df -k / and see if it reports similar numbers to those of bdf

3) do a du -x / and see if you see enormous directories.

If indeed your filesystem is that large, I suspect two possible answers: a) enormous directories i.e. they once had many files in them but the files have been deleted. b) sparse files. Either one of these would appear large but would in fact not be.

Re: root size discrepancy between bdf and backup

Craig Rants — Wed, 28 Nov 2001 20:22:19 GMT

The problem I descriped between cpio and fbackup is a sparse file problem. We may be on to something for your situation since a couple of us have mentioned it now.

C

Re: root size discrepancy between bdf and backup

Damien Lass — Wed, 28 Nov 2001 21:36:42 GMT

Here is some more information based on Clay's response...

Background information:
400MB - space allocated for root
337MB - reported space used (df -k /)

I ran a bdf and then a du -skx on a couple logical volumes and these are the results:

/var - bdf: 965MB du -skx /var: 955MB
/usr - bdf: 620MB du -skx /usr: 613MB
/ - bdf: 337MB du -skx / : 120MB

These are approximately the same results reported when an Omniback backup reports its results. As you can see root is the only one that is not similar. Is this an accurate test? Does du -skx / capture all files/directories under root?

Any more ideas?

Re: root size discrepancy between bdf and backup

Damien Lass — Wed, 28 Nov 2001 21:51:01 GMT

There are 8 other servers set up in my environment. They are all set up very similar to each other. I am not having problems with any of the other servers reporting large root filesystems.

When I run and compare 'bdf', 'df -k', and 'du -skx' they all come back the same. Root is about 100-120MB on all.

I have looked around for any database files that might have been dropped into root accidentally. I can find no files (or combination of files) that would make root so abnormally large. The serve has been rebooted and the same results are reported.

Re: root size discrepancy between bdf and backup

A. Clay Stephenson — Wed, 28 Nov 2001 22:11:20 GMT

Hi Damian:

I never meant to imply that a large root filesystem would not function perfectly; it's just that there is very little reason to ever have a root file system bigger than about 200GB (and that is very, very generous). The reason for that is that once the OS is loaded along with patches the size of the root filesystem should be all but static.

I really think your problem is sparse files. I just need to think of a way to find them.

Here is a very common way for a sparse file to get created:
1) Open a file, write 1K of data at block 0;
2) lseek to Block 1024
3) Write 1K of data.
4) Close the file.

You just wrote a 1MB file that really has 2K of data.

Re: root size discrepancy between bdf and backup

A. Clay Stephenson — Wed, 28 Nov 2001 22:17:51 GMT

Hi Damian:

Let's not overlook one of the most common sparse files - core files.

cd /

find . -xdev -name 'core'

A core file could do this very easily.

Clay

Re: root size discrepancy between bdf and backup

Darrell Allen — Thu, 29 Nov 2001 00:47:22 GMT

Hi Damien,

I'd just about bet you have a large file held open by a running process that was deleted by hand. Until that process "closes" the file, bdf will reflect the space used as if the file is there. du only looks at the space used by directories and files that it sees. Since your backup is reporting basically the same amount of space backed up as du shows, I believe your backup is fine.

Now you've just got to find the "open but deleted" file.

Darrell

Re: root size discrepancy between bdf and backup

Darrell Allen — Thu, 29 Nov 2001 00:53:20 GMT

Here's a couple of threads that may help you find the "deleted while open" file:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x19248ffa98a2d5118ff10090279cd0f9,00.html

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xfecdf715edc6d5118ff10090279cd0f9,00.html

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xa89f0cb17a32d5118fef0090279cd0f9,00.html

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xc0621012aa92d5118ff10090279cd0f9,00.html

Darrell

Re: root size discrepancy between bdf and backup

Damien Lass — Thu, 29 Nov 2001 20:56:37 GMT

Clay (or anybody else)... Would a system reboot change the size of a sparsed file? In other words would a file that reports being 2MB in size using 'bdf' (that is actually only 3k) report only 3k after a system reboot? I assume it would not but I'm not sure. Would du -skx report the file differently?

Does anybody know of a way to find sparsed files if this is the case?

Re: root size discrepancy between bdf and backup

John Payne_2 — Thu, 29 Nov 2001 21:03:39 GMT

What about having a filesystem mounted on top of data? Ie. If you have a /data filesystem, and at some point it was unmounted while users were on the box. A large file or files were created. Since the directory hangs off of / when the filesystem is unmounted, it uses /'s space. When the filesystem is mounted, / is still filled, but you do not see the files. Your backup problibly also does not see the files. If you suspect this, you could boot to single user mode and look at all your mount points hanging off / for files that shouldn't be there since the filesystems aren't unmounted.

If you suspect open files, (which I would not expect, since you rebooted.) do a du -r and find folders that look too big. Then use lsof to see if there are any open files...

How it helps.

John