Hot backups with fbackup

[Randy Hagedorn]

Hi:
I found on Oracle Metallink that you cannot do a hot backup with fbackup because it loses inodes changes that occur during a backup.

Why do some of you tell me that fbackup is working for them while the Oracle datafiles are in backup mode?
Are your datafiles being accessed while your backup is running?

Thanks,

Dewy

[Martin Johnson]

We use oracle to dump the data prior to using fbackup to backup the data. Trying to backup the data directly using fbackup usually results in corrupted data.

HTH
Marty

[Juan González]

Hi Dewy,
AFAIK once the datafiles are in backup mode you can do the backup of these files with whatever tool: cp, dd, fbackup, ...

Could you write down the link within Metalink?

Best regards,
Juan

[Jeanine Kone]

I have said that we were able to use fbackup for hot backups. We did do them in the middle of the night, so I doubt that anyone was actually using the data at the time. We are not using it right now because we found other problems with fbackup that we have not yet been able to fix while backing up large files.

[R. Allan Hicks]

This perhaps more detail than what the others have said, but it's basically the same thing so no points are expected.

I have been using fbackup for about 5 years from Oracle 7.3.x to 8.1.6. I have also successfully recovered databases using fbackup tapes.

We do things alittle differently in that I put the tablespace into hotbackup and copy and compress the datafiles to another volumne. That volume is copied with fbackup.

When a tablespace is placed into hot backup mode, it is basically frozen and all updates that dbwr would send to the datafiles are sent to the redo log files instead. This keeps the datafile from changing. Should you try to backup a file without it being in hot backup mode, then you are correct you will get a corrupt backup. (I've done it).

The reason we do ours during low activity is that you are filling your redo logs more quickly in hot backup mode because more information go into to them.

From my notes on the backup and recovery workshop (which I highly recommend. It's one of the best oracle classes ever) Pages 4-16 and 17....

1. Put tablespace into backup mode

alter tablespace yourspacehere begin backup;

2. In unix,
cp /users/disk1/users01.dbf /users/backup/user01.dbf

3. alter tablespace yourspacehere end backup;

4. force a checkpoint to synchronize all of the headers with

alter system switch logfile;


I like the disk to disk copy because it's faster than disc to tape and I don't want the tablespaces in hot backup mode any longer than necessary. I also like to compress the file so I don't have to change tapes.

I'd like to see the metalink doc id too.

Oracle has made some improvements to RMAN under 9i and I plan to look into that more and possibly give up my fbackup scheme.

[Yogeeraj_1]

Hi,

The best way is to use Oracle Recovery Manager (RMAN). Do your backups to disk then use fbackup if you wish to sent the files to tape.

RMAN has lots of parameters that you may wish to customise for your own environment.

RMAN is the future in Oracle Backup.

We do only hot backups using RMAN.

Hope this helps!

Best Regards
Yogeeraj

[Juan González]

Hi again,
I've done some research and I find in Metalink what Dewy said:

Literally from Metalink:

**Can't HOT backup - fbackup loses inodes changes that occur during a backup

frecover -s causes sparse files causes paging for >1GB files

I understand the problem with frecover -s. Oracle datafiles can't be sparse files. Speaking quickly, a sparse file is a file where unused blocks doesn't use space on the filesystem, just a mark on the inode.

And from fbackup(1M):
Due to present file-system limitations, files whose inode data, but not their contents, are modified while a backup is in progress might be omitted from the next incremental backup of the same graph. Also, fbackup does not reset the inode change times of files to their original values.

I am not sure how this can affect the fbackup of the datafiles. When a datafile is turned on backup mode, Oracle still access the datafile, I supossed that only to read, so the inode has to be changed to reflect the change in the access time field. I don't know whether other field in the inode is changed.

fbackup first read the inode information and then the file's content, but i don't why is this a problem with a hot backup...

Indeed, I find more information about fbackup of datafiles, mostly from forums, and some of them afirm that they have no problems with fbackup.

Fortunately I use rman :-)

Best regards,
Juan

[Simeon Fox]

Please note - oracle datafiles are NOT frozen when in hot backup mode. This is a common misconception. It is only the headers that are frozen. Oracle can write to the datafiles as they are being backed up, and hence fbackup cannot be used for the initial copy. I use "cp" to transfer the datafiles to a different disk, and apply fbackup to the copied and therefore "cold" files. I believe "tar" can be used to transfer directly to a backup device.

It is the freezing of the file headers that is the key to the hot backup mechanism. When restoring, the archive logs are applied from the start of the backup, leaving the restored file in a consistent state.

[Juan González]

Ok, so what makes the difference between cp and fbackup? If the datafile is being writen why can i use cp or tar but not fbackup?

Best regards
Juan

[Simeon Fox]

Juan - the unix gurus might have a technical explanation. What I know is that fbackup may refuse to copy a file that is being written to (with fbackup(3007): WARNING: File number nnnn was active), making several attempts and eventually rejecting. Whereas tar and cp will copy the file without objecting. And the oracle recovery mechanism takes care of the inconsistent or "fuzzy" state of the copied file as explained above. ie fbackup is not guaranteed to actually copy the file so can't be used for a hot backup.


Regards
SMF

[Rory R Hammond · Accepted solution]

Our databse is mirrored, and I have been performing recoverable automatic backs for several years. Via the following steps. My actual scripts are highly automated and gets its information from a table. So I listed the steps and put examples to each step. This should be enough info for you to figure out how to doit. It took about 2 hours to write the script and tables. I have a program that scans the logfile of the scripts which reports any errors. Some one mentioned fbackup file size limitations. Once the mirrors are mounted you can back them using any method. Make sure you backup the archive logs so that you do forward recovery of the database.

Rory

# Step 1. Set up databases for hot backups by putting them
# in backup mode
#
A: alter system archive log stop;
b: For all of the tablespaces:
alter tablespace tablespace begin backup;

# Step 2. Split mirrors
#
# split the mirrored logical volumes using a single lvsplit command
# instead of separate ones, as this will ensure that
# the drives are consistent in time. This assumes all of lvols are in the same volume group.

A. /usr/sbin/lvsplit /dev/vg01/lvol /dev/vg01/lvol2 /dev/vg01/lvol3

B./usr/sbin/lvsplit /dev/vg02/lvol /dev/vg02/lvol2 /dev/vg02/lvol3

#Step 3. Resume normal database operation. NOTE When automated the database is down in backup mode for a very short time! --less than a minute.
A: alter system archive log start;
b: For all of the tablespaces:
alter tablespace tablespace end backup;
# Step 4. Mount the split filesystems. You need to run fsck on hfs., the veritas doesn't seem to need any thing done.
# filesystem check (fsck) the formerly mirrored logical volumes
# the 'b' extension to the names indicates they're the copies
/usr/sbin/fsck -p /dev/vg01/lvolb
/usr/sbin/fsck -p /dev/vg01/lvol2b
/usr/sbin/fsck -p /dev/vg01/lvol3b

mount /dev/vg01/lvol1b /mnt/db1
mount /dev/vg01/lvol2b /mnt/db2
mount /dev/vg01/lvol3b /mnt/db3
# Step 5. Back up the database
# Use fbackup graphs pointing to backup mount points

# Step 6. Unmount the split filesystems
#
umount /dev/vg01/lvol1b
umount /dev/vg01/lvol2b

umount /dev/vg01/lvol3b
# Step 7. Merge filesystems to re-establish mirroring
#
lvmerge /dev/vg01/lvol1b /dev/vg01/lvol1
lvmerge /dev/vg01/lvol2b /dev/vg01/lvol2
lvmerge /dev/vg01/lvol3b /dev/vg01/lvol

[Randy Hagedorn]

Forum People,

First I would like to thank everyone for the lively and pointed discussion regarding this question. We have been researching this question for about 2 months now and have learned there is not easy answer.

Rory, thank-you for your explaination and well laid out scenario. It just so happens that we do use Mirror/Ux and have been testing the split mirror backup. One further question though. If per chance we were to have a disaster and lost both copies of the mirrored data, how would we recover the data? Would we frecover the data to the mirror drive set and then lvsync them, since the backup is really of the mirror directory, in your example the 'b' version?

Thanks again for your time answering these questions.

Dewy

[Yogeeraj_1]

Hi,

To add to what Mr. Rory said:
============================================================
Before you split the mirrors, you should also issue an ALTER SYSTEM SUSPEND;


From the backup and recovery guide:


Making Backups in a Suspended Database

After a successful database suspension, you
can back up the database to disk or break the mirrors. Because suspending a
database does not guarantee immediate termination of I/O, Oracle recommends
that you precede the SUSPEND statement with a BEGIN BACKUP statement to
place the tablespaces in hot backup mode.

You must use conventional operating system backup methods to back up split
mirrors. RMAN cannot make database backups or copies because these operations
require reading the datafile headers. After the database backup is finished or
the mirrors are re-silvered, then you can resume normal database operations
using the RESUME statement.

Backing up a suspended database without splitting mirrors can cause an extended
database outage because the database is inaccessible during this time. If
backups are taken by splitting mirrors, however, then the outage is nominal. The
outage time depends on the size of cache to flush, the number of datafiles, and
the time required to break the mirror.


That quote was taken from:

http://technet.oracle.com/docs/products/oracle8i/doc_library/817_doc/server.817/a76993/osbackup.htm#940

So, I you won't be using many RMAN features on the split mirror database. You
can use RMAN to basically "copy the file". You'll be taking advantage of the
recovery catalog but thats about it. RMAN will just be copying files. You can
also use RMAN with a standby (data guard) database to offload backups to another
machine.

see

http://technet.oracle.com/docs/products/oracle8i/doc_library/817_doc/server.817/a76990/usingrma.htm#441865
for details on backing up a split mirror database with RMAN.
============================================================
Hope this helps!
Best Regards
Yogeeraj

[Yogeeraj_1]

hi again,

A few more from my notes about COLD v/s HOT backups!
============================================================
Why would you be doing cold backups at all???

You do not want to do the split mirrors. You will obviate many of the benefits
and features of RMAN. You should just be doing HOT backups with RMAN (no need
to "alter tablespace begin/end backup" with RMAN).

You can do the split mirrors with RMAN but its just like doing the backup
without RMAN. You will lose the features available in RMAN that aren't
available without RMAN.

Just do hot backups right now. They are not any harder then cold backups. Your
system should be 100% available right now, today, nothing special need be done.

But -- the bottom line is -- you should have ALWAYS been doing hot backups.
Split mirrors, unless you have triple mirroring, will open you up to failure
during your backup and re-silvering window. Just do a plain old hot backup, its
all you need or want. Don't over complicate the issue with split mirrors,
re-silvering, datafile copies instead of RMAN backups and such.


Hot backups and cold backups are not significantly different from each other in
many respects. there are much FEWER options with cold then hot but they are
very similar.

by the way -- it is really not HOT vs COLD, rather it is ARCHIVE LOG vs
NO-ARCHIVE log. Whether you do hot or cold is up to you.

Lets take an example. We are in ARCHIVE LOG mode. I did a backup last night at
12am. At 8am this morning I did a non-recoverable action -- I did a create
table as select without logging it and dropped the table I just copied (i did a
"reorg" if you will). At 8:30am -- the disks that held the datafiles all
failed. So, what can I do? I can restore my backup from lastnight and either:

o recover it to 7:59:59 AM (or any point in time between 12am and 7:59:59).
I'll have my entire database back to what it looked like at that point in time.
I'll have my table before I dropped it. We won't have the newly created table
of course.

o recover it completely. Now I'll have all of my data EXCEPT that table since
it was not logged.

in the real world, we would have backed up our datafile we did the create
table as select without logging IMMEDIATELY after doing that operation to make
it so we would NOT lose this data. We would tend to want to HOT BACKUP this
file


Now, lets say we did the same thing in noarchive log mode. I did a backup last
night at 12am. At 8am this morning I did a non-recoverable action -- I did a
create table as select without logging it and dropped the table I just copied (i
did a "reorg" if you will). At 8:30am -- the disks that held the datafiles all
failed. So, what can I do?

o I can restore the database to 12am. Period, zippo, thats it, nothing else.


So, just to re-iterate -- it is not HOT vs COLD, its archive log vs no-archive
log and you need to backup (probably in HOT mode) datafiles that are affected by
non-logged operations ASAP after doing them to protect yourself. (eg: in my
example above, I would have created the table as select, BACKED UP the affected
file(s), dropped the old table. At no point in time would I have lost any data)

============================================================

Hope this helps!
Best Regards
Yogeeraj

[Simeon Fox]

Just on the fbackup vs cp issue - My fbackup reports sometimes have instances of :
"fbackup(3009): WARNING: File number NNN was not successfully backed up" for an active file.
This maybe reflects the metalink warnings about using fbackup with a changing inode.

[Bill Hassell]

When fbackup complains about not backing up a file, it is usually because the file is opened exclusively and even root is denied read access. I believe that cp and tar just silently ignore the file and move on. At least fbackup warns you. Concerning sparse files, whether the original file is sparse or not, it is saved with all the missing data full of zeros, and restored that way. There is no easy way to determine that a file is sparse. The only way a sparse file can be restored without writing the zero blocks is to tell frecover to explicitly perform that action...the default is to restore all blocks.

Concerning tar (and other classic Unix backup tools) almost every database is pushing the 2Gb limit and these tools cannot be used for large files.

[Simeon Fox]

Bill - the file which caused the 3009 was the active oracle online redo log. Kind of red herring, as these are not required to be backed up as they cannot be restored, but just happened to be in the backup path. But as the two inactive redo logs were backed up successfully would this be an inode issue?
I can reproduce this now and fbackup generally objects to the active logfile. cp just copies it.


Regards
SMF