Offline backup of Oracle Database in MC/ServiceGuard

[Mary Grace Inumerable]

I would like to do a script that will do an offline backup of Oracle database located in the vg02 volume group which is controlled by MC/ServiceGuard. I want to ask how to manually activate a volume group w/out involving MC/ServiceGuard so I can backup the oracle datafiles.

I dont have any formal training in MC/ServiceGuard but here are the series of commands in my script(in order) :
1. Stop the package
cmhaltpkg -v pkg1

2. Remove vg02 from high availability cluster
vgchange -c n /dev/vg02

3. Activate vg02
vgchange -a y /dev/vg02

4. Mount vg02
mount /dev/vg02/datavg /datavg

5. Backup command specifically cpio command

6. Unmount vg02
umount /datavg

7. Deactivate vg02
vgchange -a n /dev/vg02

8. Mark vg02 as a member of the high availability cluster
vgchange -c y /dev/vg02

9. Run the package
cmrunpkg -v pkg1

10. Enable Package Switching
cmmodpkg -v -e pkg1

I would appreaciate if you could give me some tips on how to do an offline oracle database backup with MC/ServiceGuard configuration.

Thanks,
Grace

[Sanjay_6]

Hi Mary,

Why do you want to take an offline backup of the oracle database everyday. That would be a lot of problem. We normally take the oracle database backup in hot backup mode, where we use an oracle script to put the database in hot backup mode before the start of the backup and take it out of the hot backup mode at the end of the backup. The users are still able to use the database in hot backup mode, but i believe the access is a little slow as compare to the normal access. However if you still want to do the backup in cold mode, you have to do this,

stop the packge, keep the cluster running,
# cmhaltpkg -v pkg_name

Activate the packge VG on one of the nodes,

# vgchange -a e /dev/vg_name

Do this vgchnage for all the VG's in that packge if you want to backup the data on those VG's. You can activate the VG in exclusive mode on any of the cluster nodes.

Now mount the lv's in that VG to their mount points. I have a small mount script for my maintainance purpose, which simply mounts the lv manually and another to simply unmount them. Do take care of the mount and unmount order for the LV's if the mount point of one lv reides inside the other mount point.

Once you have mounted the filesystems, simply take the backup you want. When done, unmount the lv's and then deactivate the VG,

# vgchange -a n /dev/vg_name

Now start the package again on the node,

# cmmodpkg -e pkg_name

Hope this helps.

Regds

[Sanjay_6]

Hi Mary,

You don't have to do a "vgchange -c n /dev/vg_name" and "vgchange -c y /dev/vg_name" to activate the vg and deactivate it. You use the "-c y" and "-c n " option with vgchange to make the VG cluster aware/unaware when you want to use the VG if the cluster service in not running on the node. If you don't halt the cluster service on the node, you can activate the vg in exclusive mode on that node using "vgchange -a e /dev/vg_name" and then deactivate it using "vgchange -a n /dev/vg_name".

Hope this helps.

Regds

[Steven Sim Kok Leong]

Hi,

An offline backup is preferred over an online backup if you are using an XP256 or XP512 etc, and you have Business Copy (BC).

If you have MC/ServiceGuard and Business Copy for XP512 or XP256, remember to perform a HORCM pairsplit after shutting down your MC/ServiceGuard package before initiating the backup.

echo "Halting Serviceguard cluster on XXX."
echo "ZZZ cluster stopped on `date`" >> /var/adm/sa/log/ZZZ.log
/usr/sbin/cmhaltpkg ZZZ

echo "Splitting instance vgYY"
/usr/bin/pairsplit -g vgYY

...

Hope this helps. Regards.

Steven Sim Kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

[Mary Grace Inumerable]

Hi Sanjay!

Thanks for your prompt reply! I still have a problem with MC/ServiceGuard. To test the procedures you have given to me, I have tried populating a test database from the live database while the package is down. I have completely stop the package,
#cmhaltpkg -v pkg_name

activate the volume group
#vgchange -a e /dev/vg_name

then mount the specific volume group.

After mounting the volume group, i have copied the live oracle database datafiles from test database directory. Then remove the *.ctl and *.log files then run the script to create and start the database

But during startup of the test database i got this error message, " ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '/datavg/q1/oradata/flwweb/system01.dbf'

It seems the file is not consistent, I have done populating the test database by copying the live database in systems without MC/ServiceGuard wherein I can normally shutdown the database and do the copying. Is it possible that the backup will not work fine because of this error?

Thanks again for your help!

Grace

[Sanjay_6]

Hi Grace,

I'm sorry to say this seems like a oracle database issue. You see i'm not so good with the database stuff. I've two great dba's sitting within shouting distance and i just holler at them if there is something i need to know and right now i'm at home, so they cannot help me.

But let me tell you this much, this is not related to service guard. if the database should have started the way you have got it configured, it should have started even with service guard running on the system. Take a look at the database configuration stuff. You are probably missing something. Look at the oracle database error manual and it will give you more detail on the ora error you are getting. there is a possibility some environment variable might be wrong or missing. You can use the "oerr" command in $ORACLE_HOME/bin to learn more about the oracle ora error you are getting. Use this syntax,
# $ORACLE_HOME/bin/oerr ora xxxxx

here xxxxx is the error no you are getting with ora. Use tns in place of ora with the command if you get a tns error or so on. If you don't get this resolved by the time i get to office, i'll talk to my dba about this.

Post a message if this porblem gets resolved.

Hope this helps.

Regds

[Steven Gillard_2]

Its failing because you've copied database files from another system, then deleted the control files. These are very important! A cold backup should include the control files, and I also like to include the redo logs because their inclusion simplifies the recovery procedure slightly.

If you're trying to build a replica of your production database, do not remove the control files and redo logs. Simply copy everything over while you have the production database shut down, then start the database on the test system.

If the script you are running on the test system is performing a CREATE DATABASE, then you do not need to copy anything over from production except your init.ora file, because all necessary files will be created from scratch.

Hope this makes sense.

Regards,
Steve

[John Palmer]

Hi Grace, welcome to the ITRC!

Why do you want to go to the trouble of shutting the cluster down in order to do an offline backup?

Can you not just close the database, back it up then start it up again while the package is running? I do this in several Serviceguard clusters that I support. You need to check however that your package is not monitoring that the database is up. If it does it'll have to be modified to allow closure for backup.

Regards,
John

[Justo Exposito]

Hi Mary,

This is my backup scripts for the Oracle Database, hope this help you.

Regards,

Justo.

[Justo Exposito]

Hi Mary,

This is my second backup scripts for the Oracle Database, hope this help you.

Regards,

Justo.

[Justo Exposito]

Hi Mary,

This is my param file for check wich kind of stop or startup Oracle Database. This is because we have two boxes (hp3 and hp4) in cluster and two no cluster, hope this help you.

Regards,

Justo.

[Mary Grace Inumerable]

Thanks to all of you who reply to my message!

I dont think this is an oracle stuff problem coz our dba is very much sure that this is a service guard problem. I may say she's an expert in oracle database and as I have said we have tried populating databases in other systems without a service guard configuration wherein we can normally shut down the database.

Finally, we have successfully populated the database but the problem is we have halted the secondary server as in shut it down so the package will not switch to the secondary server. Then normally shut down the live database and do the copying, removing and creation of the new database and we have started the new database with no errors.

Is there another way to do this rather than halting the secondary server? Coz it is not advisable to halt the secondary server everytime we do a cold backup just to make sure the consistency of database datafiles.

For those who have attached their scripts, thank you very much but I would appreaciate if the scripts were in english type.

Thanks again,
Grace

[Mary Grace Inumerable]

Hi John! Good day!

It caught my attention when you said that I can backup my database without halting the package of the service guard? Actually I have tried to normally shut down the database but the problem is that the package switch to the other node. Can you give me a detailed procedure on how to backup the database without the trouble of halting the cluster.

How can I also check that the package
is not monitoring that the database is up?
If it does then how can i modify it?

Thanks again for your reply!

Grace

[John Palmer]

Hi Grace,

It certainly sounds as though you have a monitor script that considers that the package has failed when you close the database.

I can't give you a definite answer without seeing the scripts that you are using to control and monitor your package. Could you post your Serviceguard package control script? It's probably in directory /etc/cmcluster/. There should be a configuration file there as well which will reference it.

Look for the config file first. This will have at the beginning:-

# **********************************************************************
# ****** HIGH AVAILABILITY PACKAGE CONFIGURATION FILE (template) *******
# **********************************************************************
# ******* Note: This file MUST be edited before it can be used. ********
# * For complete details about package parameters and how to set them, *
# * consult the MC/ServiceGuard or ServiceGuard OPS Edition manpages *
# * or manuals. *
# **********************************************************************

Look for a line that starts with 'RUN_SCRIPT', this will be the name of the control script.

Regards,
John

[Justo Exposito]

Hi Mary,

Sorry for the spanish comments, this is the english version of my script for shutdown Oracle, you only need the shutdown_oracle_cluster option.

He allready had the same problem as you and we are using this script every week in order to do a cold backup in our cluster.

Regards,

Justo.

[Justo Exposito]

Hi Mary,

This is the english version of my script for startup Oracle, you only need the startup_oracle_cluster option.

The bkp_orasid file don't need translation.

Best Regards,

Justo.

[Mary Grace Inumerable]

Hi John,

I think you're right because everytime I manually shut down the database the package will switch to the secondary server and the database will automatically started in the secondary server as if nothing happens.

Attached is the package configuration file.

Thanks again,
Grace

[Justo Exposito]

Hi Mary,

I think that is normal that if you shutdown the oracle instance or kill a oracle process the package pass to the other node, if don't do it you have a problem when some oracle process die or fail.

In my cluster this is similar for instance if I shutdown the database by oracle commands or kill the pmon process the package pass to the other node of the cluster. Then if you want to stop the database you must run the cmhaltpkg command.

When you use the cmhaltpkg command the package asociated vg are umounted and you can not use it, then you must use vgchange to get the control over the vg and then you must mount all the lvol's in order to use it in a backup.

This is my case, one night in a week we do a cold oracle backup in our cluster and for this we use cmhaltpkg for all the packages runing in the box, we take the vg control by the vgchange command and then we mount all the lvol's. We do the backup and then we umount the lvol's, we change the vg mode and then we start the same packages in the box.

This is made by bkp_shutdown_oracle, then we made a backup with omniback and in the end we startup the pakages by the bkp_satrtup_oracle script.

If you change the package/monitor definitions when some of your oracle process fail the package don't pass to the other node and the cluster don't run like it must.

Regards,

Justo.

[Mary Grace Inumerable]

Hi Justo,

Attached is the backup script for your reference.

We're trying to populate a test database by copying the live database. Based on the
attached script, I have done the following commands

# Stop the package
cmhaltpkg -v pkg1

# Activate vg02
vgchange -a e /dev/vg02

# Mount vg02
mount /dev/vg02/datavg /datavg

After mounting the volume group "vg02" where the database datafiles are located.
I have copied the oracle live database to a test database instance. Unfortunately, I
cant start the test database. We have done this procedures of populating a test database by
copying the live database files on other systems without MC/ServiceGuard configuration
wherein we can normally shut down the database and do the copying so our DBA is very
much sure that this is not an oracle stuff and it has something to do with the MC/Service Guard.
I got this error message when starting the test database :

ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '/datavg/q1/oradata/flwweb/system01.dbf'

She also told me that maybe there is something wrong with the offline backup script because
the files are not consistent even in simply copying the files. I also checked the logs and it
seems just fine and I dont see any errors during backup. I have also read the backup tape and
I've seen the database files. But she told me that she's referring to the consistency of the
database files.

We have successfully populated a test instance by copying the live database files but the
problem is I have halted the secondary server as in shut it down so the package will not switch
then manually shut down the database in normal mode. then do the copying and i have started
the test database without problems. Is there other way to do this without halting the secondary server? There's a possibility that I'm backing-up inconsistent data files...how can I get rid
of MC/ServiceGuard problem during offline backup. Do you have any tips on how to do
an offline oracle database backup in MC/Service Guard?

Thanks again,
grace

[Steven Gillard_2]

Are you sure that your package halt script is correctly shutting down the database? Serviceguard will kill any processes that are still using the file systems with fuser -k when the package is shut down. If Oracle is still running at this point then you will end up with inconsistent data files as you have seen.

Have a look at your customer_defined_halt_commands function in your package control script which will be in /etc/cmcluster/. Ensure that a "SHUTDOWN IMMEDIATE" is being performed on the Oracle database.

Regards,
Steve

[Justo Exposito]

Hi Mary,

Tell me if I understand well, you want to copy your instance in cluster to other system (box) and then startup in the other system ??ok?.

I was talking with our Oracle DBA and he tell me that you must copy all the data files, the system.dbf, log files, redologs and the control files of your original database by a cold backup.

Then you must to put the files in the other system in the same directoy structure.

If you can not recreate the same directory structure you must to startup your new database with the option mount and do:
alter database bdname rename file 'fich1' to 'fich2';
To change the redologs files to other directory you must to do:
alter database bdname rename file 'fichredo1' to 'fichredo2';
Then do an "alter database open";

Before all of this you must change the init.sid.ora file in order to change the location of your control files.

Another important thing is that the system.dbf file must reside in the same directory (name)as the original database in order to can startup the database.

About your script, I think that it's correct, perhaps the problem is in the oracle side.

Do you copy all the files needed? (The 'get_db_files' sql script is right?)
Do you start the second database with the mount option and do the rename?

Hope this help,

Justo.

[John Palmer]

Hi Grace,

There are two issues here.

First, your Serviceguard configuration doesn't allow you to shut your database without failing the package.

Personally, I don't find this acceptable as there are several circumstances where you would want to do so:
- cold backup
- init file parameter change
- other maintenance.

Even if you monitor for database failure, it's likely that simply aborting it and restarting on the same node is going to fix it. Failing over to the alternate node should be avoided if possible. The monitor code needs to be amended to allow database maintenance to occur. There are many ways to achieve this...

Please post your package control file (/etc/cmcluster/pkg1/pkg1.cntl)and any monitoring script that it calls. This may be configured as:
SERVICE_CMD[n]='...'

Second, it's apparent that your copy of the live database needs recovery because the live database was running when you did the copy.

I don't know exactly what's happening but in the script that you posted, you have the following:

# List the files to be backup in $FILELIST
$ORACLE_HOME/bin/sqlplus system/flwmgr @/oracle/app/oracle/admin/backup/get_db_files flw

immediately followed by cpio of the datafiles to tape.

Now if your sqlplus is working then the database MUST be up and running although I don't know what's starting it.

Regards,
John

[Justo Exposito]

Hi Mary,

If you want that the package don't startup in the second node you must to use the command "cmmmodpkg -d -n secondnode package" before you normal shutdown the instance with the "shutdown immediate" command. I probe it in my cluster an this run fine.

Like Steven tell you probably you have a problem with your shutdown script for the package, then the database didn't close correct and when you try to startup in the other box you obtain the oracle error. This error is because you need to apply logs in order to recovery the database, then the database was copied in inconsistent state (perhaps open, perhaps bad closed).

Hope this help,
Justo.

[Mary Grace Inumerable]

I have attached the ORACLE.sh script of the MC/Service Guard.

pls. take a look of this part of the script.... it says that cmhaltpkg will do a database abort
so maybe that's the reason of file inconsistency. I just want to ask where is the "oracle_
_cntl shutdown" that the author is referring to so I can do a clean shutdown
of the database? Or how can I do that script that will do a clean shutdown of the database
in MC/Service Guard?


---------- script --------------
############################################
# Function: oracle_shutdown_cmds
#
# Use "oracle__cntl shutdown" to do a clean shutdown of the
# database, do not use "cmhaltpkg" to shutdown the database, as
# cmhaltpkg does a database abort and all transactions in memory will be lost.
###############################################################################

function oracle_shutdown_cmds
{
if [[ ${ORA_7_3_X} = yes ]]
then
su oracle -c ${ORACLE_HOME}/bin/svrmgrl <connect internal
shutdown immediate
EOF
else
su oracle -c ${ORACLE_HOME}/bin/sqldba lmode=y <connect internal
shutdown immediate
EOF
fi

if [[ $? != 0 ]]
then
print "Oracle shutdown failed."
else
print "Oracle shutdown done."
fi
}

------------------ script -----------------


thanks again,
grace

[Sanjay_6]

Hi Grace,

If you wish to stop the monitoring of the oracle database, you'll have to comment the line "set -A MONITOR_PROCESSES ..." in the oracle.sh script attached by you. Then you have to halt the packge and retsart the packge. This will stop the monitoring of the packge completely. you have to do the change on all the cluster nodes where this package is supposed to run, so that the packge related files are in sync on all the nodes.

Hope this helps.

Regds