Offline backup of Oracle Database in MC/ServiceGuard

[Steven Gillard_2]

The "shutdown abort" is definitely the reason for the inconsistency. The comment is saying you can run the script itself with the "shutdown" argument to correctly shut down the database (ie "ORACLE.sh shutdown"). The problem with this, however, is it will cause the package to fail over to the alternate node because the monitor will fail, so its not going to help you get your backup.

The quickest way for you to get your backup is going to be to start then stop Oracle without serviceguard monitoring:

1. # cmhaltpkg pkg_name
2. # vgchange -a e /dev/vg_name
3. Mount the appropriate file systems
4. As Oracle, startup the database using svrmgrl. Media recovery will be automatically performed.
5. Now shutdown the database in svrmgrl with "shutdown immediate".

Now that the database has been shutdown correctly you can take a consistent copy of the database files.

To assist with this type of activity, I use a file /etc/cmcluster//ADMIN_STOPPED, which my monitor script checks for existence before reporting package failure. If this file exists the monitor will just sleep for 5 seconds in a loop until the file is gone, then it will resume normal monitoring.

This allows me to do a "quick restart" of the application without the package being halted and the volume group being deactivated. My procedure then is:

# touch /etc/cmcluster//ADMIN_STOPPED
#

This is something you might want to look at implementing in your control script.

Regards,
Steve

[John Palmer]

Hi Grace,

What I would do is change your script so that you have the ability to switch monitoring off and on while the package is running.

Probably the easiest way is to use a marker file and get the script to stop monitoring if that file exists and start monitoring again when the file is removed.

Because you are likely to close and restart the database in the no monitor period, the function 'monitor_processes' will have to be called again to refresh its PID list.

In the following code I have used '/etc/nomonitor.flw' as the marker file. First amend your monitor loop to return if the file exists:-

while true
do
for i in ${MONITOR_PROCESSES_PID[@]}
do
kill -s 0 ${i} > /dev/null
if [[ $? != 0 ]]
then
print "\n\n"
ps -ef
print "\n *** ${i} has failed. Aborting Oracle. ***"
set -m
nohup ${0} fault & # The script calls itself with the fault option.
set +m
sleep 999999
fi
done
sleep ${MONITOR_INTERVAL}

######## new code starts here ########
if [[ -f /etc/nomonitor.${SID_NAME} ]];
then return
fi
######## new code ends here ######
done


Then change the code that calls function 'monitor' to the following:-

monitor)
while true
do
monitor_processes
while [[ -f /etc/nomonitor.${SID_NAME} ]];
do
sleep 10
done
done
;;

Make sure that your amended script is in place on all servers in your cluster then restart your package to enable it.

In order to stop monitoring simply:-
touch /etc/nomonitor.flw

You will then be able to shut your database down without the package being failed. After you have restarted the database, simply:-
rm /etc/nomonitor.flw
to resume monitoring.

Regards,
John

[Mary Grace Inumerable]

Hi Sanjay,

Thanks for the help! After uncommenting the
line set -a MONITOR PROCESSES...in the
ORACLE.sh script, I can now shutdown the
database in normal mode without switching to
the second node. I have successfully populated
the live database while in that state so I
think the backup script will work fine.

But still I have one problem. How can I bring back that set -a MONITOR PROCESSES... after my backup without manually editing(vi) the ORACLE.sh script? Can I just run it the
command mode? can I just add it in my script
so everything will be in normal for the MC/SG
after the backup?

I have also attached the new backup script for
your reference.

Thanks again,
Grace

[Justo Exposito]

Hi Mary,

I think that the John Palmer reply is the best, because if you disable the monitor permanent if your database fail then don't restart in the other node. It's better to have a file marker and only when the file exist the monitor don't run.

Other important thing is change the package control script in order to shutdown the database properly when you use cmhaltpkg, with the option shutdown immediate and not abort, because if you use shutdown abort you are killing the database process and then you have an inconsistent database (talk with your DBA). This is the problem that you have with your backups and your script.

If you change the control file, when you use cmhaltpkg the database is closed with shutdown immediate your script is valid as you have at the start of this issue (you must have in mind that the normal way to stop a package is using cmhaltpkg).

Regards,

Justo.

[Mary Grace Inumerable]

I just want to thank all of you who take time
to reply to my questions. I really really
appreaciate all the effort and suggestions.

This is also to inform you that I have
successfully made a script for the offline
database backup. The script will disable the MC/SG package monitoring, then normal shutdown
of the database, backup of the database, normal
startup of the database and lastly enable the
MC/SG package monitoring after backup. I have
also tried populating the database while the
backup is going and it was also successful.

Again, thanks for all the help!

Grace