Oracle Enterprise Toolkit Script and Oracle Maintenance

[Steve Bonds]

We have a ServiceGuard configuration running Oracle using the HP-provided Enterprise Toolkit ORACLE.sh script. HP has admonished us not to change this script around since then it becomes "unsupported".

Our problem is when Oracle is configured as a monitored service, if the DBAs bring Oracle down for maintenance (i.e. parameter changes) ServiceGuard detects this as a failure, unmounts everything, and fails over to the standby node.

Is there some way to tell ServiceGuard to ignore the warnings from the monitor portion of the script for a limited period of time? (i.e. some kind of maintenance mode)

I can't figure out how to do this without hacking up that ORACLE.sh script...

[melvyn burnard]

Yes there is a way of doing this, but of the top of my head I cannot remember it ;-{
I do know that this precise subject is covered as one of the topics in the MC/ServiceGuard II course that HP runs at it's education centres.

I will try to look it up again.

[melvyn burnard]

try something like this:




Update the oracle.sh Monitoring Script to Check for Maintenance Flag

1. Edit the oracle.sh monitoring script:

# vi /etc/cmcluster/oracle/oracle.sh

Add the following 16 lines starting at around line 358.

while true
do
if
[ -f /etc/cmcluster/oracle/maint_flag ]
then
while [ -f etc/cmcluster/oracle/maint_flag ]
do
print "In Oracle Maintenance State"
sleep ${MONITOR_INTERVAL}
done
print "Exiting Oracle Maintenance State"
n=0
for i in ${MONITOR_PROCESSES[@]}
do
MONITOR_PROCESSES_PID[$n]=`ps -fu oracle | awk `/'${i}'/ { print $2 }'`
(( n = n + 1 ))
done
else
for i in ${MONITORED_PROCESSES_PID[@]}
do
. . .
done
sleep ${MONITOR_INTERVAL}
fi
done

2. Distribute the updated monitoring script to other node in cluster:

# ftp second_system
> put /etc/cmcluster/oracle/oracle.sh
3. Restart the oracle package to use the new monitoring script:

# cmhaltpkg oracle
# cmrunpkg oracle
# cmmodpkg -e oracle (package got disabled by cmhaltpkg command)

4. Test the maintenance state by creating the maint_flag and halting Oracle:

# touch /etc/cmcluster/oracle/maint_flag
# su - oracle
$ svrmgrl
SVRMGR> connect internal
SVRMGR> shutdown normal
SVRMGR> exit
$ exit

5. Verify that MC/ServiceGuard still believes the package to be up:

# cmviewcl -v -p oracle

6. Verify that the package is in a maintenance state by checking the log file:

# tail -f /etc/cmcluster/oracle/oracle.cntl.log

7. Take the package out of maintenance mode:

# su - oracle
$ svrmgrl
SVRMGR> connect internal
SVRMGR> startup PFILE=/ora1/dbs/initoracle.ora
SVRMGR> exit

8. Remove the maintenance flag:

# rm /etc/cmcluster/oracle1/maint_flag

9. Verify package is no longer in maintenance mode:

# tail -f /etc/cmcluster/oracle/oracle.cntl.log



There are a lot of other nice things to learn on the course.
Hope that helps

[Steve Bonds]

Thanks for the detailed answer.

Unfortunately, I was hoping to avoid hacking up the ORACLE.sh script if at all possible.

Perhaps it's not possible.

[melvyn burnard]

when you say:
"HP has admonished us not to change this script around since then it becomes "unsupported". "

you are aware that these scripts are supplied as examples only, and you are REQUIRED to change them to configure them for your own environment?
Our normal view is that yes you buy the ECMT, and we will enhance and ad to these over time, but as they are supplied as examples only, the runing scripts are seen as if the were normal customer written scripts, i.e. we do not support them for fixes.

Hope I have not confused you.

[Steve Bonds]

We've been told that changing environment variables is OK, but to keep the rest of it pristine.

It's killing us to do so. I'm just about ready to toss the whole thing into version control and start hacking away... ;-)

[Sanjay_6]

Hi Steve,

Like Melvyn has said, the scripts are provided as example onlt which helps you in running an oracle database with much of a problem. But to exploit different scenarios and to do what you want to do, you havb to modify the script to suit your requirement. That won't be hacking. It is customisation of the a sample script.

Hope this helps.

Regds