Shutdown script spanning more than one server

[A. Clay Stephenson]

Hmmm I am now testing this on the PRD environment. I am receiving some mixed results.

Check to see if the server.pl is actually running on your "boss" server.

RSLT=$(client.pl -P 7777 Resource_1 SET 0)
This will only work is server.pl and client.pl are running on the same host otherwise a "-h boss" argument is required.

Another "gotcha" is that if Test and Production backups ever run simultaneously then you need to choose different ports (e.g. -P 7777 and -P 7778) for Production and Test respectively so that "Resource_1" for Production doesn't interfere with "Resource_1" for Test. If server.pl is run of different hosts then this doesn't matter.

[Dan Sargis]

I did what you recommended in regards to using the "INC" to manage the DB backup scripts.


Also, the server.pl and client.pl(s) ARE running on the application server with the client.pl(s) also running on the DB server. To ensure which server the client.pl is pointed to I explictly use the -h parameter each time for client.pl.

I am continuing to test this in PRD. Instead of actually issuing the Starts and Stops of the Apps and DBs (bad thing during work hours), I am issuing echos to label where in the scripts the process is at any given time. I am now getting the following series of errors:

titan01:/usr/contrib/bin# control1.sh
control1.sh BEFORE
Socket could not be created. Connection refused
Socket could not be created. Connection refused
LSPRD DATABASE BACKUP complete
CBSPRD DATABASE BACKUP complete
Socket could not be created. Connection refused
Socket could not be created. Connection refused
control1.sh AFTER
titan01:/usr/contrib/bin#

Again, I am wondering if the semaphores are getting set to 0 properly and also HOW I ensure that they are indeed getting set back.

Please indulge me a take a look at my scripts I have uploaded. I the zip file I have my control script, the two application scripts and the two DB scripts.

Thanks again.

[A. Clay Stephenson]

Do a "netstat -an | grep 7777". I suspect that you are going to find that 1) either you already have a server.pl running and using port 7777 or 2) another listener is already using that port. It's important that you find an unused port. You should call server.pl only once initially. It would be a good idea to check ${?} after that call to make sure that it started. I didn't put all the error checking in; I intentionally left that as an exercise.

You should do something like this:

typeset SYNCPORT="omnisync"
typeset SYNCHOST="boss"
typeset -i SSTAT=0

server.pl -h ${SYNCHOST} -p ${SYNCPORT}
SSTAT=${?}
if [[ ${SSTAT} -ne 0 ]]
then
echo "Sema4 server failed to start; status ${SSTAT}" >72
exit ${SSTAT}
fi


I used 7777 merely as an example; you must find an unused port for each instance of server.pl running on a given host. If both your production and test versions are running server.pl on the same host then they must use different port number.

You should also note that both server.pl and client.pl can also use the /etc/services file so that you could use -p omnisync rather than -P 7777 assuming that you have added an "omnisync tcp/7777" entry in /etc/services.


The good news if you successfully start server.pl, you are assured that all the sema4 values and limits are initialized at zero.

[Dan Sargis]

netstat -an | grep 7777 returns nothing:

titan01:/usr/contrib/bin# netstat -an | grep 7777
titan01:/usr/contrib/bin#

[Dan Sargis]

Also, it appears that the error message originates from both appl1.sh and appl2.sh.

Please, if you would, check my "Resource_x" specifications. Do you see any incompatibities between the control1.sh and applx.sh scripts?

[Dan Sargis]

I think that I might be a getting a little crossed up on my resources and servers.

This all works fine in my TST environment. The only thing that's really changed is what am passing in "-h" in various places AND the fact that I split up the DB stops and starts.

[Dan Sargis]

OK, I've gotten a little further with my scripts. I need for you to take a look at the following which runs in the DB backup script. As you can see I want both apps to finish backup and each INC RSLT1 by 1 such that RSLT1 becomes 2 and breaks out of the loop. The echo you see which returns the current value of RSLT1 is coming back as "2". Why would it not break out of the loop and continue?:

RSLT1=$(client.pl -h titan01 -P 7777 Resource_1 GET 0)
while [[ ${RSLT1} -lt 2 ]]
echo "CBS STAT1: ${STAT1} RSLT1: ${RSLT1} "
do
sleep 10
echo CBSPRDbackup.sh WAIT
RSLT1=$(client.pl -h titan01 -P 7777 Resource_1 GET 0)
STAT1=${?}
done



CBSPRDbackup.sh WAIT
LSPRDbackup.sh WAIT
LS STAT1: 0 RSLT1: 2
CBS STAT1: 0 RSLT1: 2

[A. Clay Stephenson]

You have a heap big logical error:

RSLT1=$(client.pl -h titan01 -P 7777 Resource_1 GET 0)
while [[ ${RSLT1} -lt 2 ]]
echo "CBS STAT1: ${STAT1} RSLT1: ${RSLT1} "
do
sleep 10
echo CBSPRDbackup.sh WAIT
RSLT1=$(client.pl -h titan01 -P 7777 Resource_1 GET 0)
STAT1=${?}
done

should be something like this:

RSLT1=$(client.pl -h titan01 -P 7777 Resource_1 GET 0)
while [[ ${RSLT1} -lt 2 ]]
do
echo "CBS STAT1: ${STAT1} RSLT1: ${RSLT1} "sleep 10
echo CBSPRDbackup.sh WAIT
RSLT1=$(client.pl -h titan01 -P 7777 Resource_1 GET 0)
STAT1=${?}
done

[Prashant Zanwar_4]

off course it would be application specific and DB specific script and you can place it in same location on all hosts, so that if you are calling it from a host, it will be easier to make it consistently work..
Usuauly I got in situation like this & oracle DBA folks provide apps/DB stop scripts which usually stop apps or DB on clustered or application nodes etc..
Ideas here can be many, but surely need to confine to your need..
I had a EMC snap backup in my setup, wherein we used to run a kshell script, which used to do all stuff like, look for apps is running, db is running on particular host, if yes, call DB /apps script to stop it, then export the snap VG's, do a sync, import it back.. activate VG's. Once backup is done, call same script and start DB etc..it was possible to get the status looking at pmon / or using wget for oracle apps application..

just tried putting as thought it can help somewhere

thanks
Prashant

[Dan Sargis]

Yes, heap big error indeed. That fixed it sir.

I will be testing PRD tonight and let you know how it turns out.

Thank you again for your many excellent replies to my post.