Shutdown script spanning more than one server

[Dan Sargis]

Please take a quick look at the attached DB shutdown and startup I implemented using your client.cl code example from the "Coordinating Multi-Host Backups Using Perl Sockets" article.

I am receiving "Socket could not be created Invalid argument." when running the client.cl script I adapted. Attached is a zip file which contains that file plus the DB shutdown and startup scripts. (FYI, I do not get that error message on the application client.cl script I adapted).

Additional FYI, the application and DB run on the same server in this case as I am trying this in a TST environment first.

One more question once the above issue gets resolved:

There are two databases that I need stop/start before the application comes back up. How might I go about implementing that with your scripts?

Thanks again so much, in advance...

[Dan Sargis]

I just attached the server.pl script file and the adapted application server script file in a zip file just in case you wound up asking for them.

Thanks!

[A. Clay Stephenson]

When I looked at your client.clLSTST.sh script this line jumped out at me:

RSLT=$(client.pl -h boss -P 7777 Resource_1 GET 0)

when all the other references in the file are to a host 'atlas03'.

I also suggest that you add a "shebang" line to every one of your shell scripts
#!/usr/bin/sh

so that there is no ambiguity about what shell you wish to execute under.

I would also replace all the references to "atlas03" and port 7777 with variables so that these values are set in exactly one place in each script. This ensures that you always "zig" or always "zag". In your case with hosts "boss" and "atlas03" in your script you both "zigged" and "zagged" unintentionally.

If you need to add another layer of processes to wait for that run independently then all you do is a logical AND waiting for two distinct sema4's to reach a given value.

Something like this:

#!/usr/bin/sh

typeset -i RSLT1=0
typeset -i RSLT2=0
typeset -i STAT1=0
typeset -i STAT2=0

typeset SERVER="atlas03"
typeset -i PORT=7777

typeset DB1="Resource_1"
typeset DB2="Resource_2"
typeset -i STAT=0
typeset -i STAT2=0

RSLT1=$(client.pl -h ${SERVER} -P ${PORT} ${DB1} GET 0)
STAT=${?}
RSLT2=$(client.pl -h ${SERVER} -P ${PORT} ${DB2} GET 0)
STAT2=${?}
while [[ ${STAT} -eq 0 && ${RSLT1} -lt 2
&& ${STAT2} -eq 0 && ${RSLT2} -lt 2 ]]
do
sleep 10
RSLT1=$(client.pl -h ${SERVER} -P ${PORT} ${DB1} GET 0)
STAT=${?}
RSLT2=$(client.pl -h ${SERVER} -P ${PORT} ${DB2} GET 0)
STAT2=${?}
done

Note that I replaced the more or less meaningless "Resource_1" and "Resource_2" with (hopefully) more meaning mnemonics ${DB1} and ${DB2}.

[A. Clay Stephenson]

Ooops, I'm stupid. If you were waiting on two separate databases to change state then you would as long as either sema4 were less than 2.

WRONG:
while [[ ${STAT} -eq 0 && ${RSLT1} -lt 2
&& ${STAT2} -eq 0 && ${RSLT2} -lt 2 ]]
do

RIGHT:
while [[ ( ${STAT} -eq 0 && ${STAT2} -eq 0 ) && ( ${RSLT1} -lt 2 || ${RSLT2} -lt 2 ) ]]
do
....
....
done

Note the extra ()'s so that the AND's and OR's take the correct precedence. The idea is that if either execution status returns a non-zero result the while will termninate BUT it will not terminate until BOTH sema4's reach at least 2. Of course, these sema4 are set in another co-process.

[Dan Sargis]

Thanks for the last update. I modified my script accordingly.

The client.cl script for the databases will not come back to a Unix command prompt. I'm attaching it (client.cl.ORACLE.sh and would like for you to take a look to see why that might be the case.

Also, I have yet to implement the "snapshot" concept you suggested. I still need to "round up" the disk needed for buffer. Hence, in the interim I would like to continue using my current disk to disk copy scripts for cold backup. In the attached client.cl.ORACLE.sh script, how would you recommend that I have both DBs perform their respective disk copy scripts simultaneously? Obviously this will not be a concern once we implement snapshots.

Thanks again.

[A. Clay Stephenson]

You are beginning to want too much sugar for a dime but ...

I suggest that you add some probes:

while [[ ( ${STAT} -eq 0 && ${STAT2} -eq 0 ) && ( ${RSLT1} -lt 2 || ${RSLT2} -lt 2 ) ]]
do
echo "STAT: ${STAT} STAT2: ${STAT2} RSLT1: ${RSLT1} RSLT2: ${RSLT2}"
sleep 10
RSLT1=$(client.pl -h ${SERVER} -P ${PORT} ${DB1} GET 0)
STAT=${?}
RSLT2=$(client.pl -h ${SERVER} -P ${PORT} ${DB2} GET 0)
STAT2=${?}
done


However, I see what is wrong; you are missing the fundamental point of these sema4 routines. Note that you are looping doing sema4 GET operations but the SET operation doesn't occur until after the loop terminates. You are waiting on an event which will never occur.

The whole idea is that you start a set of simultaneous processes and while a process is looping waiting for a sema4 to reach a certain value and another process (possibly on another host) actually sets the sema4.

It would be normal for some code that is looping waiting on a sema4 to reach some value to then see that value reached. That loop would finish and it would then be normal for this process to alter the value of ANOTHER sema4 that presumably some other process is looping on.

[Dan Sargis]

Sorry about the excessive "sugar". I hope the "10s" for each of your excellent responses eases things a bit. :-)

What you say makes sense about the loop. But I just adapted your code for the database server verbatim save for the host name specifics.

[A. Clay Stephenson]

Go back and look at the examples again in the article.

#3) Code to be executed on the database server; it is started at
# the same time as the #2) above.
# Loop until ERP applications are down indicated by Resource_1 being set to 2.
RSLT=$(client.pl -h boss -P 7777 Resource_1 GET 0)
STAT=${?}
while [[ ${STAT} -eq 0 && ${RSLT} -lt 2 ]]
do
sleep 10
RSLT=$(client.pl -h boss -P 7777 Resource_1 GET 0)
STAT=${?}
done
if [[ ${STAT} -eq 0 ]]
then #shutdown database and snapshot
shutdown_db.sh
create_db_snapshots.sh
startup_db.sh
STAT=${?}
if [[ ${STAT} -eq 0 ]]
then # set Resource_2 semaphore to 1
RSLT=$(client.pl -h boss -P 7777 Resource_2 SET 1)
STAT=${?}
fi
fi
exit ${STAT}

------------------------------

Note that the database co-process loops waiting for Resource_1 to reach 2 but it then sets Resource_2. In your case, you might simply drop the two processes into the background via '&'. The important point is that two co-processes are running simultaneously waiting on different events (ie different semaphores). In this case, Resource_2 is what the other process is waiting on. I didn't copy and paste 2) (the other process) but the key to understanding this is to run 2) and 3) simultaneously (either mentally or running client.pl manually). In my test environment, all of this occurs on the same host but in production these events occur on 5 hosts; nevertheless the same code works in both environments unchanged -- with 1 minor exception. I simply use different tcp ports for production and test.

[Dan Sargis]

Excuse my persistence ...(ignorance?) but I don't see in the above script where "RSLT" ever gets past 0. There is in no INC or SET that I observe which would take it out of that loop:

while [[ ${STAT} -eq 0 && ${RSLT} -lt 2 ]]
do
sleep 10
RSLT=$(client.pl -h boss -P 7777 Resource_1 GET 0)
STAT=${?}
done

[A. Clay Stephenson]

Exactly what I was trying to explain. The idea is that ANOTHER process running at the same time is altering the semaphore value. To the process looping on the semaphore, it appears that some outside force changed the value. Note that that outside force could be another client.pl running on a totally separate host.

[Dan Sargis]

Ah, I see. Duh.

Yes, with different hosts the scripts snap right in. With the two DBs I have to add a "Resource_3".

Let me work with this a bit and I'll get back to you.

Thanks again for your patience.

[A. Clay Stephenson]

Okay, I've attached a fairly complete example using 2 application servers, 1 database instance, and a backup manager, 'boss'. There are 3 scripts and 3 sema4's.

[Dan Sargis]

OK yet another stupid question.

How do I view the attachments for this thread?

[Dan Sargis]

OK ne'er mind. Got it.

[Dan Sargis]

Great script, thanks again!

I'm now rethinking the backup methodology I want to use for the DB's. Rather than sticking the old disk copy script in there I would like to use snapshotting. However, I would like to implement this using the EVA provided business copy technology thus taking advantage of our SAN capabilities. However, we have no documentation on this. Could you possibly either a) Point me to documentation which would describe how to do this or b) if you have similar knowledge to the JFS snapshotting that you can impart relative to SAN business copying, please do so.

Thank you.

[A. Clay Stephenson]

Sorry, I don't run EVA's although the concepts are essentially the same.

[Dan Sargis]

OK , then back to the first question, could you point me in the direction of documentation that might assist me in this effort?

[A. Clay Stephenson]

Start with man vgchgid. After you split off your copies, essentially you will run the vgchgid command on the copies and then vgimport those LUN's as a new VG. An ioscan will be required to detect the new disks and insf will need to be executed to create the new device nodes. One approach that you could take is that these copies might be imported onto an entirely different host -- this is one area where the hardware hased snapshots have an advantage over the vxfs snapshots. If you were running raw devices then BCV's also make sense as vxfs snapshots would not apply.

You are adding some level of additional work
by not using vxfs snapshots and my experience has been that the performance impact while the backups are running is almost imperceptible.

[Steven E. Protter]

How do I view the attachments for this thread?

Depends on your browser configuration.

Under windows, most browers will use the file extension name association.

If the attachment is .doc, word will open it.

If its a null or unknown attachment, save it to local disk and then use your best guess, based on the post describing it to open it up.

For this thread vi on Unix or wordpad on Windows will do the trick.

SEP

[Dan Sargis]

I've been working with the "super" script you provided and am getting a little confused. I am still working on the TST server which is the apps and databases on it (in contrast to PRD where they are distributed across three servers). Will this script you just provided work on a single server as well or are multiple servers assumed?

[A. Clay Stephenson]

Yes, in fact, in my test environment everything is on one server although production is spread over 5 boxes.

I did find one small syntax error in control.sh:

RSLT=(client.pl -P 7777 Resource_3 GET 0)
should be:
RSLT=$(client.pl -P 7777 Resource_3 GET 0)

In my case, the control shell script which runs everything else resides on my DataProtector Cell Server but all the other scripts are remsh'ed (from control.sh) on the test environment server.

In the case of production, control.sh still runs on the DP Cell Server but now the remsh's are executed on 5 different servers.

The semaphore client and server scripts don't care if the activity is spread over one box or many.

If your "control.sh" and all the other scripts run on just 1 box then you would not use remsh but simply locally execute shell scripts although in both cases the scripts would be run as background processes.

[Dan Sargis]

I just wanted to let you know that the shutdown and startup scripts using semaphores you set me up with are going to work out nicely. I am testing them in my TST environment right now. I'll let you know what happens the first time through the PRD environment.

Thanks again.

[Dan Sargis]

OK, one more question ... ( just one more teaspoon of sugar for the "dime").

I need for my two databases to be backed up simultaneously AFTER application shutdown and BEFORE application startup.

Similar to the two identical shell scripts "appl1.sh" and "appl2.sh", may I simply duplicate the db.sh, (inserting the proper shutdown, snapshot and startup scripts) to backup at the same time? Then I would assume that the applications would not be started till the last database backup had completed.

Does this sound accurate?

[Dan Sargis]

Hmmm I am now testing this on the PRD environment. I am receiving some mixed results.

When starting the "appl1.sh" script, I am getting Socket could not be created. Connection refused." Is that because sema4's are not being reset to 0? I issued the following commands in a separate shell:

RSLT=$(client.pl -P 7777 Resource_1 SET 0)
RSLT=$(client.pl -P 7777 Resource_2 SET 0)
RSLT=$(client.pl -P 7777 Resource_3 SET 0)

Should this not set the semaphores back to 0?

If not, I'm unclear on how that takes place.

[A. Clay Stephenson]

Similar to the two identical shell scripts "appl1.sh" and "appl2.sh", may I simply duplicate the db.sh, (inserting the proper shutdown, snapshot and startup scripts) to backup at the same time? Then I would assume that the applications would not be started till the last database backup had completed.


I would approach it like this:

In the current version of db.sh there is an explicit SET 1 operation to indicate that the database has restarted and the applications wait until that sema4 reaches 1; you should modify the db.sh (that will now run twice simulataneously either on the same host but different instances or on different hosts) but instead of doing a SET 1 you should do an INC 1 and have the application server scripts wait until the value reaches 2. Each db instance will add 1 to whatever the current value is.