Databases and App server processes

[harry d brown jr]

Have server-A shutdown server-B, then have server-A shutdown.

Upon reboot of server-A (all the way up), have server-B reboot and wait UNTIL server-A writes a file to server-B telling it that it can start the application on server-B.

Basically server-B would boot, but before starting the application it would loop until that special file was created by server-A. It would be important that when shutting down server-B to remove that special file. You should put something in that file also, in case server-A and server-B take a hard shutdown where server-B can't remove that special file. Like maybe the PID of the database server - but be careful it could be the same PID as the last time - so maybe some kind of date also? Maybe the early process of booting server-A, before the network is up, you could delete a duplicate of the special file on server-A and also have server-B check not only it's special file, but compare it to server-A's via remsh? Watch out for those deadly embraces - bear hugs!



live free or die
harry

[A. Clay Stephenson]

Micheal:

I have an almost identical problem and I'll tell you how I solved it.

Our ERP systems consists of two application servers and a database server. I can't do a hot database backup becuase unless the database data is synchronized with the metadata on the app servers, the backup is useless. My approach is to shutdown the ERP applications first then shutdown the database. I then snapshot everything and bring it back up in reverse order. The backups are then done using the snapshots for a total downtime of about two minutes. I have exactly the same situation you have.

My solution was to write a small Perl program called Omnisync_server.pl that uses bidirectional sockets. Each script then coordinates it activity using a small Perl client called Omnisync_client.pl. Basically, the client makes a request of the server to see if a resource is up or down.

This absolutely solved a tricky problem that I was completely unable to fix using timing loops or other approaches that almost worked well.

[Michael Tully]

Hi,

Tom, unfortunately at this time we don't have a proper scheduling package like control-m to do this type of stuff. We are being lumbered with TNG unicentre (YUK!!) to take care of snmp stuff if and when they get to it.

Informix could run a query to see if the apps server is up or down, but the DBA doesn't want to do that.

Justo, That is a possibility, I'll look into it further.

My Old mate Harry, I don't really want to let
one system have any type of control over the
other. Using 'remsh' is definitely not going to
happen. It took all my energy when I got here to get rid of it....

Clay, Using some perl scripts sounds like a goodbet. I am not a perl monger yet. (Procura is trying to convert me!) If I can I would prefer to use sockets over remote shells. Any chance of posting the scripts, please please.....

Again thanks to everyone....

Cheers
~Michael~

[A. Clay Stephenson · Accepted solution]

Hi Michael:

I'll try to put together a set of watered-down scripts tomorrow. You don't even need to remsh, the sockets take care of that for you. You could even have a 3rd server run the server piece, if you like. The good news is that you can simply put the calls to the client inside a shell script so your Perl ability can be quite minimal.

[A. Clay Stephenson]

Okay Micheal:

I spent a few minutes diluting the real version but this should still suit your needs quite well. I attach these as two posting's.

The first is called server.pl. You will need to make an entry in /etc/services (or the NIS services map) for some tcp port. I use omnisync/5554 but you can use anything you like.

At some point, you need to start the server on some host; it could be done as part of a script or in /sbin/rc3.d.

You can invoke it as server.pl -u for full usage but typically all you do is server.pl.

Now here is the concept, this version manages 4 'resources' (Resource_1,Resource_2, ... Resource_4). Each resource has a 'value' (initially 0) and a 'limit' (initially 0).

When you start the server, all that happens is that you have a listener waiting for client requests to manage these resources.

Here's the server.pl piece.

[A. Clay Stephenson]

Now on the client side we do have a number of atomic operations.

1) Set - set a resource value to a given value
2) Get - get the current value of a resource
3) Inc - Increment the current resource value and return the reult
4) Dec - Decrement the current resource value and return the result
5) set_limit - set a resource limit
6) get_limit - get a resource limit
7) stop - kills the server

All the calls require 3 parameters:
tag - the name of the resource (Resource_1 .. Resource_4)
opcode - (Set,Get, ... Stop)
value - a numerical value may be zero

For example to increment the 'Resource_1' value by 2 (host hostname timeout 5 seconds)
VAL=$(client.pl -h hostname -t 5 Resource_1 INC 2)
STAT=$?
if [ ${STAT} -eq 0 ]
then
echo "New Value = ${VAL}
fi

To stop the server:
client.pl -h hostname Resource_1 STOP 0

Basically, you can use these calls to loop until a resource reaches a value. Typicallly, the database startup script might SET a value and the co-operating process might loop getting the Resource value until it reaches 1 with a sleep in the loop.

You can start the server manually and the make interactive calls to client.pl to get an idea of how it works. The client and server can be on the same host but the real beauty of this comes into play when you are synchronizing tasks across many hosts.

Typically, I initially do a query of each host
to set the limit value. For example, when you start and you have 2 app servers, you might want to ping each. If you can ping both, the limit is set to 2 otherwise it is set to 1. This would avoid waiting on a resource that would never become available.

I hope I have been somewhat clear in this; it's really easier than it sounds.

Here's client.pl

[Michael Tully]

Thanks very much Clay!

When I'm in the office on Monday I'll test them out.

You always go the extra step for everyone!

Thanks again!
Cheers
Michael

[Tom Geudens]

Clay,
I hope you don't mind that I also "borrowed" the scripts (even though I do have all "ze fancy tools" like ITO and Control-M :-). They are simplicity itself and they work great !
I might have to start studying perl after all ... :-)

< picture of me taking my hat off :-) >

Regards,
Tom