Simpler Navigation for Servers and Operating Systems - Please Update Your Bookmarks
Completed: a much simpler Servers and Operating Systems section of the Community. We combined many of the older boards, so you won't have to click through so many levels to get at the information you need. Check the consolidated boards here as many sub-forums are now single boards.
If you have bookmarked forums or discussion boards in Servers and Operating Systems, we suggest you check and update them as needed.
General
cancel
Showing results for 
Search instead for 
Did you mean: 

Databases and App server processes

SOLVED
Go to solution
Michael Tully
Honored Contributor

Databases and App server processes

Hi,

I have an Informix database that resides on server A, and dependent application server processes (PeopleSloth) for it that live on server B. Now every week we do system reboots. (please don't ask why!) what I must do, is to make sure the application server processes that
are on server b are shutdown before the database on server A. Here's the hiccup. I have to have the database on server A back on line before the application server processes on server B start.

Suggestions....on the best way to achieve this...

Thanks
Michael
Anyone for a Mutiny ?
17 REPLIES
Steven Sim Kok Leong
Honored Contributor

Re: Databases and App server processes

Hi Michael,

On my site, we have Oracle database server (server A) and SAP application server (server B).

For daily offline database backups, the omniback pre-backup script on server A ssh's into server B, stops the application server before shutting down the database on server A. The omniback post-backup script on server A starts up the database on server A, then ssh's into server B and starts the application server.

I don't know if that is the best method but this is the method I have been using because it is secure (over ssh) and can be automated (based on public key authentication).

Hope this helps. Regards.

Steven Sim Kok Leong
Email: steven@beepz.com. Homepage: https://www.beepz.com
S.K. Chan
Honored Contributor

Re: Databases and App server processes

Hi Michael,
The way I figure .. "single point of trigger" is the approach I would take. Within server B, i a single script ..
a) Shutdown/kill application processes
b) Remsh to server A and shutdown it's DB or reboot.
c) Monitors A's heartbeat, once it's up remsh to it again making sure DB started up fine.
d) Now reboot B.

just a thought ..
Michael Tully
Honored Contributor

Re: Databases and App server processes

Hi Steven and S.K. ,

Thanks for the input thus far. After posting my question I had another thought. We do our reboots at 5AM on a Sunday.

On server A at say 4:30AM we run a script
through cron that terminates the application
server processes.

At 5AM Server B (which has the databases)
gracefully reboots.

At 6AM Server A get rebooted.

I know there is no actual checking but I really
do want something simple. I don't want to go and install 'ssh' and I don't really to open up the systems to run 'remsh'. The part about the Application server being terminated first is the crucial part. (Mind you I wouldn't be fixing it either.... DBA problem....)

Any more thoughts??

Cheers
~Michael~




Anyone for a Mutiny ?
Steven Sim Kok Leong
Honored Contributor

Re: Databases and App server processes

Hi Michael,

After looking at S.K. Chan's reply, I realised that I have misread your query. :P

Two suggestions to add on to what S.K. has already mentioned:
1) use ssh (with public key authentication) instead of remsh for security reasons
2) if the script is part of a shutdown or startup script, it should not timeout too quickly while monitoring the heartbeat of the database server for availability.

Just my 2 cents. Hope this helps. Regards.

Steven Sim Kok Leong
Email: steven@beepz.com. Homepage: https://www.beepz.com
S.K. Chan
Honored Contributor

Re: Databases and App server processes

Ok ..
1) 4.30am A's apps processes terminated
2) 5.00am B (DB srv) reboots
3) 6.00am A (apps srv) reboots
The time between 1 & 2 should be fine assuming 1 is always a guaranteed success, thus 2 need not check if A's processes are terminated before it reboots.
The tricky part is between 2 & 3, how to really make sure the the DB is up and running before you shutdown A (apps srv). I'm no DBA but I think you can run simple SQL queries from A to B just to verify the DB is up before A gets the reboot. If that's possible I would incorporate the SQL script to do the checking before step 3 is executed.
S.K. Chan
Honored Contributor

Re: Databases and App server processes

Hi (again) Michael , I missed the point you made .. "apps srv terminated first is the crucial part" ..
Which means at step 2, before DB srv is rebooted the check to make sure apps processes have been shutdown is more important. Is it suffice to assume if there is no DB connection then the apps processes have actually terminated ? If it is, again, a DBA ahould be able to tell you how to check that.
Michael Tully
Honored Contributor

Re: Databases and App server processes

Thanks again... I'll have a chat with the DBA and see what he wants to do. Ultimately it is his decision, as it's his responsibility. I'll leave this post open until tomorrow morning my time, now 4:45PM and nearly time to go home. (hit the toe! - aussie humour!)

Generous points will be awarded then, just in case someone has a brainstorm!

Cheers
~Michael~
Anyone for a Mutiny ?
Tom Geudens
Honored Contributor

Re: Databases and App server processes

Hi Michael,
Wow, some coordination problem. I would start by getting both the database and application start scripts out of the boots. Next I would use Control-M to accomplish the start/stop coordination. As an alternative (the chances that you have Control-M are rather small) a ITO/Openview automatic action would have to do the trick (which has the benefit over Control-M that you have the ITO tools to do all the checking/heartbeat stuff).

Without those softwares ... tricky. Does Informix have options (like Oracle) to execute external procedures (Operating System commands) ?

Hope this helps ... probably not but you asked for a brain storm :-)

Regards,
Tom

A life ? Cool ! Where can I download one of those from ?
Justo Exposito
Esteemed Contributor

Re: Databases and App server processes

Hi Michael,

Do you have mail between this boxes?

You can define an aliases in the DB server that when receive a mail shutdown the database.

Then if you define that when the app process stop ok send a mail to the DB server and start the reboot process.

This is do it with somethin like that in the aliases file:
stopdb:"| scriptstopdb"

Other posibility is to ftp a control file to the DB server and when run the stop database check if the file is in the server.

Regards,

Justo.
Help is a Beatiful word
harry d brown jr
Honored Contributor

Re: Databases and App server processes


Have server-A shutdown server-B, then have server-A shutdown.

Upon reboot of server-A (all the way up), have server-B reboot and wait UNTIL server-A writes a file to server-B telling it that it can start the application on server-B.

Basically server-B would boot, but before starting the application it would loop until that special file was created by server-A. It would be important that when shutting down server-B to remove that special file. You should put something in that file also, in case server-A and server-B take a hard shutdown where server-B can't remove that special file. Like maybe the PID of the database server - but be careful it could be the same PID as the last time - so maybe some kind of date also? Maybe the early process of booting server-A, before the network is up, you could delete a duplicate of the special file on server-A and also have server-B check not only it's special file, but compare it to server-A's via remsh? Watch out for those deadly embraces - bear hugs!



live free or die
harry
Live Free or Die
A. Clay Stephenson
Acclaimed Contributor

Re: Databases and App server processes

Micheal:

I have an almost identical problem and I'll tell you how I solved it.

Our ERP systems consists of two application servers and a database server. I can't do a hot database backup becuase unless the database data is synchronized with the metadata on the app servers, the backup is useless. My approach is to shutdown the ERP applications first then shutdown the database. I then snapshot everything and bring it back up in reverse order. The backups are then done using the snapshots for a total downtime of about two minutes. I have exactly the same situation you have.

My solution was to write a small Perl program called Omnisync_server.pl that uses bidirectional sockets. Each script then coordinates it activity using a small Perl client called Omnisync_client.pl. Basically, the client makes a request of the server to see if a resource is up or down.

This absolutely solved a tricky problem that I was completely unable to fix using timing loops or other approaches that almost worked well.

If it ain't broke, I can fix that.
Michael Tully
Honored Contributor

Re: Databases and App server processes

Hi,

Tom, unfortunately at this time we don't have a proper scheduling package like control-m to do this type of stuff. We are being lumbered with TNG unicentre (YUK!!) to take care of snmp stuff if and when they get to it.

Informix could run a query to see if the apps server is up or down, but the DBA doesn't want to do that.

Justo, That is a possibility, I'll look into it further.

My Old mate Harry, I don't really want to let
one system have any type of control over the
other. Using 'remsh' is definitely not going to
happen. It took all my energy when I got here to get rid of it....

Clay, Using some perl scripts sounds like a goodbet. I am not a perl monger yet. (Procura is trying to convert me!) If I can I would prefer to use sockets over remote shells. Any chance of posting the scripts, please please.....

Again thanks to everyone....

Cheers
~Michael~
Anyone for a Mutiny ?
A. Clay Stephenson
Acclaimed Contributor
Solution

Re: Databases and App server processes

Hi Michael:

I'll try to put together a set of watered-down scripts tomorrow. You don't even need to remsh, the sockets take care of that for you. You could even have a 3rd server run the server piece, if you like. The good news is that you can simply put the calls to the client inside a shell script so your Perl ability can be quite minimal.
If it ain't broke, I can fix that.
A. Clay Stephenson
Acclaimed Contributor

Re: Databases and App server processes

Okay Micheal:

I spent a few minutes diluting the real version but this should still suit your needs quite well. I attach these as two posting's.

The first is called server.pl. You will need to make an entry in /etc/services (or the NIS services map) for some tcp port. I use omnisync/5554 but you can use anything you like.

At some point, you need to start the server on some host; it could be done as part of a script or in /sbin/rc3.d.

You can invoke it as server.pl -u for full usage but typically all you do is server.pl.

Now here is the concept, this version manages 4 'resources' (Resource_1,Resource_2, ... Resource_4). Each resource has a 'value' (initially 0) and a 'limit' (initially 0).

When you start the server, all that happens is that you have a listener waiting for client requests to manage these resources.

Here's the server.pl piece.

If it ain't broke, I can fix that.
A. Clay Stephenson
Acclaimed Contributor

Re: Databases and App server processes

Now on the client side we do have a number of atomic operations.

1) Set - set a resource value to a given value
2) Get - get the current value of a resource
3) Inc - Increment the current resource value and return the reult
4) Dec - Decrement the current resource value and return the result
5) set_limit - set a resource limit
6) get_limit - get a resource limit
7) stop - kills the server

All the calls require 3 parameters:
tag - the name of the resource (Resource_1 .. Resource_4)
opcode - (Set,Get, ... Stop)
value - a numerical value may be zero

For example to increment the 'Resource_1' value by 2 (host hostname timeout 5 seconds)
VAL=$(client.pl -h hostname -t 5 Resource_1 INC 2)
STAT=$?
if [ ${STAT} -eq 0 ]
then
echo "New Value = ${VAL}
fi

To stop the server:
client.pl -h hostname Resource_1 STOP 0

Basically, you can use these calls to loop until a resource reaches a value. Typicallly, the database startup script might SET a value and the co-operating process might loop getting the Resource value until it reaches 1 with a sleep in the loop.

You can start the server manually and the make interactive calls to client.pl to get an idea of how it works. The client and server can be on the same host but the real beauty of this comes into play when you are synchronizing tasks across many hosts.

Typically, I initially do a query of each host
to set the limit value. For example, when you start and you have 2 app servers, you might want to ping each. If you can ping both, the limit is set to 2 otherwise it is set to 1. This would avoid waiting on a resource that would never become available.

I hope I have been somewhat clear in this; it's really easier than it sounds.

Here's client.pl
If it ain't broke, I can fix that.
Michael Tully
Honored Contributor

Re: Databases and App server processes

Thanks very much Clay!

When I'm in the office on Monday I'll test them out.

You always go the extra step for everyone!

Thanks again!
Cheers
Michael
Anyone for a Mutiny ?
Tom Geudens
Honored Contributor

Re: Databases and App server processes

Clay,
I hope you don't mind that I also "borrowed" the scripts (even though I do have all "ze fancy tools" like ITO and Control-M :-). They are simplicity itself and they work great !
I might have to start studying perl after all ... :-)

< picture of me taking my hat off :-) >

Regards,
Tom
A life ? Cool ! Where can I download one of those from ?