MC/ServiceGuard and application start/stop scripts

[David Niska]

I have an application running a progress database that has a series of start/stop scripts. These scripts perform the start/stop of the application processes and the progress database perfectly when ServiceGuard is NOT running. I have been able to cluster the two nodes this app. runs on, and get the application configured into a package, but when I run the application start/stop scripts, they do not work properly. In fact, the stop script hangs until the DBA manually shuts the database down. Then the stop script will continue to shut everything else down properly and unmount the filesystems. Has anyone ever seen anything like this before?

In summary, the scripts work fine when ServiceGuard is NOT running and don't work when it is. Same exact scripts in both instances.

[Sridhar Bhaskarla]

Hi David,

You will find more details in /etc/cmcluster/your_package_name/your_package_name.ctl.log

The above file name is dependent on your package name.

When you are halting the cluster, make sure you do a tail -f of the file and see the progress. You will get good clues from there.

-Sri

[Ashwani Kashyap]

What does your package control script log says. There should be some information there that can give a clue whats wrong .

THe log file should be in /etc/cmcluster//package_name.cntl.log .

Also look for any errors in syslog.log file .

[Jeff Schussele]

Hi David,

Remember that you don't have a "real" environment when the MC/SG script runs the start/stop scripts. You need to make sure the scripts source whatever env vars you need to run them. Set up all the needed env vars in a file and source that first thing in the start/stop scripts.
I suspect that some var(s) are not being set when MC/SG runs the stop script.

Rgds,
Jeff

[David Niska]

Here is some more information:

I am sourcing in the environment variables at the beginning of the start/stop script. In the package.cntl.log file I get a message that the shutdown is executing which translates to a proshut command for the progress database. That is where it hangs.

I don't get any helpful information from the log. Basically what's happening is that the progress database shutdown script will not shut the database down properly when ServiceGuard is running. These scripts work fine when MCSG is not running. I am currently looking into any environmental issues that could be causing this. Thanks,

Dave

[David Niska]

In looking at the differences between the environments, I found that the shell parameter for MCSG is set to /usr/bin/sh and it is set to /usr/bin/sh without MCSG. Also, the PATH variable does not have all of the paths as when MCSG is not running.

Where does MCSG get it's environment variables?

Dave

[David Niska]

Sorry, that would be /usr/bin/sh when MCSG is running and /sbin/sh when it is not.

Dave

[Oleg Zieaev_1]

Do you have designated user to manage the database?
If so, use this user to control your start and stop procedure.
In your case it will translate to line like this in package.cntl for stop user defined apps:

su - proguser -c ". .profile; proshut > /log/proshut.log 2>&1" &

This approach we use for oracle and it's working like a charm :)

Hope this helps,
0leg

[Oleg Zieaev_1]

One more important thing:
add wait right after that line.

'&' in this case is not needed.
This will wait for DB to be down before continue and umount lvols.

Hope this helps,
0leg

[David Niska]

I already have the "su user -c shutdown script" in my start/stop script and the problem is that the start/stop script hangs and will not shutdown the database or kill the processes.

In order to get the package to shutdown, we need to kill off the stop script process and run the proshut manually.

Let me know if you have any other ideas. Do you know where MCSG gets it's environment variables?

Dave