UPS shutdown scripts

[Matthew Couper]

I've been meaning to get this setup correctly for a while and finally have a bit of time to go over all of this. I know the basics of the ups_config and how it performs the shutdown, but I need to verify if this will 'properly' shutdown the running databases (improper database shutdown recovery is simple but a time consuming pain). I don't find anything in the normal shutdown to do this.

With the shutdown_timeout_mins, is that time how long until the shutdown -h is executed? So if it's set for 5 min, after 5 minutes is when it starts the shutdown...correct? So need to account for the shutdown time in the UPS battery life span?

Is it the /etc/local/powerfail file that the custom shutdown scripts to shutdown the databases should be placed?

This is a N4000 running HP-UX 11.0(64)

Let me know if any of this isn't very clear, at least I know what I mean =)

[Pete Randall]

Steve,

The ups_mond will initiate a graceful shutdown via the shutdown command. As long as you're DB shutdown is part of your normal shutdown process, you should be fine. Shutdown_timeout_mins is the amount of time that is allowed to for the normal shutdown. If it hasn't happened in the default 5 minutes, a reboot -h is initiated. This is a quick, ungraceful shutdown that won't do any of your normal shutdown scripts.


Pete

[Matthew Couper]

Pete,
Thanks that helps bit.

So once powerfail is detected it begins the powerfail script? And after 5 minutes it 'gives up' on that and goes into a quick shutdown?

That doesn't seem right to me, since power failure only lasts less then a minute most of the time. Would it be reasonable to put a sleep in the powerfail and extend the timeout accordingly? Or just take it as is and hope I'm on-site to perform a DB shutdown if it looks like it might last more then 5 minutes?

[Pete Randall]

Steve,

The shutdown_delay_mins paramater specifies how long, after the powerfail is detected, that ups_mond will wait before initiating the normal graceful shutdown. A 10 minute delay should ensure that your little one minute outages don't shut down the system.


Pete

[Darren Prior]

Hi,

Here's a sequence of events for ups_mond:

a power failure is detected by the ups and signalled to ups_mond. If the power stays off for shutdown_delay_mins then shutdown -h is run. If the power comes back on before shutdown_delay_mins then the server carries on running.

After shutdown_timeout_mins the ups power will switch off. This is where you need to account for the time it takes your system to shutdown, 'cos the power's gonna go off at this point whether the box is safely down or not!

I've not used the powerfail file you mention, so unfortunately I can't comment on its' usage. I would have thought that your normal shutdown scripts should shutdown the databases properly - otherwise you would surely need manual intervention everytime you need to perform a shutdown. Do you have S/K scripts to automatically start/stop the database?

Check the man page for ups_mond and ups_conf for more info, including the limits and defaults for these values.

regards,

Darren

[Matthew Couper]

Alright, let me get this right (it must be Monday morning...)

the shutdown_delay_mins is how long you give the system before it starts a graceful shutdown and shutdown_timeout_mins is the time (after) the delay has ended and then actually performs the shutdown -h?

So if the delay is 10 minutes, it will wait for 10 minutes THEN start the powerfail.cfg which runs the powerfail script for user commands. And after 5 minutes it runs shutdown -h?
And the /etc/powerfail is where I would place the shutdown scripts for the DBs.

This is assuming the settings of
shutdown_delay_mins 10
shutdown_timeout_mins 5

[Matthew Couper]

I think I got it...
using the example of
shutdown_delay_mins 10
shutdown_timeout_mins 5

1) The UPS looses power

2) System watches power for 10 minutes. If power is restored it stops monitoring and resumes normal activity. If power is still lost after 10 minutes continue to 3

3) System runs /etc/powerfail.cfg which runs /etc/local/powerfail for custom scripts

4) During this time the system has 5 minutes to finish with shutdown

5) The 5 minutes pass and the UPS is SHUT OFF regardless if the system is still shutting down or not.

6) Wait for power to restore and hope things were shutdown before power was removed.


So the shutdown_timeout_mins should be set to
a) how long your UPS can run everything
b) the time a shutdown -h takes to execute including any custom scripts needed to run.

[Darren Prior · Accepted solution]

Hi Steve,

Your latest post is correct - if you replace step 3 with "a shutdown -h is performed."

Your calculation of shutdown_timeout_mins is fine, but bear in mind that the ups must also be running on battery for the duration of shutdown_delay mins as well. Another point to consider is the recharge time of the ups. What happens if you have a 7 mins power failure, then 10 minutes of power, followed by another 7 mins of outage? The system won't inititate a shutdown as each failure is less than shutdown_delay_mins, but this will reduce the available storage capacity of the ups.

There was a limit of 15 mins for the 2 values in ups_conf summed. This was changed in a patch, but you must also check whether your ups supports longer settings. The patch is:

PHCO_24153 10.20
PHCO_24172 11.00
PHCO_27828 11i

Please read the patch text for more info.

I also checked into the powerfail stuff a bit more. There's very little mention of it in our internal knowledge base since about 1998. There's a customer viewable document ref: KBAN00000229 in the ITRC that explains a little more, also
see the inittab and signal(5) man pages. It appears to be a hang-on from the days when systems had internal batteries for holding the memory contents rather than using a ups.

I still believe that your rc scripts should handle shutting down the databases, this ensures the system shuts down cleanly - and will save you having to use the powerfail scripts.

regards,

Darren

[Matthew Couper]

Thanks Darren,

I'll look into putting the DB shutdown scripts into the system shutdown also, but using the /etc/powerfail to insert a wall script to let the users know it's going down might be useful at least.

We're going to take a weekend sometime where we can test all this so we can get some good estimated times for the shutdown (and install that patch by then)

Thanks for the help!