topic Re: no pkg switching when node is powered down in Operating System - HP-UX

no pkg switching when node is powered down

Henrik Magnusson — Fri, 10 Dec 2004 10:15:37 GMT

Hopefully someone here can point me in the correct direction. We have a two node cluster, two rx2600, typically with one package on each node. If I kill a node by pressing the power button for >5 seconds, or with shutdown -h or -r, the package switches over nicely.

However, if I just press the power button like a normal person, something else happens. The package that was running on the node is halted, which disables global switching, and the package is never started on the node that's still running.

What exactly is happening when I press the power button, and where is it configured?

Not a huge problem, generally pressing the power button is probably intentional and an administrator could move the package manually before it happens.

Re: no pkg switching when node is powered down

Steven E. Protter — Fri, 10 Dec 2004 10:25:32 GMT

Few things to think about.

1) Current on ServiceGuard patches?
2) Any notes when you do the package configuration?
3) Log entries on the node you left running that the package was supposed to fail over to?

My understanding that in the scenario you present the package should fail over. Heartbeat is lost and the surving node should take over IP addresses, volume groups and packages, if they are configured correctly.

if they are not configured correctly there should be log entries on the surviving noce to help you diagnose the problem.

SEP

Re: no pkg switching when node is powered down

Henrik Magnusson — Fri, 10 Dec 2004 11:40:23 GMT

Since the package is halted by the shutdown process, no switching occurs. An ordinary shutdown runs cmhaltnode -fv, and if a node dies the hard way it's detected on the other node and the package is restarted there.

The question is, what is triggered when I just press the power button? The system seems to be configured to halt the package (which disables global switching) but I want it to halt the node.

Perhaps this is more of a general hp-ux/hardware related question than a SG problem.

Re: no pkg switching when node is powered down

melvyn burnard — Fri, 10 Dec 2004 11:46:27 GMT

Well my first suggestion would be to check the syslog of the surviving node to see what it saw, then check the switching status of the package that is supposed to start. The next thing would be to check the package log file on that surviving node to see if it attempted to start.
One thing to also check, is the package AUTO_RUN variable set to YES?

And a thought here, the rx2600 may do a soft shutdown when you hit the power button.
try just pulling the power out of the system for a real test

Re: no pkg switching when node is powered down

Stephen Doud — Fri, 10 Dec 2004 12:59:46 GMT

I second Melvyn's idea - but also check /var/log/messages.n on the node you powered off - to see if package or node- switching was disabled for the package.

-StephenD.

Re: no pkg switching when node is powered down

Sanjay_6 — Fri, 10 Dec 2004 14:19:07 GMT

Hi Henrik,

Maybe this doc could shed some light,

http://www2.itrc.hp.com/service/cki/search.do?category=c0&mode=id&searchString=UXSGKBAN00000102&searchCrit=allwords&docType=Security&docType=Patch&docType=EngineerNotes&docType=BugReports&docType=Hardware&docType=ReferenceMaterials&docType=ThirdParty&search.x=10&search.y=10

The itrc doc id is UXSGKBAN00000102.

Hope this helps.

Regds

Re: no pkg switching when node is powered down

Duncan Edmonstone — Sun, 12 Dec 2004 06:31:33 GMT

I assume you have already read the following from the rx2600 operations guide:

If the OS is running, pressing the power button:
Momentarily (less than 5 seconds) requests a soft power down by the operating
system. The OS determines if the power off request is to be honored.
For more than 5 seconds (and then released) causes an immediate/hard power
down without allowing the OS to close open files or flush disk caches possibly
resulting in a corrupted disk system or file. This should only be used when the OS
is not responding to user requests.

What's not clear is how HP-UX 'determines if the power off request is to be honored'... maybe someone can shed some light on this.

Cheers

Duncan

Re: no pkg switching when node is powered down

Henrik Magnusson_1 — Mon, 13 Dec 2004 04:04:37 GMT

Thanks for all the replies!

First of all: Yes, package switching is disabled when I push the power button. Something is halting the package, which disables global switching.

Sanjay: I can't find that document using the ID. The search produces no results. What's the title of the document?

Duncan/melvyn: A "soft shutdown" seems to be exactly what's happening. How hp-ux determines whether or not the power off request is to be honored, and exactly how the computer is shutdown if it should is exactly what I'm looking for! Can someone explain that (or point me in the right direction)?

Re: no pkg switching when node is powered down

Sridhar Bhaskarla — Mon, 13 Dec 2004 04:09:09 GMT

Hi Henrik,

// Something is halting the package, which disables global switching.//

With shutdown -h or -r (either manually or by giving it a soft reset), the rc scripts are run. You should have CMCLD set to 1 in your /etc/rc.config.d/cmcluster file. That will try to halt the node with -f option which would bring down the package normally.

-Sri

Re: no pkg switching when node is powered down

Nguyen Anh Tien — Mon, 13 Dec 2004 05:14:20 GMT

what MC-SG you use. Check all required patches on your server by
#show_patches
My system (MC-SG 11.16) have same problem like your. then I found that i lack of patches. after patching It's OK.
you can manual change by:
#cmmodpkg -e

Re: no pkg switching when node is powered down

Henrik Magnusson_1 — Mon, 13 Dec 2004 08:54:47 GMT

I don't have show_patches, but MC-SG is 11.15 so I guess that could be it.

I'm still curious to exactly what's happening. Did some testing again:

These all work as i expect, the package moves nicely to the surviving node:

* shutdown -h
* shutdown -r
* pulling the power out of the system

But, if I push the button on one node (ha2), initiating a soft power down, the package is NOT moved, and these messages are written in the syslog on the surviving node (ha1):

ha1 cmcld: (ha2) Halted package *** on node ha2.
ha1 cmcld: Request from node ha1 to disable global switching for package ***.
ha1 cmcld: Disabled switching for package ***.
ha1 cmcld: Communication to node ha2 has been interrupted
ha1 cmcld: Node ha2 may have died

Re: no pkg switching when node is powered down

Henrik Magnusson_1 — Mon, 13 Dec 2004 09:16:48 GMT

Oops, pushed 'submit' a bit prematurely...

To clarify, with shutdown -r or -h, the syslog also contains messages about disabling switching at first, but then it's enabled again and the package starts. '/sbin/init.d/cmcluster stop'runs cmhaltnode -fv.

I'm guessing that a soft power down doesn't run cmcluster stop, but cmhaltnode (without -f) but I have no idea where that would be configured.