Operating System - HP-UX
1834007 Members
1943 Online
110063 Solutions
New Discussion

no pkg switching when node is powered down

 
SOLVED
Go to solution
Henrik Magnusson
New Member

no pkg switching when node is powered down

Hopefully someone here can point me in the correct direction. We have a two node cluster, two rx2600, typically with one package on each node. If I kill a node by pressing the power button for >5 seconds, or with shutdown -h or -r, the package switches over nicely.

However, if I just press the power button like a normal person, something else happens. The package that was running on the node is halted, which disables global switching, and the package is never started on the node that's still running.

What exactly is happening when I press the power button, and where is it configured?

Not a huge problem, generally pressing the power button is probably intentional and an administrator could move the package manually before it happens.
11 REPLIES 11
Steven E. Protter
Exalted Contributor

Re: no pkg switching when node is powered down

Few things to think about.

1) Current on ServiceGuard patches?
2) Any notes when you do the package configuration?
3) Log entries on the node you left running that the package was supposed to fail over to?

My understanding that in the scenario you present the package should fail over. Heartbeat is lost and the surving node should take over IP addresses, volume groups and packages, if they are configured correctly.

if they are not configured correctly there should be log entries on the surviving noce to help you diagnose the problem.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Henrik Magnusson
New Member

Re: no pkg switching when node is powered down

Since the package is halted by the shutdown process, no switching occurs. An ordinary shutdown runs cmhaltnode -fv, and if a node dies the hard way it's detected on the other node and the package is restarted there.

The question is, what is triggered when I just press the power button? The system seems to be configured to halt the package (which disables global switching) but I want it to halt the node.

Perhaps this is more of a general hp-ux/hardware related question than a SG problem.
melvyn burnard
Honored Contributor

Re: no pkg switching when node is powered down

Well my first suggestion would be to check the syslog of the surviving node to see what it saw, then check the switching status of the package that is supposed to start. The next thing would be to check the package log file on that surviving node to see if it attempted to start.
One thing to also check, is the package AUTO_RUN variable set to YES?

And a thought here, the rx2600 may do a soft shutdown when you hit the power button.
try just pulling the power out of the system for a real test
My house is the bank's, my money the wife's, But my opinions belong to me, not HP!
Stephen Doud
Honored Contributor

Re: no pkg switching when node is powered down

I second Melvyn's idea - but also check /var/log/messages.n on the node you powered off - to see if package or node- switching was disabled for the package.

-StephenD.
Sanjay_6
Honored Contributor

Re: no pkg switching when node is powered down

I assume you have already read the following from the rx2600 operations guide:

If the OS is running, pressing the power button:
Momentarily (less than 5 seconds) requests a soft power down by the operating
system. The OS determines if the power off request is to be honored.
For more than 5 seconds (and then released) causes an immediate/hard power
down without allowing the OS to close open files or flush disk caches possibly
resulting in a corrupted disk system or file. This should only be used when the OS
is not responding to user requests.

What's not clear is how HP-UX 'determines if the power off request is to be honored'... maybe someone can shed some light on this.

Cheers

Duncan

I am an HPE Employee
Accept or Kudo
Henrik Magnusson_1
New Member

Re: no pkg switching when node is powered down

Thanks for all the replies!

First of all: Yes, package switching is disabled when I push the power button. Something is halting the package, which disables global switching.

Sanjay: I can't find that document using the ID. The search produces no results. What's the title of the document?

Duncan/melvyn: A "soft shutdown" seems to be exactly what's happening. How hp-ux determines whether or not the power off request is to be honored, and exactly how the computer is shutdown if it should is exactly what I'm looking for! Can someone explain that (or point me in the right direction)?
Sridhar Bhaskarla
Honored Contributor

Re: no pkg switching when node is powered down

Hi Henrik,

// Something is halting the package, which disables global switching.//

With shutdown -h or -r (either manually or by giving it a soft reset), the rc scripts are run. You should have CMCLD set to 1 in your /etc/rc.config.d/cmcluster file. That will try to halt the node with -f option which would bring down the package normally.

-Sri
You may be disappointed if you fail, but you are doomed if you don't try
Nguyen Anh Tien
Honored Contributor
Solution

Re: no pkg switching when node is powered down

what MC-SG you use. Check all required patches on your server by
#show_patches
My system (MC-SG 11.16) have same problem like your. then I found that i lack of patches. after patching It's OK.
you can manual change by:
#cmmodpkg -e
HP is simple
Henrik Magnusson_1
New Member

Re: no pkg switching when node is powered down

I don't have show_patches, but MC-SG is 11.15 so I guess that could be it.

I'm still curious to exactly what's happening. Did some testing again:

These all work as i expect, the package moves nicely to the surviving node:

* shutdown -h
* shutdown -r
* pulling the power out of the system

But, if I push the button on one node (ha2), initiating a soft power down, the package is NOT moved, and these messages are written in the syslog on the surviving node (ha1):

ha1 cmcld: (ha2) Halted package *** on node ha2.
ha1 cmcld: Request from node ha1 to disable global switching for package ***.
ha1 cmcld: Disabled switching for package ***.
ha1 cmcld: Communication to node ha2 has been interrupted
ha1 cmcld: Node ha2 may have died
Henrik Magnusson_1
New Member

Re: no pkg switching when node is powered down

Oops, pushed 'submit' a bit prematurely...

To clarify, with shutdown -r or -h, the syslog also contains messages about disabling switching at first, but then it's enabled again and the package starts. '/sbin/init.d/cmcluster stop'runs cmhaltnode -fv.

I'm guessing that a soft power down doesn't run cmcluster stop, but cmhaltnode (without -f) but I have no idea where that would be configured.