topic Re: Issues with starting cluster in Georgraphic Redundancy Configuration in Operating System - HP-UX

Issues with starting cluster in Georgraphic Redundancy Configuration

NBA — Tue, 11 Nov 2003 15:32:10 GMT

Hi Folks,

I am having issues starting the cluster in a single node cluster configuration i.e. Geogrphic Redundancy. Here is the output of cmviewcl -v on the node that is having issues.

cmviewcl -v

CLUSTER STATUS
vancluster down

NODE STATUS STATE
vanemsc down unknown

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY unknown 0/0/0/0 lan0
STANDBY unknown 0/6/0/0 lan2

UNOWNED_PACKAGES

PACKAGE STATUS STATE PKG_SWITCH NODE
sncPkg down unowned

Policy_Parameters:
POLICY_NAME CONFIGURED_VALUE
Failover unknown
Failback unknown

Script_Parameters:
ITEM STATUS NODE_NAME NAME
Subnet unknown vanemsc 135.93.27.0

Node_Switching_Parameters:
NODE_TYPE STATUS SWITCHING NAME
Primary down vanemsc

Here is the output of the node that is active.

cmviewcl -v

CLUSTER STATUS
toremcluster up

NODE STATUS STATE
toremsc up running

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY up 0/0/0/0 lan0
STANDBY up 0/2/0/0 lan2

PACKAGE STATUS STATE PKG_SWITCH NODE
sncPkg up running enabled toremsc

Policy_Parameters:
POLICY_NAME CONFIGURED_VALUE
Failover configured_node
Failback manual

Script_Parameters:
ITEM STATUS MAX_RESTARTS RESTARTS NAME
Service up Unlimited 0 sncMonitor
Subnet up 135.92.27.0

Node_Switching_Parameters:
NODE_TYPE STATUS SWITCHING NAME
Primary up enabled toremsc (current)

The showtop command on the node that is having issues is indicates standby and the one that is working is indicated as active ACTIVE.:

Here is the output of the syslog.log file from the node that is having issues. Also, it dumps a core file in the /var/adm/cmluster/.

Nov 11 16:48:22 vanemsc : su : + 0 ems-root
Nov 11 17:59:15 vanemsc : su : + 1 ems-root
Nov 11 18:07:41 vanemsc CM-CMD[11496]: cmruncl
Nov 11 18:07:41 vanemsc cmclconfd[11502]: Executing "/usr/lbin/cmcld" for node vanemsc
Nov 11 18:07:41 vanemsc cmcld: Daemon Initialization - Maximum number of packages supported for this incarnation is 8.
Nov 11 18:07:36 vanemsc : su : + 1 ems-root
Nov 11 18:07:41 vanemsc cmcld: Reserving 2048 Kbytes of memory and 64 threads
Nov 11 18:07:42 vanemsc cmcld: The maximum # of concurrent local connections to the daemon that will be supported is 22.
Nov 11 18:07:42 vanemsc cmcld: Warning. No cluster lock is configured.
Nov 11 18:07:42 vanemsc cmcld: Assertion failed: pnet != NULL, file: comm_link.c, line: 146
Nov 11 18:07:44 vanemsc cmclconfd[11502]: The ServiceGuard daemon, /usr/lbin/cmcld[11503], died upon receiving the signal 6.
Nov 11 18:07:44 vanemsc cmsrvassistd[11507]: Lost connection to the cluster daemon.
Nov 11 18:07:44 vanemsc cmsrvassistd[11509]: Unable to notify ServiceGuard main daemon (cmcld): Connection reset by peer
Nov 11 18:07:44 vanemsc cmsrvassistd[11507]: Lost connection with ServiceGuard cluster daemon (cmcld): Software caused connec
tion abort
Nov 11 18:07:44 vanemsc cmclconfd[11512]: Unable to lookup any node information in CDB: Connection refused
Nov 11 18:07:44 vanemsc cmlogd: Unable to initialize with ServiceGuard cluster daemon (cmcld): Connection reset by peer
Nov 11 19:12:51 vanemsc : su : + 3 ems-root
Nov 11 19:13:07 vanemsc CM-CMD[29065]: cmhaltcl -v
Nov 11 19:13:25 vanemsc CM-CMD[29070]: cmhaltcl -n vanemsc
Nov 11 19:13:42 vanemsc CM-CMD[29097]: cmhaltcl -f vanemsc
Nov 11 19:13:48 vanemsc CM-CMD[29098]: cmhaltcl
Nov 11 19:18:32 vanemsc : su : + 3 ems-ems
Nov 11 19:23:19 vanemsc : su : + 1 ems-root
Nov 11 19:23:49 vanemsc CM-CMD[2584]: cmhaltpkg -v -n vanemsc sncPkg
Nov 11 19:33:30 vanemsc CM-CMD[5174]: cmhaltnode
Nov 11 19:37:39 vanemsc CM-CMD[6399]: cmrunpkg -n vanemsc
Nov 11 19:37:51 vanemsc CM-CMD[6400]: cmrunpkg -v -n vanemsc sncPkg
Nov 11 19:37:57 vanemsc CM-CMD[6471]: cmruncl
Nov 11 19:37:58 vanemsc cmclconfd[6477]: Executing "/usr/lbin/cmcld" for node vanemsc
Nov 11 19:37:58 vanemsc cmcld: Daemon Initialization - Maximum number of packages supported for this incarnation is 8.
Nov 11 19:37:58 vanemsc cmcld: Reserving 2048 Kbytes of memory and 64 threads
Nov 11 19:37:58 vanemsc cmcld: The maximum # of concurrent local connections to the daemon that will be supported is 22.
Nov 11 19:37:58 vanemsc cmcld: Warning. No cluster lock is configured.
Nov 11 19:37:58 vanemsc cmcld: Assertion failed: pnet != NULL, file: comm_link.c, line: 146
Nov 11 19:38:01 vanemsc cmclconfd[6477]: The ServiceGuard daemon, /usr/lbin/cmcld[6478], died upon receiving the signal 6.
Nov 11 19:38:01 vanemsc cmsrvassistd[6484]: Unable to notify ServiceGuard main daemon (cmcld): Connection reset by peer
Nov 11 19:38:01 vanemsc cmsrvassistd[6482]: Lost connection to the cluster daemon.
Nov 11 19:38:01 vanemsc cmsrvassistd[6482]: Lost connection with ServiceGuard cluster daemon (cmcld): Software caused connect
ion abort
Nov 11 19:38:01 vanemsc cmlogd: Unable to initialize with ServiceGuard cluster daemon (cmcld): Connection reset by peer
Nov 11 19:41:04 vanemsc : su : + 1 ems-ems
Nov 11 19:56:56 vanemsc : su : + 1 ems-root

Please advise me as to what the cause of the problem could be.

Thanks.
NBA

Re: Issues with starting cluster in Georgraphic Redundancy Configuration

Michael Steele_2 — Tue, 11 Nov 2003 18:52:12 GMT

Since your cluster isn't up the packages aren't yet an issue. Check your cluster this way:

cmcheckconf -v -C /etc/cmcluster/cluster.ascii

Please attach this file:

/etc/cmcluster/cluster.ascii

Re: Issues with starting cluster in Georgraphic Redundancy Configuration

Jeff Schussele — Tue, 11 Nov 2003 19:58:57 GMT

Hi,

"single node cluster configuration i.e. Geogrphic Redundancy."

I'm sorry - to me, these are mutually exclusive terms.

Please explain.

Rgds,
Jeff

Re: Issues with starting cluster in Georgraphic Redundancy Configuration

Michael Steele_2 — Tue, 11 Nov 2003 20:08:22 GMT

Oouuu. I like that. Mutally Exclusive.

Re: Issues with starting cluster in Georgraphic Redundancy Configuration

melvyn burnard — Wed, 12 Nov 2003 03:13:47 GMT

Ok, so why bohter with the toremcluster if it is just the vancluster htat i san issue?
Anyway, what version of SG are you using and what patch level for Sg do you hav einstalled?
do what /usr/lbin/cmcld to get this.

What happens when you do a cmquerycl on this node?
cmquerycl -n vanemsc

Re: Issues with starting cluster in Georgraphic Redundancy Configuration

Stephen Doud — Wed, 12 Nov 2003 09:04:23 GMT

Focus all of your attention on the first error encountered.
The first error that was encountered in the syslog.log file was this:
cmcld: Assertion failed: pnet != NULL, file: comm_link.c, line: 146

This issue has been addressed in patches for Serviceguard versions A.11.13 and A.11.14

Use 'what /usr/lbin/cmcld | grep Date" to determine the version and patch level of Serviceguard loaded.
If SG is not patched, consider loading one:
PHSS_28849 :A.11.13
PHSS_29915 :A.11.14

-sd

Re: Issues with starting cluster in Georgraphic Redundancy Configuration

NBA — Wed, 12 Nov 2003 09:45:43 GMT

Guys,

Thanks for the swift response. Here is the output of the cmcheckconf -v -C /etc/cmcluster/sncCluster.ascii

cmcheckconf -v -C /etc/cmcluster/sncCluster.ascii

Checking cluster file: /etc/cmcluster/sncCluster.ascii
Checking nodes ... Done
Checking existing configuration ...
Done
Gathering configuration information ........... Done
Warning: The disk at /dev/dsk/c0t1d0 on node vanemsc does not have an ID, or a disk label.
Warning: Disks which do not have IDs cannot be included in the topology description.
Use pvcreate(1m) to initialize disks for use with LVM, or
use vxdiskadm(1m) to initalize disks for use with VxVM.
Cluster vancluster is an existing cluster
Checking for inconsistencies .. Done
Cluster vancluster is an existing cluster
Maximum configured packages parameter is 8.
Configuring 1 package(s).
7 package(s) can be added to this cluster.
Modifying configuration on node vanemsc
Modifying the cluster configuration for cluster vancluster.
Validating update for /cluster - value information is identical.
Modifying node vanemsc in cluster vancluster.

Verification completed with no errors found.
Use the cmapplyconf command to apply the configuration.

Here is the output of the what /usr/lbin/cmcld | grep Date
A.11.09 Date: 05/16/2001; PATCH: PHSS_24033.

The other facts I would like to bring to your attention is when I give a command er_status on both the nodes, it indicates "DOWN".

Re: Issues with starting cluster in Georgraphic Redundancy Configuration

Michael Steele_2 — Wed, 12 Nov 2003 10:27:47 GMT

Regarding disk c0t1d0, the course of action is clearly indicated, it needs to pvcreate'd, or, if you're using VXVM then use vxdiskadm.

Warning: The disk at /dev/dsk/c0t1d0 on node vanemsc does not have an ID, or a disk label.
Warning: Disks which do not have IDs cannot be included in the topology description.
Use pvcreate(1m) to initialize disks for use with LVM, or
use vxdiskadm(1m) to initalize disks for use with VxVM.

############################################

LVM or VXVM has to be completed without error before any ServiceGuard work can be accomplished. So go back to either and start your SG afterwards.

Re: Issues with starting cluster in Georgraphic Redundancy Configuration

NBA — Wed, 12 Nov 2003 10:46:26 GMT

Michael,

The disk /dev/dsk/c0t1d0 is a HP DVD-ROM.

regards,
NBA

Re: Issues with starting cluster in Georgraphic Redundancy Configuration

melvyn burnard — Wed, 12 Nov 2003 10:55:28 GMT

The disk message is just a warning, you do not have to use pvcreate on it.
If as you say you have 11.09, this is 7 patches out of date, you should obtain PHSS_27158 and install that, then try again.

Re: Issues with starting cluster in Georgraphic Redundancy Configuration

Michael Steele_2 — Wed, 12 Nov 2003 10:56:00 GMT

Ok then, the same LVM info. exists on both servers for each vg? Right?

If so then apply the cluster bin file and start the cluster without starting the packages.

######################################

Here's how to sync up the alt. node with the pri. node's LVM info.

pri node:

copy down minor number of vg on pri. node
ll /dev/vg##/group
vgchange -a n /dev/vg##
vgexport -s -m /tmp/lvm_map /dev/vg##
ftp /tmp/lvm_map over to alt node

alt node:

mkdir /dev/vg##
mknod /dev/vg##/group c 64 0x0#0000
vgimport -s -m /tmp/lvm_map /dev/vg##

##########################################

Done - cmquerycl -n nodea -n nodeb -C cluster.ascii (* Right? *)

Done - cmcheckconf -C /etc/cmcluster/cluster.ascii (* Right? *)

##########################################

Then its just this:

cmapplyconf -C /etc/cmcluster/cluster.ascii
cmruncl
cmviewcl -v

Verify the cluster is up but not the packages.

Re: Issues with starting cluster in Georgraphic Redundancy Configuration

NBA — Wed, 12 Nov 2003 11:25:59 GMT

Melvyn,

Do I really have to install the patch PHSS_27158. The reason why I am asking is the working toremsc is also having the same version and patch level and doesn't report any problem with starting the cluster on it.

Re: Issues with starting cluster in Georgraphic Redundancy Configuration

Michael Steele_2 — Wed, 12 Nov 2003 11:34:09 GMT

NBA - its critical to have the same patch levels and versions within the cluster.

Re: Issues with starting cluster in Georgraphic Redundancy Configuration

NBA — Wed, 12 Nov 2003 12:28:47 GMT

Michael,

Sorry for any misunderstanding.

The toremsc reports the following information for the "what /usr/lbin/cmcld | grep Date

A.11.09 Date: 05/16/2001; PATCH : PHSS_24033

similar to the version and patch level on the vanemsc which is having cluster issuesie.
A.11.09 Date: 05/16/2001; PATCH : PHSS_24033.

My question is do I really have to install the patch PHSS 27158 when the toremsc is working fine without the PHSS 27158?

Re: Issues with starting cluster in Georgraphic Redundancy Configuration

Michael Steele_2 — Wed, 12 Nov 2003 13:33:39 GMT

NBA: Here is the patch description but I don't see a reason for using it either. HP has replaced PHSS_24033 with PHSS_27158 for OPS reasons.

Are you using OPS?

http://www1.itrc.hp.com/service/patch/patchDetail.do?BC=patch.breadcrumb.pdb|patch.breadcrumb.search|&patchid=PHSS_27158&context=hpux:800:11:00

#########################################

Where you able to bring the cluster up without the packages?

Re: Issues with starting cluster in Georgraphic Redundancy Configuration

melvyn burnard — Wed, 12 Nov 2003 13:34:04 GMT

These patches fix a lot of issues, some not listed in the patch text.
If you read the text file for this patch PHSS_27158 you will see:

At cmcld start up, i.e. cmrunnode or cmruncl, syslog shows this message,
"cmcld: Assertion failed: pnet != NULL, file:comm_link.c, line: 140."

cmcld immediately aborts and dumps core.

I think this is youir problem?

Re: Issues with starting cluster in Georgraphic Redundancy Configuration

NBA — Wed, 12 Nov 2003 14:06:31 GMT

Michael,

I am not using OPS edition, because the system is based on Informix database.

Melvyn,

I have to get a clearance from the customer, before I could install the patch PHS 27158 on the system. Meanwhile, I am open to any other bright ideas.

regards,
NBA

Re: Issues with starting cluster in Georgraphic Redundancy Configuration

NBA — Thu, 13 Nov 2003 11:56:16 GMT

Folks,

I installed the patch PHSS_27158 on the vanemsc server. Somehow, the system reports the same error as I saw with the patch PHSS_24033.

Here is the output of the what /usr/lbin/cmcld command.

vanemsc:what /usr/lbin/cmcld | grep Date
A.11.09 Date: 01/15/2003; PATCH: PHSS_27158

Here is the excerpt from the syslog.log file of the vanemsc server.

Nov 13 16:33:57 vanemsc CM-CMD[23180]: cmruncl -v
Nov 13 16:33:58 vanemsc cmclconfd[23205]: Executing "/usr/lbin/cmcld" for node vanemsc
Nov 13 16:33:58 vanemsc cmcld: Daemon Initialization - Maximum number of packages supported for this incarnation is 8.
Nov 13 16:27:04 vanemsc : su : + 2 ems-ems
Nov 13 16:33:58 vanemsc cmcld: Reserving 2048 Kbytes of memory and 64 threads
Nov 13 16:33:58 vanemsc cmcld: The maximum # of concurrent local connections to the daemon that will be supported is 22.
Nov 13 16:33:58 vanemsc cmcld: Warning. No cluster lock is configured.
Nov 13 16:33:58 vanemsc cmcld: Assertion failed: pnet != NULL, file: comm_link.c, line: 146
Nov 13 16:34:02 vanemsc cmlogd: Unable to initialize with ServiceGuard cluster daemon (cmcld): Connection reset by peer
Nov 13 16:34:02 vanemsc cmsrvassistd[23244]: Unable to notify ServiceGuard main daemon (cmcld): Connection reset by peer
Nov 13 16:34:02 vanemsc cmsrvassistd[23228]: Lost connection to the cluster daemon.
Nov 13 16:34:02 vanemsc cmsrvassistd[23228]: Lost connection with ServiceGuard cluster daemon (cmcld): Software caused connection abort
Nov 13 16:34:02 vanemsc cmclconfd[23351]: Unable to lookup any node information in CDB: Connection refused
Nov 13 16:34:02 vanemsc cmclconfd[23205]: The ServiceGuard daemon, /usr/lbin/cmcld[23206], died upon receiving the signal 6.
Nov 13 16:34:29 vanemsc CM-CMD[25026]: cmrunnode vanemsc
Nov 13 16:34:29 vanemsc cmclconfd[25031]: Executing "/usr/lbin/cmcld" for node vanemsc
Nov 13 16:34:29 vanemsc cmcld: Daemon Initialization - Maximum number of packages supported for this incarnation is 8.
Nov 13 16:34:29 vanemsc cmcld: Reserving 2048 Kbytes of memory and 64 threads
Nov 13 16:34:30 vanemsc cmcld: The maximum # of concurrent local connections to the daemon that will be supported is 22.
Nov 13 16:34:30 vanemsc cmcld: Warning. No cluster lock is configured.
Nov 13 16:34:30 vanemsc cmcld: Assertion failed: pnet != NULL, file: comm_link.c, line: 146
Nov 13 16:34:32 vanemsc cmsrvassistd[25037]: Unable to notify ServiceGuard main daemon (cmcld): Connection reset by peer
Nov 13 16:34:32 vanemsc cmsrvassistd[25038]: Unable to notify ServiceGuard main daemon (cmcld): Connection reset by peer
Nov 13 16:34:32 vanemsc cmsrvassistd[25036]: Lost connection to the cluster daemon.
Nov 13 16:34:32 vanemsc cmsrvassistd[25036]: Lost connection with ServiceGuard cluster daemon (cmcld): Software caused connection abort
Nov 13 16:34:32 vanemsc cmclconfd[25031]: The ServiceGuard daemon, /usr/lbin/cmcld[25032], died upon receiving the signal 6.
Nov 13 16:35:00 vanemsc CM-CMD[25090]: cmmodpkg -e sncPkg
Nov 13 16:35:00 vanemsc CM-CMD[25110]: cmmodpkg -e -n vanemsc sncPkg

I will continue my investigation. Meanwhile, if you have any ideas, please let me know.

Thanks.

Re: Issues with starting cluster in Georgraphic Redundancy Configuration

Michael Steele_2 — Thu, 13 Nov 2003 12:59:50 GMT

I would endeavor to mirror the same patches on both nodes before proceeding.

I would also assume corrupted cluster binaries, work to get one node up and copy the data over to the other.

Try this command from :

http://forums1.itrc.hp.com/service/forums/parseCurl.do?CURL=%2Fcm%2FQuestionAnswer%2F1%2C%2C0xb02a4b3ef09fd611abdb0090277a778c%2C00.html&admit=716493758+1068745972259+28353475

#########################################

# cmquerycl -C config.ascii -n lr006b04 -n lr006b05

This will NOT list any VGs that are already "clustered" but it will tell you if the nodes have concurrent versions of serviceguard, whether the nodes can communicate via the hacl ports (/etc/services), whether the security files (~/.rhosts or /etc/cmcluster/cmclnodelist) allows the communication etc.
If this command does not work - at least one fundamental system configuration issue exists which prevents ServiceGuard from operating properly in the present state.

Re: Issues with starting cluster in Georgraphic Redundancy Configuration

NBA — Thu, 13 Nov 2003 13:36:15 GMT

#Michael,

At the moment, the toremsc is the only server that is monitoring the network without any problems. It will be difficult to convince the customer to bring down the package and cluster to install the patch PHSS_27158. Therefore, I will continue to troubleshoot the vanemsc.

Here is the output of the command from the vanemsc
cmquerycl -C config.ascii -n vanemsc -n toremsc

Warning: The disk at /dev/dsk/c0t1d0 on node vanemsc does not have an ID, or a disk label.
Warning: The disk at /dev/dsk/c0t2d0 on node toremsc does not have an ID, or a disk label.
Warning: Disks which do not have IDs cannot be included in the topology description.
Use pvcreate(1m) to initialize disks for use with LVM, or
use vxdiskadm(1m) to initalize disks for use with VxVM.
Warning: Network interface lan3 on node toremsc couldn't talk to itself.
Warning: Network interface lan4 on node toremsc couldn't talk to itself.

Node Names: toremsc
vanemsc

Bridged networks:

1 lan0 (toremsc)
lan2 (toremsc)

2 lan1 (toremsc)

3 lan0 (vanemsc)

4 lan1 (vanemsc)
lan2 (vanemsc)

IP subnets:

135.93.27.0 lan0 (vanemsc)

64.251.200.0 lan1 (vanemsc)

135.92.27.0 lan0 (toremsc)
lan1 (toremsc)

Possible Heartbeat IPs:

Possible Cluster Lock Devices:

LVM volume groups:

/dev/vg00 vanemsc

/dev/vg01 vanemsc

/dev/vg02 vanemsc

/dev/vg03 vanemsc

/dev/vg00 toremsc

/dev/vg01 toremsc

/dev/vg02 toremsc

/dev/vg03 toremsc

Warning: No possible heartbeat networks found.
All nodes must be connected to at least one common network.
This may be due to DLPI not being installed.
Warning: Failed to find a configuration that satisfies the minimum network configuration requirements.
Minimum network configuration requirements are:
- 2 or more heartbeat networks OR
- 1 heartbeat network with local switch OR
- 1 heartbeat network with serial line.

NOTE: Please ignore the warnings on disk c0t1d0 and c0t2d0 as these are DVD ROM drives.
Also, since these servers are configured on Geographic Redundancy, they do not have hearbeat LAN configured, please ignore the Warning: Failed to find a configuration that satisfies the minimum network configuration requirements.