Cluster Disk error

[Sakui]

This is an error from a HP rp 3440 cluster server.

Aug 25 13:33:56 cg1 cmclconfd[1511]: Executing "/usr/lbin/cmcld" for node cg1
Aug 25 13:33:56 cg1 cmcld: Logging level changed to level 0.
Aug 25 13:33:56 cg1 cmcld: Daemon Initialization - Maximum number of packages supported for this incarnation is 1.
Aug 25 13:33:56 cg1 cmcld: Global Cluster Information:
Aug 25 13:33:56 cg1 cmcld: Heartbeat Interval is 1 seconds.
Aug 25 13:33:56 cg1 cmcld: Node Timeout is 5 seconds.
Aug 25 13:33:56 cg1 cmcld: Network Polling Interval is 2 seconds.
Aug 25 13:33:56 cg1 cmcld: Logging level changed to level 0.
Aug 25 13:33:56 cg1 cmcld: Auto Start Timeout is 600 seconds.
Aug 25 13:33:56 cg1 cmcld: Information Specific to node cg1:
Aug 25 13:33:56 cg1 cmcld: Cluster lock disk: /dev/dsk/c6t0d0.
Aug 25 13:33:56 cg1 cmcld: Cluster lock disk: /dev/dsk/c4t0d0.
Aug 25 13:33:56 cg1 cmcld: lan0 0x0014c210830f 192.168.1.3 bridged net:1
Aug 25 13:33:56 cg1 cmcld: lan1 0x001279b314fe 10.3.12.53 bridged net:2
Aug 25 13:33:56 cg1 cmcld: lan3 0x001279b32402 standby bridged net:2
Aug 25 13:33:56 cg1 cmcld: Heartbeat Subnet: 192.168.1.0
Aug 25 13:33:56 cg1 cmcld: Heartbeat Subnet: 10.3.12.32
Aug 25 13:33:56 cg1 cmcld: The maximum # of concurrent local connections to the daemon that will be supported is 994.
Aug 25 13:33:56 cg1 cmcld: bytes is 64
Aug 25 13:33:56 cg1 cmcld: Link level address on network interface lan0 has been updated from 0x0014c210830f to 0x00306e4b47a7.
Aug 25 13:33:56 cg1 cmcld: Proceeding with the new configuration.
Aug 25 13:33:57 cg1 cmcld: lan3 failed
Aug 25 13:33:57 cg1 cmcld: rcomm health: Initializing timeout to 120000000 microseconds
Aug 25 13:33:57 cg1 cmcld: Total allocated: 2810548 bytes, used: 112760 bytes, unused 2509640 bytes
Aug 25 13:33:57 cg1 cmcld: Starting cluster management protocols.
Aug 25 13:33:57 cg1 cmcld: Attempting to form a new cluster
Aug 25 13:34:00 cg1 cmcld: Turning on safety time protection
Aug 25 13:34:00 cg1 cmcld: 2 nodes have formed a new cluster, sequence #4
Aug 25 13:34:00 cg1 cmcld: The new active cluster membership is: cg2(id=2), cg1(id=1)
Aug 25 13:35:14 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 25 13:35:14 cg1 cmcld: Check device, power, and cables.
Aug 25 13:35:16 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.1 for node cg1 (length = 69119).
Aug 25 14:36:31 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 25 14:36:31 cg1 cmcld: Check device, power, and cables.
Aug 25 14:36:32 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.2 for node cg1 (length = 512096).
Aug 25 15:37:46 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 25 15:37:46 cg1 cmcld: Check device, power, and cables.
Aug 25 15:37:47 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.3 for node cg1 (length = 512096).
Aug 25 16:39:01 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 25 16:39:01 cg1 cmcld: Check device, power, and cables.
Aug 25 16:39:02 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.4 for node cg1 (length = 512096).
Aug 25 17:40:16 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 25 17:40:16 cg1 cmcld: Check device, power, and cables.
Aug 25 17:40:17 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.5 for node cg1 (length = 512096).
Aug 25 18:41:31 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 25 18:41:31 cg1 cmcld: Check device, power, and cables.
Aug 25 18:41:32 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.6 for node cg1 (length = 512096).
Aug 25 19:42:46 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 25 19:42:46 cg1 cmcld: Check device, power, and cables.
Aug 25 19:42:47 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.7 for node cg1 (length = 512096).
Aug 25 20:44:01 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 25 20:44:01 cg1 cmcld: Check device, power, and cables.
Aug 25 20:44:02 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.8 for node cg1 (length = 512096).
Aug 25 21:45:16 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 25 21:45:16 cg1 cmcld: Check device, power, and cables.
Aug 25 21:45:17 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.9 for node cg1 (length = 512096).
Aug 25 22:46:31 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 25 22:46:31 cg1 cmcld: Check device, power, and cables.
Aug 25 22:46:32 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.0 for node cg1 (length = 512096).
Aug 25 23:47:46 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 25 23:47:46 cg1 cmcld: Check device, power, and cables.
Aug 25 23:47:47 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.1 for node cg1 (length = 512096).
Aug 26 00:49:01 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 26 00:49:01 cg1 cmcld: Check device, power, and cables.
Aug 26 00:49:02 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.2 for node cg1 (length = 512096).
Aug 26 01:50:16 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 26 01:50:16 cg1 cmcld: Check device, power, and cables.
Aug 26 01:50:17 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.3 for node cg1 (length = 512096).
Aug 26 02:51:31 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 26 02:51:31 cg1 cmcld: Check device, power, and cables.
Aug 26 02:51:32 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.4 for node cg1 (length = 512096).
Aug 26 03:52:46 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 26 03:52:46 cg1 cmcld: Check device, power, and cables.
Aug 26 03:52:47 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.5 for node cg1 (length = 512096).
Aug 26 04:54:01 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 26 04:54:01 cg1 cmcld: Check device, power, and cables.
Aug 26 04:54:02 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.6 for node cg1 (length = 512096).
Aug 26 05:55:16 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 26 05:55:16 cg1 cmcld: Check device, power, and cables.
Aug 26 05:55:17 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.7 for node cg1 (length = 512096).
Aug 26 06:56:31 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 26 06:56:31 cg1 cmcld: Check device, power, and cables.
Aug 26 06:56:32 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.8 for node cg1 (length = 512096).
Aug 26 07:57:46 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 26 07:57:46 cg1 cmcld: Check device, power, and cables.
Aug 26 07:57:47 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.9 for node cg1 (length = 512096).
Aug 26 08:59:01 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 26 08:59:01 cg1 cmcld: Check device, power, and cables.
Aug 26 08:59:02 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.0 for node cg1 (length = 512096).
Aug 26 10:00:16 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 26 10:00:16 cg1 cmcld: Check device, power, and cables.
Aug 26 10:00:17 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.1 for node cg1 (length = 512096).
Aug 26 11:01:31 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 26 11:01:31 cg1 cmcld: Check device, power, and cables.
Aug 26 11:01:32 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.2 for node cg1 (length = 512096).
Aug 26 12:02:46 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 26 12:02:46 cg1 cmcld: Check device, power, and cables.
Aug 26 12:02:47 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.3 for node cg1 (length = 512096).
Aug 26 13:04:01 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 26 13:04:01 cg1 cmcld: Check device, power, and cables.
Aug 26 13:04:02 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.4 for node cg1 (length = 512096).
Aug 26 14:05:16 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 26 14:05:16 cg1 cmcld: Check device, power, and cables.
Aug 26 14:05:17 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.5 for node cg1 (length = 512096).
Aug 26 15:06:31 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 26 15:06:31 cg1 cmcld: Check device, power, and cables.
Aug 26 15:06:32 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.6 for node cg1 (length = 512096).
Aug 26 16:07:46 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 26 16:07:46 cg1 cmcld: Check device, power, and cables.
Aug 26 16:07:47 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.7 for node cg1 (length = 512096).
Aug 26 17:09:01 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 26 17:09:01 cg1 cmcld: Check device, power, and cables.
Aug 26 17:09:02 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.8 for node cg1 (length = 512096).
Aug 26 18:10:16 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 26 18:10:16 cg1 cmcld: Check device, power, and cables.
Aug 26 18:10:17 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.9 for node cg1 (length = 512096).
Aug 26 19:11:31 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 26 19:11:31 cg1 cmcld: Check device, power, and cables.
Aug 26 19:11:32 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.0 for node cg1 (length = 512096).
Aug 26 20:12:46 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 26 20:12:46 cg1 cmcld: Check device, power, and cables.
Aug 26 20:12:47 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.1 for node cg1 (length = 512096).
Aug 26 21:14:01 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 26 21:14:01 cg1 cmcld: Check device, power, and cables.
Aug 26 21:14:02 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.2 for node cg1 (length = 512096).
Aug 26 22:15:16 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 26 22:15:16 cg1 cmcld: Check device, power, and cables.
Aug 26 22:15:17 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.3 for node cg1 (length = 512096).
Aug 26 23:16:31 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 26 23:16:31 cg1 cmcld: Check device, power, and cables.
Aug 26 23:16:32 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.4 for node cg1 (length = 512096).
Aug 27 00:17:46 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 27 00:17:46 cg1 cmcld: Check device, power, and cables.
Aug 27 00:17:47 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.5 for node cg1 (length = 512096).
Aug 27 00:26:10 cg1 cmcld: Request from node cg2 to disable global switching for package pkg_a8965.
Aug 27 00:26:24 cg1 cmcld: (cg2) Halted package pkg_a8965 on node cg2.
Aug 27 00:26:56 cg1 cmcld: Request from node cg2 to start package pkg_a8965 on node cg1.
Aug 27 00:26:56 cg1 cmcld: Executing '/etc/cmcluster/pkg_a8965/control.sh start' for package pkg_a8965, as service PKG*4353.
Aug 27 00:27:01 cg1 cmcld: Service PKG*4353 terminated due to an exit(0).
Aug 27 00:27:01 cg1 cmcld: Started package pkg_a8965 on node cg1.
Aug 27 00:43:42 cg1 cmcld: Request from node cg2 to enable global switching for package pkg_a8965.
Aug 27 01:19:01 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 27 01:19:01 cg1 cmcld: Check device, power, and cables.
Aug 27 01:19:02 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.6 for node cg1 (length = 512096).
Aug 27 02:20:16 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 27 02:20:16 cg1 cmcld: Check device, power, and cables.
Aug 27 02:20:17 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.7 for node cg1 (length = 512096).
Aug 27 03:21:31 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 27 03:21:31 cg1 cmcld: Check device, power, and cables.
Aug 27 03:21:32 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.8 for node cg1 (length = 512096).
Aug 27 04:22:46 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 27 04:22:46 cg1 cmcld: Check device, power, and cables.
Aug 27 04:22:47 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.9 for node cg1 (length = 512096).
Aug 27 05:24:01 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 27 05:24:01 cg1 cmcld: Check device, power, and cables.
Aug 27 05:24:02 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.0 for node cg1 (length = 512096).
Aug 27 06:25:16 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 27 06:25:16 cg1 cmcld: Check device, power, and cables.
Aug 27 06:25:17 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.1 for node cg1 (length = 512096).
Aug 27 07:26:31 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 27 07:26:31 cg1 cmcld: Check device, power, and cables.
Aug 27 07:26:32 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.2 for node cg1 (length = 512096).
Aug 27 08:04:48 cg1 cmcld: Request from node cg2 to disable global switching for package pkg_a8965.
Aug 27 08:04:48 cg1 cmcld: Request from node cg2 to halt package pkg_a8965 on node cg1.
Aug 27 08:04:48 cg1 cmcld: Executing '/etc/cmcluster/pkg_a8965/control.sh stop' for package pkg_a8965, as service PKG*4353.
Aug 27 08:05:03 cg1 cmcld: Service PKG*4353 terminated due to an exit(0).
Aug 27 08:05:03 cg1 cmcld: Halted package pkg_a8965 on node cg1.
Aug 27 08:05:47 cg1 cmcld: (cg2) Started package pkg_a8965 on node cg2.
Aug 27 08:14:17 cg1 cmcld: (cg2) Halted package pkg_a8965 on node cg2.
Aug 27 08:25:45 cg1 cmcld: Request from node cg2 to start package pkg_a8965 on node cg1.
Aug 27 08:25:45 cg1 cmcld: Executing '/etc/cmcluster/pkg_a8965/control.sh start' for package pkg_a8965, as service PKG*4353.
Aug 27 08:25:46 cg1 cmcld: Service PKG*4353 terminated due to an exit(1).
Aug 27 08:25:46 cg1 cmcld: Package pkg_a8965 run script exited with NO_RESTART.
Aug 27 08:25:46 cg1 cmcld: Examine the file /etc/cmcluster/pkg_a8965/control.sh.log for more details.
Aug 27 08:25:46 cg1 cmcld: Switching disabled on package pkg_a8965.
Aug 27 08:26:21 cg1 cmcld: Request from node cg2 to enable global switching for package pkg_a8965.
Aug 27 08:26:21 cg1 cmcld: Request from node cg2 to start package pkg_a8965 on node cg1.
Aug 27 08:26:21 cg1 cmcld: Executing '/etc/cmcluster/pkg_a8965/control.sh start' for package pkg_a8965, as service PKG*4353.
Aug 27 08:26:21 cg1 cmcld: Service PKG*4353 terminated due to an exit(1).
Aug 27 08:26:21 cg1 cmcld: Package pkg_a8965 run script exited with NO_RESTART.
Aug 27 08:26:21 cg1 cmcld: Examine the file /etc/cmcluster/pkg_a8965/control.sh.log for more details.
Aug 27 08:26:21 cg1 cmcld: Switching disabled on package pkg_a8965.
Aug 27 08:26:21 cg1 cmcld: Request from node cg2 to disable global switching for package pkg_a8965.
Aug 27 08:26:30 cg1 cmcld: Request from node cg2 to start package pkg_a8965 on node cg1.
Aug 27 08:26:30 cg1 cmcld: Executing '/etc/cmcluster/pkg_a8965/control.sh start' for package pkg_a8965, as service PKG*4353.
Aug 27 08:26:30 cg1 cmcld: Service PKG*4353 terminated due to an exit(1).
Aug 27 08:26:30 cg1 cmcld: Package pkg_a8965 run script exited with NO_RESTART.
Aug 27 08:26:30 cg1 cmcld: Examine the file /etc/cmcluster/pkg_a8965/control.sh.log for more details.
Aug 27 08:26:30 cg1 cmcld: Switching disabled on package pkg_a8965.
Aug 27 08:27:46 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 27 08:27:46 cg1 cmcld: Check device, power, and cables.
Aug 27 08:27:47 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.3 for node cg1 (length = 512096).
Aug 27 08:29:48 cg1 cmcld: Communication to node cg2 has been interrupted
Aug 27 08:29:48 cg1 cmcld: Node cg2 may have died
Aug 27 08:29:48 cg1 cmcld: Attempting to form a new cluster
Aug 27 08:29:49 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.4 for node cg1 (length = 60897).
Aug 27 08:29:54 cg1 cmcld: Obtaining First Dual Cluster Lock
Aug 27 08:29:56 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.5 for node cg1 (length = 5220).
Aug 27 08:29:57 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.6 for node cg1 (length = 1862).
Aug 27 08:29:58 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.7 for node cg1 (length = 1764).
Aug 27 08:30:04 cg1 cmcld: Unable to get First Dual Cluster Lock. Operation timed out.
Aug 27 08:30:04 cg1 cmcld: Obtaining Second Dual Cluster Lock
Aug 27 08:30:05 cg1 cmcld: Turning off safety time protection since the cluster
Aug 27 08:30:05 cg1 cmcld: may now consist of a single node. If ServiceGuard
Aug 27 08:30:05 cg1 cmcld: fails, this node will not automatically halt
Aug 27 08:30:48 cg1 cmcld: 1 nodes have formed a new cluster, sequence #5
Aug 27 08:30:48 cg1 cmcld: The new active cluster membership is: cg1(id=1)
Aug 27 08:30:48 cg1 cmcld: One of the nodes is down.
Aug 27 08:30:48 cg1 cmcld: One or more packages is not currently running because AUTO_RUN is disabled so that it cannot start automatically. To start these packages, enable AUTO_RUN via cmmodpkg -e .
Aug 27 08:31:20 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 27 08:31:20 cg1 cmcld: Check device, power, and cables.
Aug 27 08:33:34 cg1 cmcld: Request from node cg1 to start package pkg_a8965 on node cg1.
Aug 27 08:33:34 cg1 cmcld: Executing '/etc/cmcluster/pkg_a8965/control.sh start' for package pkg_a8965, as service PKG*4353.
Aug 27 08:34:16 cg1 cmcld: Service PKG*4353 terminated due to an exit(0).
Aug 27 08:34:16 cg1 cmcld: Started package pkg_a8965 on node cg1.
Aug 27 08:38:29 cg1 cmcld: New node cg2 is joining the cluster
Aug 27 08:38:29 cg1 cmcld: Attempting to adjust cluster membership
Aug 27 08:38:31 cg1 cmcld: Enabling safety time protection
Aug 27 08:38:31 cg1 cmcld: Clearing First Dual Cluster Lock
Aug 27 08:38:32 cg1 cmcld: 2 nodes have formed a new cluster, sequence #6
Aug 27 08:38:32 cg1 cmcld: The new active cluster membership is: cg1(id=1), cg2(id=2)
Aug 27 08:38:33 cg1 cmcld: One or more packages is not currently running because AUTO_RUN is disabled so that it cannot start automatically. To start these packages, enable AUTO_RUN via cmmodpkg -e .
Aug 27 08:38:33 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.8 for node cg1 (length = 255441).
Aug 27 08:39:47 cg1 cmcld: WARNING: Cluster lock disk /dev/dsk/c6t0d0 has failed (still busy after 75 retries).
Aug 27 08:39:47 cg1 cmcld: Until it is fixed, a single failure could
Aug 27 08:39:47 cg1 cmcld: cause all nodes in the cluster to fail.
Aug 27 08:39:47 cg1 cmcld: Moving on to clear Second Dual Cluster Lock
Aug 27 08:39:48 cg1 cmcld: Health query for cluster lock disk /dev/dsk/c4t0d0 shows BUSY lock
Aug 27 08:39:48 cg1 cmcld: Check device, power, and cables.
Aug 27 08:39:50 cg1 cmcld: Health query for cluster lock disk /dev/dsk/c4t0d0 shows BUSY lock
Aug 27 08:41:06 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 27 08:41:07 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.9 for node cg1 (length = 128372).
Aug 27 08:41:06 cg1 cmcld: Check device, power, and cables.
Aug 27 08:41:26 cg1 cmcld: Request from node cg1 to halt package pkg_a8965 on node cg1.
Aug 27 08:41:26 cg1 cmcld: Executing '/etc/cmcluster/pkg_a8965/control.sh stop' for package pkg_a8965, as service PKG*4353.
Aug 27 08:41:43 cg1 cmcld: Service PKG*4353 terminated due to an exit(0).
Aug 27 08:41:43 cg1 cmcld: Halted package pkg_a8965 on node cg1.
Aug 27 08:42:18 cg1 cmcld: (cg2) Started package pkg_a8965 on node cg2.
Aug 27 09:42:21 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 27 09:42:21 cg1 cmcld: Check device, power, and cables.
Aug 27 09:42:22 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.0 for node cg1 (length = 512096).
Aug 27 10:43:36 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 27 10:43:36 cg1 cmcld: Check device, power, and cables.
Aug 27 10:43:37 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.1 for node cg1 (length = 512096).
Aug 27 11:41:08 cg1 cmcld: (cg2) Halted package pkg_a8965 on node cg2.
Aug 27 11:41:08 cg1 cmcld: Package pkg_a8965 did not halt cleanly. Owning node encountered an error during halt.
Aug 27 11:44:51 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 27 11:44:51 cg1 cmcld: Check device, power, and cables.
Aug 27 11:44:52 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.2 for node cg1 (length = 512096).
Aug 27 11:54:11 cg1 cmcld: Disabled package pkg_a8965 on node cg2.
Aug 27 11:54:38 cg1 cmcld: Communication to node cg2 has been interrupted
Aug 27 11:54:38 cg1 cmcld: Node cg2 may have died
Aug 27 11:54:38 cg1 cmcld: Attempting to adjust cluster membership
Aug 27 11:54:40 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.3 for node cg1 (length = 219195).
Aug 27 11:54:44 cg1 cmcld: Obtaining First Dual Cluster Lock
Aug 27 11:54:47 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.4 for node cg1 (length = 5394).
Aug 27 11:54:48 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.5 for node cg1 (length = 1116).
Aug 27 11:54:49 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.6 for node cg1 (length = 1862).
Aug 27 11:54:53 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.7 for node cg1 (length = 5531).
Aug 27 11:54:55 cg1 cmcld: Unable to get First Dual Cluster Lock. Operation timed out.
Aug 27 11:54:55 cg1 cmcld: Obtaining Second Dual Cluster Lock
Aug 27 11:54:56 cg1 cmcld: Turning off safety time protection since the cluster
Aug 27 11:54:56 cg1 cmcld: may now consist of a single node. If ServiceGuard
Aug 27 11:54:56 cg1 cmcld: fails, this node will not automatically halt
Aug 27 11:55:39 cg1 cmcld: 1 nodes have formed a new cluster, sequence #7
Aug 27 11:55:39 cg1 cmcld: The new active cluster membership is: cg1(id=1)
Aug 27 11:55:39 cg1 cmcld: One of the nodes is down.
Aug 27 11:55:39 cg1 cmcld: One or more packages is not currently running because AUTO_RUN is disabled so that it cannot start automatically. To start these packages, enable AUTO_RUN via cmmodpkg -e .
Aug 27 11:56:11 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 27 11:56:11 cg1 cmcld: Check device, power, and cables.
Aug 27 11:57:20 cg1 cmcld: Request from node cg1 to start package pkg_a8965 on node cg1.
Aug 27 11:57:20 cg1 cmcld: Executing '/etc/cmcluster/pkg_a8965/control.sh start' for package pkg_a8965, as service PKG*4353.
Aug 27 11:58:02 cg1 cmcld: Service PKG*4353 terminated due to an exit(0).
Aug 27 11:58:02 cg1 cmcld: Started package pkg_a8965 on node cg1.
Aug 27 12:03:52 cg1 cmcld: New node cg2 is joining the cluster
Aug 27 12:03:52 cg1 cmcld: Attempting to adjust cluster membership
Aug 27 12:03:54 cg1 cmcld: Enabling safety time protection
Aug 27 12:03:54 cg1 cmcld: Clearing First Dual Cluster Lock
Aug 27 12:03:55 cg1 cmcld: 2 nodes have formed a new cluster, sequence #8
Aug 27 12:03:55 cg1 cmcld: The new active cluster membership is: cg1(id=1), cg2(id=2)
Aug 27 12:03:55 cg1 cmcld: One or more packages is not currently running because AUTO_RUN is disabled so that it cannot start automatically. To start these packages, enable AUTO_RUN via cmmodpkg -e .
Aug 27 12:04:40 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.8 for node cg1 (length = 294642).
Aug 27 12:04:47 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.9 for node cg1 (length = 9704).
Aug 27 12:04:49 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.0 for node cg1 (length = 1951).
Aug 27 12:04:50 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.1 for node cg1 (length = 1384).
Aug 27 12:04:54 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.2 for node cg1 (length = 5197).
Aug 27 12:05:09 cg1 cmcld: WARNING: Cluster lock disk /dev/dsk/c6t0d0 has failed (still busy after 75 retries).
Aug 27 12:05:09 cg1 cmcld: Until it is fixed, a single failure could
Aug 27 12:05:09 cg1 cmcld: cause all nodes in the cluster to fail.
Aug 27 12:05:09 cg1 cmcld: Moving on to clear Second Dual Cluster Lock
Aug 27 12:06:25 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 27 12:06:25 cg1 cmcld: Check device, power, and cables.
Aug 27 13:07:40 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
Aug 27 13:07:40 cg1 cmcld: Check device, power, and cables.
Aug 27 13:07:41 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.3 for node cg1 (length = 512096).

[IG_2]

Hi,

Check if yout quorum disk is up and running.

[Steven E. Protter]

Shalom,

Few problems PSB

Aug 25 13:33:56 cg1 cmcld: Link level address on network interface lan0 has been updated from 0x0014c210830f to 0x00306e4b47a7.
Aug 25 13:33:56 cg1 cmcld: Proceeding with the new configuration.
Aug 25 13:33:57 cg1 cmcld: lan3 failed
....
Aug 25 13:34:00 cg1 cmcld: The new active cluster membership is: cg2(id=2), cg1(id=1)
Aug 25 13:35:14 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy


lan3 failed and the cluster lock disk seems unavailable. Run standard tests to see if the disk is really dead or the problem comes from another source.

Cluster lock disk does not involve any actual data.

You can change the cluster lock disk and then do a standard
cmquerycl
cmcheckconf
cmapplyconf

procedure and move on. There is a possibility that this process will fail because your cluster may have had multiple failures and serviceguard is designed to work its way through a single failure. If correctly configured.

cmviewcl -v
# lets make sure the cluster is active.


Good Advice:
Aug 27 13:07:40 cg1 cmcld: Check device, power, and cables.
Aug 27 13:07:41 cg1 cmclconfd[1734]: Updated file /var/adm/cmcluster/frdump.cmcld.3 for node cg1 (length = 512096).
SEP

[Sharma Sanjeev]

Hi

Please check Cluster Lock Disk with
dd command

Regards
Sanjeev

[Deepak Kr]

Hi,

>>>
Aug 27 09:42:21 cg1 cmcld: Unable to query the health of cluster lock disk /dev/dsk/c6t0d0: Device busy
>>>

check cluster lock disk at OS level and see if there is any issue with disk device here.

Also you can replace the disk if found faulty..

check following also:1) /var/opt/resmon/logs/event.log
2) /var/stm/logs/os/log*.cur <<< only need the latest
3) ioscan -fn
4) echo "map selall info;wait infolog" | cstm > /tmp/cstm.txt

[Sakui]

Below is the ouput from cmviewcl -v

CLUSTER STATUS
cl_a8965 up

NODE STATUS STATE
cg1 up running

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY up 0/1/2/0 lan0
PRIMARY up 0/3/1/0/6/0 lan1
STANDBY down 0/4/1/0/6/0 lan3

PACKAGE STATUS STATE AUTO_RUN NODE
pkg_a8965 up running disabled cg1

Policy_Parameters:
POLICY_NAME CONFIGURED_VALUE
Failover configured_node
Failback manual

Script_Parameters:
ITEM STATUS MAX_RESTARTS RESTARTS NAME
Service up 0 0 check_a8965
Service up 0 0 check_vgshare
Service up 0 0 check_x25
Subnet up 10.3.12.32

Node_Switching_Parameters:
NODE_TYPE STATUS SWITCHING NAME
Primary up enabled cg1 (current)
Alternate up enabled cg2

NODE STATUS STATE
cg2 up running

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY up 0/1/2/0 lan0
PRIMARY up 0/3/1/0/6/0 lan1
STANDBY down 0/4/1/0/6/0 lan3

this is the output from ioscan -fnC disk

Class I H/W Path Driver S/W State H/W Type Description
==========================================================================
disk 5 0/0/2/0.0.0.0 sdisk CLAIMED DEVICE TEAC DV-28E-C
/dev/dsk/c0t0d0 /dev/rdsk/c0t0d0
disk 0 0/1/1/0.0.0 sdisk CLAIMED DEVICE HP 36.4GST336754LC
/dev/dsk/c2t0d0 /dev/rdsk/c2t0d0
disk 1 0/1/1/0.1.0 sdisk CLAIMED DEVICE HP 36.4GST336754LC
/dev/dsk/c2t1d0 /dev/rdsk/c2t1d0
disk 2 0/1/1/1.1.0 sdisk CLAIMED DEVICE _NEC DVD_RW ND-3540A
/dev/dsk/c3t1d0 /dev/rdsk/c3t1d0
disk 3 0/3/1/0/4/0.0.0 sdisk CLAIMED DEVICE COMPAQ BD14687B52
/dev/dsk/c4t0d0 /dev/rdsk/c4t0d0
disk 4 0/4/1/0/4/0.0.0 sdisk CLAIMED DEVICE COMPAQ BD14687B52
/dev/dsk/c6t0d0 /dev/rdsk/c6t0d0

[Asif Sharif]

Hi sakui,

STANDBY down 0/4/1/0/6/0 lan3 is down(check LAN card, cables etc)

c6t0d0 Shown claimed in ioscan but check LED's of disk,STM logs, syslog and event log for more detail.

Regards,
Asif Sharif

[Deepak Kr]

test following also:

#lssf /dev/dsk/c6t0d0

and

# diskinfo -v /dev/rdsk/c6t0d0

IF you are under HP Service contract log a case with your local HP support centre.

[Sakui]

i have attached the cstm file