[root@lccsp01 lccsp_pkg]# cmviewcl -v CLUSTER STATUS lccsp up NODE STATUS STATE lccsp01 up running Cluster_Lock_LUN: DEVICE STATUS /dev/mapper/disk4basep1 up Network_Parameters: INTERFACE STATUS NAME PRIMARY up eth0 PRIMARY up eth2 PRIMARY up eth3 PRIMARY up eth1 PACKAGE STATUS STATE AUTO_RUN NODE lccsp_pkg up running enabled lccsp01 Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM STATUS MAX_RESTARTS RESTARTS NAME Service up 0 0 disk_lccsp_mon Subnet up 172.19.33.0 Subnet up 172.19.32.192 Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING NAME Primary up enabled lccsp01 (current) Alternate up enabled lccsp02 NODE STATUS STATE lccsp02 up running Cluster_Lock_LUN: DEVICE STATUS /dev/mapper/disk4basep1 unknown Network_Parameters: INTERFACE STATUS NAME PRIMARY up eth0 PRIMARY up eth2 PRIMARY up eth3 PRIMARY up eth1 [root@lccsp01 lccsp_pkg]# ifconfig -a eth0 Link encap:Ethernet HWaddr 00:1E:0B:BD:73:28 inet addr:172.19.32.193 Bcast:172.19.32.255 Mask:255.255.255.192 inet6 addr: fe80::21e:bff:febd:7328/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:3785 errors:0 dropped:0 overruns:0 frame:0 TX packets:1183 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:386616 (377.5 KiB) TX bytes:165404 (161.5 KiB) Interrupt:169 Memory:f2000000-f2012100 eth0:1 Link encap:Ethernet HWaddr 00:1E:0B:BD:73:28 inet addr:172.19.32.195 Bcast:172.19.32.255 Mask:255.255.255.192 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:169 Memory:f2000000-f2012100 eth1 Link encap:Ethernet HWaddr 00:1E:0B:BD:73:26 inet addr:172.19.245.9 Bcast:172.19.245.255 Mask:255.255.255.0 inet6 addr: fe80::21e:bff:febd:7326/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1012 errors:0 dropped:0 overruns:0 frame:0 TX packets:1172 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:149287 (145.7 KiB) TX bytes:166296 (162.3 KiB) Interrupt:177 Memory:f4000000-f4012100 eth1:1 Link encap:Ethernet HWaddr 00:1E:0B:BD:73:26 inet addr:172.19.245.66 Bcast:172.19.245.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:177 Memory:f4000000-f4012100 eth2 Link encap:Ethernet HWaddr 00:1E:0B:D1:43:7E inet addr:172.19.33.1 Bcast:172.19.33.63 Mask:255.255.255.192 inet6 addr: fe80::21e:bff:fed1:437e/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:4037 errors:0 dropped:0 overruns:0 frame:0 TX packets:4297 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:706087 (689.5 KiB) TX bytes:743326 (725.9 KiB) Interrupt:185 Memory:fa000000-fa012100 eth2:1 Link encap:Ethernet HWaddr 00:1E:0B:D1:43:7E inet addr:172.19.33.3 Bcast:172.19.33.63 Mask:255.255.255.192 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:185 Memory:fa000000-fa012100 eth3 Link encap:Ethernet HWaddr 00:1E:0B:D1:43:7C inet addr:10.10.10.1 Bcast:10.10.255.255 Mask:255.255.0.0 inet6 addr: fe80::21e:bff:fed1:437c/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1013 errors:0 dropped:0 overruns:0 frame:0 TX packets:1174 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:148692 (145.2 KiB) TX bytes:163981 (160.1 KiB) Interrupt:177 Memory:f8000000-f8012100 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:1714 errors:0 dropped:0 overruns:0 frame:0 TX packets:1714 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:290612 (283.8 KiB) TX bytes:290612 (283.8 KiB) sit0 Link encap:IPv6-in-IPv4 NOARP MTU:1480 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) [root@lccsp01 lccsp_pkg]# ifdown eth0 [root@lccsp01 lccsp_pkg]# cmviewcl -v CLUSTER STATUS lccsp up NODE STATUS STATE lccsp01 up running Cluster_Lock_LUN: DEVICE STATUS /dev/mapper/disk4basep1 up Network_Parameters: INTERFACE STATUS NAME PRIMARY down eth0 PRIMARY up eth2 PRIMARY up eth3 PRIMARY up eth1 PACKAGE STATUS STATE AUTO_RUN NODE lccsp_pkg failing failing enabled lccsp01 Policy_Parameters: POLICY_NAME CONFIGURED_VALUE Failover configured_node Failback manual Script_Parameters: ITEM STATUS MAX_RESTARTS RESTARTS NAME Service unknown Unlimited 0 disk_lccsp_mon Subnet up 172.19.33.0 Subnet down 172.19.32.192 Node_Switching_Parameters: NODE_TYPE STATUS SWITCHING NAME Primary up enabled lccsp01 (current) Alternate up enabled lccsp02 NODE STATUS STATE lccsp02 up running Cluster_Lock_LUN: DEVICE STATUS /dev/mapper/disk4basep1 unknown Network_Parameters: INTERFACE STATUS NAME PRIMARY up eth0 PRIMARY up eth2 PRIMARY up eth3 PRIMARY up eth1 /var/log/messages Mar 26 15:53:43 lccsp01 avahi-daemon[11650]: Withdrawing address record for 172.19.32.195 on eth0. Mar 26 15:53:43 lccsp01 avahi-daemon[11650]: Withdrawing address record for 172.19.32.193 on eth0. Mar 26 15:53:43 lccsp01 avahi-daemon[11650]: Leaving mDNS multicast group on interface eth0.IPv4 with address 172.19.32.193. Mar 26 15:53:43 lccsp01 avahi-daemon[11650]: iface.c: interface_mdns_mcast_join() called but no local address available. Mar 26 15:53:43 lccsp01 avahi-daemon[11650]: Interface eth0.IPv4 no longer relevant for mDNS. Mar 26 15:53:43 lccsp01 avahi-daemon[11650]: Withdrawing address record for fe80::21e:bff:febd:7328 on eth0. Mar 26 15:53:43 lccsp01 avahi-daemon[11650]: Leaving mDNS multicast group on interface eth0.IPv6 with address fe80::21e:bff:febd:7328. Mar 26 15:53:43 lccsp01 avahi-daemon[11650]: iface.c: interface_mdns_mcast_join() called but no local address available. Mar 26 15:53:43 lccsp01 avahi-daemon[11650]: Interface eth0.IPv6 no longer relevant for mDNS. Mar 26 15:53:44 lccsp01 cmnetd[12596]: Subnet 172.19.32.192 down Mar 26 15:53:44 lccsp01 cmcld[12580]: Subnet 172.19.32.192 in package lccsp_pkg is down. Mar 26 15:53:44 lccsp01 cmcld[12580]: Failing package lccsp_pkg on node lccsp01 due to subnet failure. Mar 26 15:53:44 lccsp01 cmcld[12580]: Request from node lccsp01 to fail package lccsp_pkg on node lccsp01. Mar 26 15:53:44 lccsp01 cmcld[12580]: Executing '/usr/local/cmcluster/conf/PACKAGES/lccsp_pkg/lccsp_pkg.sh stop' for package lccsp_pkg, as service PKG*23041. Mar 26 15:53:44 lccsp01 cmhaltserv: cmhaltserv disk_lccsp_mon Mar 26 15:53:44 lccsp01 logger: unishutdown: shutting down all of CA Universal_Job_Management Agent... Mar 26 15:53:44 lccsp01 logger: unishutdown: attempting to shut down CA Universal_Job_Management Agent 'sche' component... Mar 26 15:53:46 lccsp01 logger: unishutdown: attempting to shut down CA Universal_Job_Management Agent 'star' component... Mar 26 15:53:48 lccsp01 snmpd[8242]: Connection from UDP: [127.0.0.1]:33040 Mar 26 15:53:48 lccsp01 xinetd[8339]: EXIT: hacl-cfgudp status=0 pid=2401 duration=15(sec) Mar 26 15:53:48 lccsp01 logger: unishutdown: attempting to shut down CA Universal_Job_Management Agent 'emsrvc' component... Mar 26 15:53:50 lccsp01 logger: unishutdown: attempting to shut down CA Universal_Job_Management Agent 'opr' component... Mar 26 15:53:50 lccsp01 caiopr: SIGNAL 15 Mar 26 15:53:51 lccsp01 xinetd[8339]: START: hacl-cfgudp pid=3194 from=127.0.0.1 Mar 26 15:53:51 lccsp01 xinetd[8339]: START: hacl-cfg pid=3197 from=127.0.0.1 Mar 26 15:53:51 lccsp01 xinetd[3197]: USERID: hacl-cfg OTHER :root Mar 26 15:53:51 lccsp01 xinetd[8339]: EXIT: hacl-cfg status=0 pid=3197 duration=0(sec) Mar 26 15:53:52 lccsp01 syslogd 1.4.1: restart. Mar 26 15:53:55 lccsp01 logger: unishutdown: attempting to shut down CA Universal_Job_Management Agent 'cci' component... Mar 26 15:53:56 lccsp01 CCICLND[21232]: .CAICCI_I_3080 CCICLND exiting, sig=0 Mar 26 15:53:56 lccsp01 CCIRMTD[21233]: .CAICCI_W_2143 Remote server is shuting down... Mar 26 15:53:56 lccsp01 CCIRMTD[21233]: CCIRMTD ->Close_host[1] : Error <107> shutting 's channel 9 Mar 26 15:53:58 lccsp01 CAICCID[21230]: .CAICCI_I_1017 CCI shared memory has been destroyed Mar 26 15:53:58 lccsp01 CAICCID[21230]: .CAICCI_I_1029 CCI Logging file has been closed Mar 26 15:53:58 lccsp01 CAICCID[21230]: .CAICCI_I_1003 CCI has shutdown normally Mar 26 15:54:01 lccsp01 CCIRMTD[21233]: .CAICCI_I_2144 Remote server shutdown is complete. Mar 26 15:54:03 lccsp01 snmpd[8242]: Connection from UDP: [127.0.0.1]:33040 Mar 26 15:54:06 lccsp01 xinetd[8339]: EXIT: hacl-cfgudp status=0 pid=3194 duration=15(sec) Mar 26 15:54:12 lccsp01 cmmodnet: cmmodnet -r -i 172.19.33.3 172.19.33.0 Mar 26 15:54:12 lccsp01 avahi-daemon[11650]: Withdrawing address record for 172.19.33.3 on eth2. Mar 26 15:54:12 lccsp01 cmmodnet: cmmodnet -r -i 172.19.32.195 172.19.32.192 Mar 26 15:54:12 lccsp01 cmnetd[12596]: Failed to remove IP address 172.19.32.195 from 172.19.32.192 on eth0:1: Cannot assign requested address Mar 26 15:54:12 lccsp01 cmmodnet: cmmodnet -r -i 172.19.245.66 172.19.245.0 Mar 26 15:54:12 lccsp01 avahi-daemon[11650]: Withdrawing address record for 172.19.245.66 on eth1. Mar 26 15:54:12 lccsp01 multipathd: dm-25: umount map (uevent) Mar 26 15:54:12 lccsp01 multipathd: dm-23: umount map (uevent) Mar 26 15:54:12 lccsp01 multipathd: dm-22: umount map (uevent) Mar 26 15:54:12 lccsp01 multipathd: dm-19: umount map (uevent) Mar 26 15:54:12 lccsp01 multipathd: dm-18: umount map (uevent) Mar 26 15:54:12 lccsp01 multipathd: dm-17: umount map (uevent) Mar 26 15:54:12 lccsp01 multipathd: dm-21: umount map (uevent) Mar 26 15:54:12 lccsp01 multipathd: dm-20: umount map (uevent) Mar 26 15:54:12 lccsp01 multipathd: dm-24: umount map (uevent) Mar 26 15:54:12 lccsp01 multipathd: dm-17: remove map (uevent) Mar 26 15:54:12 lccsp01 multipathd: dm-18: remove map (uevent) Mar 26 15:54:12 lccsp01 multipathd: dm-19: remove map (uevent) Mar 26 15:54:12 lccsp01 multipathd: dm-20: remove map (uevent) Mar 26 15:54:12 lccsp01 multipathd: dm-21: remove map (uevent) Mar 26 15:54:12 lccsp01 multipathd: dm-22: remove map (uevent) Mar 26 15:54:12 lccsp01 multipathd: dm-23: remove map (uevent) Mar 26 15:54:12 lccsp01 multipathd: dm-24: remove map (uevent) Mar 26 15:54:12 lccsp01 multipathd: dm-25: remove map (uevent) Mar 26 15:54:12 lccsp01 cmcld[12580]: Service PKG*23041 terminated due to an exit(1). Mar 26 15:54:12 lccsp01 cmcld[12580]: Halted package lccsp_pkg on node lccsp01. Mar 26 15:54:12 lccsp01 cmcld[12580]: Package lccsp_pkg halt script exited with NO_RESTART. Mar 26 15:54:12 lccsp01 cmcld[12580]: Examine the file /usr/local/cmcluster/conf/PACKAGES/lccsp_pkg/lccsp_pkg.sh.log for more details. Mar 26 15:54:12 lccsp01 cmcld[12580]: Switching disabled on package lccsp_pkg. Mar 26 15:54:12 lccsp01 cmcld[12580]: Request from node lccsp01 to disable global switching for package lccsp_pkg. Mar 26 15:54:18 lccsp01 snmpd[8242]: Connection from UDP: [127.0.0.1]:33041 Mar 26 15:54:18 lccsp01 snmpd[8242]: Received SNMP packet(s) from UDP: [127.0.0.1]:33041 Mar 26 15:54:34 lccsp01 snmpd[8242]: Connection from UDP: [127.0.0.1]:33041 Mar 26 15:54:40 lccsp01 xinetd[8339]: START: hacl-cfgudp pid=4424 from=172.19.33.2 Mar 26 15:54:40 lccsp01 xinetd[8339]: START: hacl-cfg pid=4427 from=172.19.33.2 Mar 26 15:54:40 lccsp01 xinetd[4427]: USERID: hacl-cfg OTHER :root Mar 26 15:54:40 lccsp01 xinetd[8339]: EXIT: hacl-cfg status=0 pid=4427 duration=0(sec) Log serviceguard package ####### Node "lccsp01": Halting package at Thu Mar 26 15:53:44 CET 2009 ####### Mar 26 15:53:44 - Node "lccsp01": Halting service disk_lccsp_mon 26/03/09 15:53:44 : lccsp01 : DEBUT arret applications 26/03/09 15:53:44 STOP crontab en cours... 26/03/09 15:53:44 : lccsp01 : STOP crontab : STATUS OK 26/03/09 15:53:44 STOP TNG en cours... unishutdown: shutting down all of CA Universal_Job_Management Agent... unishutdown: attempting to shut down CA Universal_Job_Management Agent 'sche' component... unishutdown: attempting to shut down CA Universal_Job_Management Agent 'star' component... unishutdown: attempting to shut down CA Universal_Job_Management Agent 'emsrvc' component... unishutdown: attempting to shut down CA Universal_Job_Management Agent 'opr' component... unishutdown: attempting to shut down CA Universal_Job_Management Agent 'cci' component... CAICCI_S_0046 Command completed successfully 26/03/09 15:53:44 : lccsp01 : STOP tng : STATUS OK 26/03/09 15:54:04 STOP ORACLE en cours... Stopping listener LSNRCTL for Linux: Version 10.2.0.4.0 - Production on 26-MAR-2009 15:54:05 Copyright (c) 1991, 2007, Oracle. All rights reserved. Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521)) The command completed successfully Stopping Oracle databases ORACLE_HOME_LISTNER is not SET, unable to auto-stop Oracle Net Listener Usage: /app/oracle/product/10.2.0/bin/dbshut ORACLE_HOME Processing Database instance "SKCP": log file /app/oracle/product/10.2.0/shutdown.log 26/03/09 15:54:04 : lccsp01 : STOP oracle : STATUS OK 26/03/09 15:54:12 : lccsp01 : FIN arret applications Mar 26 15:54:12 - Node "lccsp01": Remove IP address 172.19.33.3 from subnet 172.19.33.0 Mar 26 15:54:12 - Node "lccsp01": Remove IP address 172.19.32.195 from subnet 172.19.32.192 cmmodnet: IP address 172.19.32.195 is not configured on the subnet. cmmodnet: Or IP address 172.19.32.195 is not valid. cmmodnet: Use the "ifconfig" command to check the configured IP addresses for the subnet. ERROR: Function remove_ip_address; Failed to remove 172.19.32.195 Mar 26 15:54:12 - Node "lccsp01": Remove IP address 172.19.245.66 from subnet 172.19.245.0 Mar 26 15:54:12 - Node "lccsp01": Unmounting filesystem on /ccsc Mar 26 15:54:12 - Node "lccsp01": Unmounting filesystem on /dbora/oexp Mar 26 15:54:12 - Node "lccsp01": Unmounting filesystem on /dbora/oadm Mar 26 15:54:12 - Node "lccsp01": Unmounting filesystem on /dbora/o05 Mar 26 15:54:12 - Node "lccsp01": Unmounting filesystem on /dbora/o04 Mar 26 15:54:12 - Node "lccsp01": Unmounting filesystem on /dbora/o03 Mar 26 15:54:12 - Node "lccsp01": Unmounting filesystem on /dbora/o02 Mar 26 15:54:12 - Node "lccsp01": Unmounting filesystem on /dbora/o01 Mar 26 15:54:12 - Node "lccsp01": Unmounting filesystem on /app/oracle Mar 26 15:54:12 - Node "lccsp01": Deactivating volume group vgora Attempting to deltag to vg vgora... deltag was successful on vg vgora. Mar 26 15:54:12 - Node "lccsp01": Deactivating volume group vgccs Attempting to deltag to vg vgccs... deltag was successful on vg vgccs. ###### Node "lccsp01": Package halted with ERROR at Thu Mar 26 15:54:12 CET 2009 ######