SERVICEGUARD PACKAGE FAILBACK ISSUE

[Tajammul_Ahmed]

Hi,

We have to HP-UNIX server in cluster environment, for OVO NNM and service desk and its using serviceguard.

I'm new to serviceguard, please help me on this, when we right click the virtual server and view the properties I'm getting both my physical server status is up, but under package configuration status on Node is down.
Attachment contain the screenshot for the same

Because of this I cannot start my failover on 1st server, all the packages have been moved to second server and itâ s not moving back to 1st server.
Please help me to solve this issue.

Thanks and best regards
Mohammed Tajammul Ahmed.

[Aneesh Mohan]

Hi Mohammed,

Please do below steps to enable package switching and start the package

# cmmodpkg -e ov-server
#cmrunpkg -n nodename ov-server

Thanks,
Aneesh

[whiteknight]

Hi Mohammed,

You can refer to Service Package and node switching concept info.

http://www1.itrc.hp.com/service/cki/docDisplay.do?docLocale=en&docId=emr_na-c00905742-2

WK

[Tajammul_Ahmed]

Hi,

Currently our packages are running on 2nd node, if I run this I'm getting the fallowing info.

[root@tshpov1],/#cmmodpkg -e ov-server
cmmodpkg : Warning: Package ov-server is already able to be switched.
cmmodpkg : Completed successfully on all packages specified.

[root@tshpov1],/#cmrunpkg -n tshpov1 ov-server
cmrunpkg : Node tshpov1 is currently unable to run package ov-server.
Check the syslog on node tshpov1 and pkg log files for more detailed information

does I need to shutdown the 2nd server if I want to run above commands?

also please let me know the location of syslog file and pk log file.

Thanks and best regards
Mohammed Tajammul Ahmed.

[Aneesh Mohan]

Hi Mohammed,

Please post the output of the below commands

#cmviewcl -v
#tail -n 16 /var/adm/syslog/syslog.log

Normally Pkg log will be in the location /etc/cmcluster/pkgname/

Thanks,
Aneesh

[Tajammul_Ahmed]

Hi

Here is the requested logs


HP-UX tshpov1 B.11.11 U 9000/800 (ta)

login: root
Password:
Please wait...checking for disk quotas
(c)Copyright 1983-2000 Hewlett-Packard Co., All Rights Reserved.
(c)Copyright 1979, 1980, 1983, 1985-1993 The Regents of the Univ. of California
(c)Copyright 1980, 1984, 1986 Novell, Inc.
(c)Copyright 1986-1992 Sun Microsystems, Inc.
(c)Copyright 1985, 1986, 1988 Massachusetts Institute of Technology
(c)Copyright 1989-1993 The Open Software Foundation, Inc.
(c)Copyright 1986 Digital Equipment Corp.
(c)Copyright 1990 Motorola, Inc.
(c)Copyright 1990, 1991, 1992 Cornell University
(c)Copyright 1989-1991 The University of Maryland
(c)Copyright 1988 Carnegie Mellon University
(c)Copyright 1991-2000 Mentat Inc.
(c)Copyright 1996 Morning Star Technologies, Inc.
(c)Copyright 1996 Progressive Systems, Inc.
(c)Copyright 1991-2000 Isogon Corporation, All Rights Reserved.


RESTRICTED RIGHTS LEGEND
Use, duplication, or disclosure by the U.S. Government is subject to
restrictions as set forth in sub-paragraph (c)(1)(ii) of the Rights in
Technical Data and Computer Software clause in DFARS 252.227-7013.

Hewlett-Packard Company
3000 Hanover Street
Palo Alto, CA 94304 U.S.A.

Rights for non-DOD U.S. Government Departments and Agencies are as set
forth in FAR 52.227-19(c)(1,2).
You have mail.



Value of TERM has been set to "ansi".
WARNING: YOU ARE SUPERUSER !!

[root@tshpov1],/#cmmodpkg -e ov-server
cmmodpkg : Warning: Package ov-server is already able to be switched.
cmmodpkg : Completed successfully on all packages specified.
[root@tshpov1],/#cmrunpkg -n tshpov1 ov-server
cmrunpkg : Node tshpov1 is currently unable to run package ov-server.
Check the syslog on node tshpov1 and pkg log files for more detailed information
.
[root@tshpov1],/#tail -n 16 /var/adm/syslog/syslog.log
Nov 28 11:32:29 tshpov1 cmcld: Disabled node tshpov1 from running package ov-ser
ver.
Nov 28 11:32:29 tshpov1 cmcld: Executing '/etc/cmcluster/ov-server/ov-server.cnt
[root@tshpov1],/#tail -n 16 /var/adm/syslog/syslog.log
Nov 28 11:32:29 tshpov1 cmcld: Disabled node tshpov1 from running package ov-ser
ver.
Nov 28 11:32:29 tshpov1 cmcld: Executing '/etc/cmcluster/ov-server/ov-server.cnt
l stop' for package ov-server, as service PKG*35329.
Nov 28 11:32:29 tshpov1 cmcld: Service ov-server_service terminated due to an ex
it(1).
Nov 28 11:32:29 tshpov1 CM-ov-server[18102]: cmhaltserv ov-server_service
Nov 28 11:32:30 tshpov1 su: + tty?? root-oracle
Nov 28 11:32:30 tshpov1 CM-ov-server[18339]: cmmodnet -r -i 91.103.144.170 91.10
3.144.160
Nov 28 11:32:33 tshpov1 LVM[18384]: vgchange -a n ov-vg
Nov 28 11:32:33 tshpov1 cmcld: Service PKG*35329 terminated due to an exit(0).
Nov 28 11:32:30 tshpov1 su: + tty?? root-oracle
Nov 28 11:32:33 tshpov1 above message repeats 2 times
Nov 28 11:32:33 tshpov1 cmcld: Halted package ov-server on node tshpov1.
Nov 28 11:33:16 tshpov1 cmomd[19346]: Request to connect as root from mohdt@192.
168.2.126.
Nov 28 11:33:18 tshpov1 cmomd[19346]: Request to start package ov-server on node
tshpov2.
Nov 28 11:33:18 tshpov1 cmcld: Request from mohdt on node 192.168.2.126 connecte
d to tshpov1 as root via cmomd(pid 19346) to start package ov-server on node tsh
pov2.
Nov 28 11:34:31 tshpov1 cmcld: (tshpov2) Started package ov-server on node tshpo
v2.
Nov 28 11:34:31 tshpov1 cmomd[19346]: Successfully started package ov-server on
node tshpov2.
[root@tshpov1],/#

[Aneesh Mohan]

I hope package ov-server is running on node tshpov2 now ,if you want to move the package to node tshpov1,do the below steps

To move package ov-server to node tshpov1, do the following:

#cmhaltpkg ov-server
#cmmodpkg -e ov-server (to enable package switching, if desired)
#cmrunpkg -n tshpov1 ov-server

Please post cmviewcl -v o/p ,still you have any issues


Aneesh

[Tajammul_Ahmed]

Hi Aneesh,

I halt the package on tshpov2, and started on tshpov1 as per your recommendation,

Still I have the same Issue, here is the requested info.


From TSHPOV1
[root@tshpov1],/#cmhaltpkg ov-server
One or more packages has been halted and will not be started automatically. To s
tart these packages, enable AUTO_RUN via cmmodpkg -e .
cmhaltpkg : Completed successfully on all packages specified.
[root@tshpov1],/#cmmodpkg -e ov-server
cmmodpkg : Completed successfully on all packages specified.
[root@tshpov1],/#cmrunpkg -n tshpov1 ov-server
cmrunpkg : Node tshpov1 is currently unable to run package ov-server.
Check the syslog on node tshpov1 and pkg log files for more detailed information

[root@tshpov1],/#cmviewcl -v

CLUSTER STATUS
openview up

NODE STATUS STATE
tshpov1 up running

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY up 0/1/2/0 lan0
PRIMARY up 0/5/2/1 lan5
PRIMARY up 0/5/2/0 lan4
STANDBY up 0/4/2/0 lan2
STANDBY up 0/1/2/1 lan1

NODE STATUS STATE
tshpov2 up running

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY up 0/1/2/0 lan0
PRIMARY up 0/5/2/1 lan5
STANDBY up 0/4/2/0 lan2
STANDBY up 0/1/2/1 lan1
PRIMARY up 0/5/2/0 lan4

PACKAGE STATUS STATE AUTO_RUN NODE
ov-server up running enabled tshpov2

Policy_Parameters:
POLICY_NAME CONFIGURED_VALUE
Failover configured_node
Failback automatic

Script_Parameters:
ITEM STATUS MAX_RESTARTS RESTARTS NAME
Service up 1 0 ov-server_service
Subnet up 91.103.144.160

Node_Switching_Parameters:
NODE_TYPE STATUS SWITCHING NAME
Primary up disabled tshpov1
Alternate up enabled tshpov2 (current)

[Aneesh Mohan]

Can you post syslog output ...


#tail -n 50 /var/adm/syslog/syslog.log


Aneesh

[Tajammul_Ahmed]

Hi Here is the output.

[root@tshpov1],/#tail -n 50 /var/adm/syslog/syslog.log
Nov 28 12:34:56 tshpov1 CM-CMD[14124]: cmrunpkg -n tshpov1 ov-server
Nov 28 12:34:56 tshpov1 CM-CMD[14124]: Request from root on node tshpov1 to star
t the package
Nov 28 12:35:39 tshpov1 cmcld: (tshpov2) Started package ov-server on node tshpo
v2.
Nov 28 12:35:39 tshpov1 cmcld: Package ov-server cannot run on this node because
switching has been disabled for this node
Nov 28 12:41:26 tshpov1 cmclconfd[16042]: Querying volume group /dev/vg00 for no
de tshpov2
Nov 28 12:41:26 tshpov1 cmclconfd[16042]: Volume group /dev/vg00 has no cluster
attributes
Nov 28 12:41:26 tshpov1 cmclconfd[16042]: Querying volume group /dev/ov-vg for n
ode tshpov2
Nov 28 12:41:28 tshpov1 cmclconfd[16045]: Querying volume group /dev/vg00 for no
de tshpov2
Nov 28 12:41:28 tshpov1 cmclconfd[16045]: Volume group /dev/vg00 has no cluster
attributes
Nov 28 12:41:28 tshpov1 cmclconfd[16045]: Querying volume group /dev/ov-vg for n
ode tshpov2
Nov 28 12:41:28 tshpov1 cmclconfd[16045]: Volume group /dev/ov-vg is configured
exclusive
Nov 28 12:41:33 tshpov1 cmclconfd[16095]: Querying volume group /dev/vg00 for no
de tshpov2
Nov 28 12:41:33 tshpov1 cmclconfd[16095]: Volume group /dev/vg00 has no cluster
attributes
Nov 28 12:41:33 tshpov1 cmclconfd[16095]: Querying volume group /dev/ov-vg for n
ode tshpov2
Nov 28 12:41:33 tshpov1 cmclconfd[16095]: Volume group /dev/ov-vg is configured
exclusive
Nov 28 12:45:01 tshpov1 cmclconfd[17235]: Querying volume group /dev/vg00 for no
de tshpov2
Nov 28 12:45:01 tshpov1 cmclconfd[17235]: Volume group /dev/vg00 has no cluster
attributes
Nov 28 12:45:01 tshpov1 cmclconfd[17235]: Querying volume group /dev/ov-vg for n
ode tshpov2
Nov 28 12:45:01 tshpov1 cmclconfd[17235]: Volume group /dev/ov-vg is configured
exclusive
Nov 28 12:46:07 tshpov1 cmomd[17672]: Request to connect as root from mohdt@192.
168.2.126.
Nov 28 12:46:08 tshpov1 cmomd[17672]: Request to modify node package switching f
or ov-server on cluster openview.
Nov 28 12:46:09 tshpov1 cmcld: Request from mohdt on node 192.168.2.126 connecte
d to tshpov1 as root via cmomd(pid 17672) to enable node tshpov1 for switching o
f package ov-server.
Nov 28 12:46:09 tshpov1 cmcld: Request from node tshpov2 to enable node switchin
g for package ov-server on node tshpov1.
Nov 28 12:46:09 tshpov1 cmomd[17672]: Successfully enabled package ov-server fro
m switching on node tshpov1.
Nov 28 12:49:17 tshpov1 cmcld: (tshpov2) Halted package ov-server on node tshpov
2.
Nov 28 12:49:17 tshpov1 cmcld: Request from node tshpov2 to start package ov-ser
ver on node tshpov1.
Nov 28 12:49:17 tshpov1 cmcld: Executing '/etc/cmcluster/ov-server/ov-server.cnt
l start' for package ov-server, as service PKG*35329.
Nov 28 12:49:17 tshpov1 LVM[19061]: vgchange -a e ov-vg
Nov 28 12:49:19 tshpov1 CM-ov-server[19132]: cmmodnet -a -i 91.103.144.170 91.10
3.144.160
Nov 28 12:49:19 tshpov1 su: + tty?? root-oracle
Nov 28 12:49:28 tshpov1 CM-ov-server[19423]: cmrunserv -r 1 ov-server_service >>
/etc/cmcluster/ov-server/ov-server.cntl.log 2>&1 /etc/cmcluster/ov-server/ov-se
rver.mon
Nov 28 12:49:28 tshpov1 cmcld: Service PKG*35329 terminated due to an exit(0).
Nov 28 12:49:28 tshpov1 cmcld: Started package ov-server on node tshpov1.
Nov 28 12:49:27 tshpov1 su: + tty?? root-oracle
Nov 28 12:49:28 tshpov1 above message repeats 2 times
Nov 28 12:49:28 tshpov1 cmcld: Service ov-server_service terminated due to an ex
it(1).
Nov 28 12:49:28 tshpov1 cmcld: Automatically restarted service ov-server_service
for the 1st time after failure.
Nov 28 12:49:28 tshpov1 cmcld: Service ov-server_service in package ov-server ha
s gone down.
Nov 28 12:49:28 tshpov1 cmcld: Disabled node tshpov1 from running package ov-ser
ver.
Nov 28 12:49:28 tshpov1 cmcld: Executing '/etc/cmcluster/ov-server/ov-server.cnt
l stop' for package ov-server, as service PKG*35329.
Nov 28 12:49:28 tshpov1 cmcld: Service ov-server_service terminated due to an ex
it(1).
Nov 28 12:49:28 tshpov1 CM-ov-server[19503]: cmhaltserv ov-server_service
Nov 28 12:49:30 tshpov1 su: + tty?? root-oracle
Nov 28 12:49:36 tshpov1 CM-ov-server[19793]: cmmodnet -r -i 91.103.144.170 91.10
3.144.160
Nov 28 12:49:39 tshpov1 LVM[19915]: vgchange -a n ov-vg
Nov 28 12:49:39 tshpov1 cmcld: Service PKG*35329 terminated due to an exit(0).
Nov 28 12:49:35 tshpov1 su: + tty?? root-oracle
Nov 28 12:49:39 tshpov1 above message repeats 2 times
Nov 28 12:49:39 tshpov1 cmcld: Halted package ov-server on node tshpov1.
Nov 28 12:49:40 tshpov1 cmcld: Package ov-server cannot run on this node because
switching has been disabled for this node