CFS Package Creation

[nasmel]

Hi


I just created a service gurad cluster and i was trying to SG CFS package
I Just follwed as below
#vxinstall
#cfscluster config -t 900
#cfscluster start
At this point my second node always reboots
OS Version : HP UX 11.31
Service guards :T2777CB A.02.01 HP Serviceguard Cluster File System for RAC



PLease check the below logs

SYSLOG from SRVTST1




Nov 22 23:22:38 SRVTST1 cmvxd[11670]: Acknowledging cluster status sequence 1
Nov 22 23:22:38 SRVTST1 cmvxd[11670]: package SG-CFS-pkg status: node 1 starting
, node 2 starting
Nov 22 23:22:40 SRVTST1 cmcld[10833]: Member SRVTST2 seems unhealthy, not receiv
ing heartbeats from it.
Nov 22 23:22:43 SRVTST1 cmcld[10833]: Closing route 10.1.2.81:5300 on fd 36 to S
RVTST2: Software caused connection abort
Nov 22 23:22:43 SRVTST1 cmcld[10833]: Closing route 192.168.0.81:5300 on fd 37 t
o SRVTST2: Software caused connection abort
Nov 22 23:22:43 SRVTST1 cmcld[10833]: Requesting cluster reformation for synchro
nization
Nov 22 23:22:43 SRVTST1 cmcld[10833]: Membership: membership at 1 is REFORMING (
coordinator 1) includes: 1 2 excludes:
Nov 22 23:22:44 SRVTST1 cmcld[10833]: Timed out SRVTST2 (2)
Nov 22 23:22:44 SRVTST1 cmcld[10833]: Lost heartbeat to SRVTST2
Nov 22 23:22:44 SRVTST1 cmcld[10833]: Resolving quorum with members SRVTST1
Nov 22 23:22:44 SRVTST1 cmdisklockd[10836]: Obtaining cluster lock device /dev/d
sk/c9t0d1
Nov 22 23:22:44 SRVTST1 cmdisklockd[10836]: Successfully obtained the lock at /d
ev/dsk/c9t0d1!
Nov 22 23:22:45 SRVTST1 cmcld[10833]: Sending file $SGRUN/frdump.cmcld.8 (512096
bytes) to file assistant daemon.
Nov 22 23:22:45 SRVTST1 cmfileassistd[10841]: Updated file /var/adm/cmcluster/fr
dump.cmcld.8 (length = 512096).
Nov 22 23:22:48 SRVTST1 vmunix: GAB INFO V-15-1-20026 Port a registration waitin
g for seed port membership
Nov 22 23:22:48 SRVTST1 vmunix: GAB INFO V-15-1-20026 Port b registration waitin
g for seed port membership
Nov 22 23:23:00 SRVTST1 cmdisklockd[10836]: Clearing cluster lock device /dev/ds
k/c9t0d1
Nov 22 23:23:00 SRVTST1 cmcld[10833]: Membership: membership at 2 is FORMED (coo
rdinator 1) includes: 1 excludes: 2
Nov 22 23:23:00 SRVTST1 cmcld[10833]: The following node(s)
the cluster.
Nov 22 23:23:00 SRVTST1 cmcld[10833]: 1 nodes have formed a
e #2
Nov 22 23:23:00 SRVTST1 cmvxd[11670]: cluster status sequen
2 down
Nov 22 23:23:00 SRVTST1 cmcld[10833]: The new active cluste
ST1(id=1)
Nov 22 23:23:00 SRVTST1 cmdisklockd[10836]: Cluster lock cl
v/dsk/c9t0d1
Nov 22 23:23:00 SRVTST1 cmvxd[11670]: Acknowledging cluster
Nov 22 23:23:00 SRVTST1 cmcld[10833]: Cluster CDB version 5
on 5
Nov 22 23:23:00 SRVTST1 cmcld[10833]: Received clear reply
Nov 22 23:23:00 SRVTST1 cmvxd[11670]: package SG-CFS-pkg st
, node 2 not owned
Nov 22 23:23:01 SRVTST1 cmcld[10833]: Sending file $SGRUN/f
bytes) to file assistant daemon.
Nov 22 23:23:01 SRVTST1 cmfileassistd[10841]: Updated file
dump.cmcld.9 (length = 45160).




/etc/cmcluster/cfs/SG-CFS-pkg.log





11/22/10 23:22:37 cmversion
11/22/10 23:22:37 ########### Node "SRVTST1": Starting package ###########
11/22/10 23:22:37 Starting service SG-CFS-vxconfigd
11/22/10 23:22:37 cmrunserv SG-CFS-vxconfigd >> /etc/cmcluster/cfs/SG-CFS-pkg.log 2>&1 /etc/cmcluster/cfs/SG-CFS-vxconfigd.sh
11/22/10 23:22:37 Cleaning up any old GAB/LLT configuration
11/22/10 23:22:37 /etc/cmcluster/cfs/vx-modules clean
11/22/10 23:22:37 Cleaning up old LLT/GAB
11/22/10 23:22:37 Monitoring vxconfigd (pid= 401) every 20 secs
Stopping gab..
Stopping gab.. Done
Stopping llt..
Stopping llt.. Done
11/22/10 23:22:38 rm -f /etc/llttab /etc/llthosts /etc/gabtab
11/22/10 23:22:38 Starting service SG-CFS-cmvxpingd
11/22/10 23:22:38 cmrunserv SG-CFS-cmvxpingd >> /etc/cmcluster/cfs/SG-CFS-pkg.log 2>&1 /usr/lbin/cmvxpingd -t 174
11/22/10 23:22:38 rm -f /var/adm/cmcluster/cmvxd.socket
11/22/10 23:22:38 Starting service SG-CFS-cmvxd
11/22/10 23:22:38 cmrunserv SG-CFS-cmvxd >> /etc/cmcluster/cfs/SG-CFS-pkg.log 2>&1 /usr/lbin/cmvxd run -s /var/adm/cmcluster/c
mvxd.socket -t 174
11/22/10 23:22:38 Creating LLT configuration
11/22/10 23:22:38 mktemp -d /etc
11/22/10 23:22:38 touch /etc/011675
11/22/10 23:22:38 chmod 644 /etc/011675
11/22/10 23:22:38 chmod 444 /etc/011675
11/22/10 23:22:38 mv /etc/011675 /etc/llttab
11/22/10 23:22:38 touch -r /etc/cmcluster/cfs/.SG-CFS-pkg.ref /etc/llttab
11/22/10 23:22:38 Creating GAB configuration
11/22/10 23:22:38 mktemp -d /etc
11/22/10 23:22:38 touch /etc/011692
11/22/10 23:22:38 chmod 644 /etc/011692
11/22/10 23:22:38 chmod 444 /etc/011692
11/22/10 23:22:38 mv /etc/011692 /etc/gabtab
11/22/10 23:22:38 touch -r /etc/cmcluster/cfs/.SG-CFS-pkg.ref /etc/gabtab
11/22/10 23:22:38 chmod 544 /etc/gabtab
11/22/10 23:22:38 Creating initial LLT hosts file
11/22/10 23:22:38 mktemp -d /etc
11/22/10 23:22:38 touch /etc/011712
11/22/10 23:22:38 chmod 644 /etc/011712
11/22/10 23:22:38 chmod 444 /etc/011712
11/22/10 23:22:38 mv /etc/011712 /etc/llthosts
11/22/10 23:22:38 touch -r /etc/cmcluster/cfs/.SG-CFS-pkg.ref /etc/llthosts
11/22/10 23:22:38 Starting Veritas stack
11/22/10 23:22:38 /etc/cmcluster/cfs/vx-modules start
11/22/10 23:22:38 Starting LLT
11/22/10 23:22:38 /sbin/init.d/llt start
11/22/10 23:22:48 Starting GAB
11/22/10 23:22:48 /sbin/init.d/gab start
11/22/10 23:22:48 Starting VXFEN







SRVTST2





syslog





Nov 22 23:17:05 SRVTST2 SQLAnywhere(veritas_dbms3_srv-tst-): Starting checkpoint of "vxdbms" (vxdbms.db) at Mon Nov 22 2010 23
:17
Nov 22 23:17:05 SRVTST2 SQLAnywhere(veritas_dbms3_srv-tst-): Finished checkpoint of "vxdbms" (vxdbms.db) at Mon Nov 22 2010 23
:17
Nov 22 23:20:55 SRVTST2 sgcfscmd[9991]: (pts/ta) cfscluster unconfig
Nov 22 23:22:20 SRVTST2 cmclconfd[10055]: Updated file /etc/cmcluster/cmclconfig.tmp for node SRVTST1 (length = 21664).
Nov 22 23:22:20 SRVTST2 cmclconfd[10055]: Updated file /etc/cmcluster/cmclconfig.tmp for node SRVTST1 (length = 0).
Nov 22 23:22:20 SRVTST2 cmcld[9444]: Member Timeout is 14.00 seconds.
Nov 22 23:22:20 SRVTST2 cmcld[9444]: Max reformation duration is 22.00 seconds.
Nov 22 23:22:20 SRVTST2 cmcld[9444]: Online Config - Successfully added package SG-CFS-pkg with id 96769.
Nov 22 23:22:20 SRVTST2 cmcld[9444]: Sending file $SGCONF/cmclconfig (14044 bytes) to file assistant daemon.
Nov 22 23:22:20 SRVTST2 cmfileassistd[9452]: Updated file /etc/cmcluster/cmclconfig (length = 14044).
Nov 22 23:22:20 SRVTST2 cmnetd[9462]: Auto Failback is enabled.
Nov 22 23:22:20 SRVTST2 cmclconfd[10055]: Added package configuration for package SG-CFS-pkg (1-5-0-1-0-0-0-0-0-0-0-0)
Nov 22 23:22:20 SRVTST2 cmclconfd[10055]: Executing configuration change callout for node SRVTST1.
Nov 22 23:22:20 SRVTST2 cmcld[9444]: Online Config - Modified max reformation from 17800000 to 22000000
Nov 22 23:23:34 SRVTST2 cmcld[9444]: Request from node SRVTST1 to enable global switching for package SG-CFS-pkg.
Nov 22 23:23:34 SRVTST2 cmcld[9444]: Request from node SRVTST1 to start package SG-CFS-pkg on node SRVTST2.
Nov 22 23:23:34 SRVTST2 cmcld[9444]: Executing '/etc/cmcluster/cfs/SG-CFS-pkg.sh start' for package SG-CFS-pkg, as service PK
G*96769.
Nov 22 23:23:34 SRVTST2 cmserviced[9448]: Request to perform run service PKG*96769
Nov 22 23:23:39 SRVTST2 syslog: cmrunserv SG-CFS-vxconfigd >> /etc/cmcluster/cfs/SG-CFS-pkg.log 2>&1 /etc/cmcluster/cfs/SG-CFS
-vxconfigd.sh
Nov 22 23:23:39 SRVTST2 cmserviced[9448]: Request to perform run service SG-CFS-vxconfigd
Nov 22 23:23:39 SRVTST2 syslog: cmrunserv SG-CFS-cmvxpingd >> /etc/cmcluster/cfs/SG-CFS-pkg.log 2>&1 /usr/lbin/cmvxpingd -t 17
4
Nov 22 23:23:39 SRVTST2 cmserviced[9448]: Request to perform run service SG-CFS-cmvxpingd
Nov 22 23:23:39 SRVTST2 syslog: cmrunserv SG-CFS-cmvxd >> /etc/cmcluster/cfs/SG-CFS-pkg.log 2>&1 /usr/lbin/cmvxd run -s /var/a
dm/cmcluster/cmvxd.socket -t 174
Nov 22 23:23:39 SRVTST2 cmserviced[9448]: Request to perform run service SG-CFS-cmvxd
Nov 22 23:23:39 SRVTST2 cmserviced[9448]: Service PKG*96769 terminated due to an exit(1).
Nov 22 23:23:39 SRVTST2 vmunix: Halting SRVTST2 to preserve data integrity
Nov 22 23:23:39 SRVTST2 vmunix: Reason: A crucial package failed
Nov 22 23:23:39 SRVTST2 cmcld[9444]: force: fd=18, safety_active=1, safety_enabled=1, toc_forced=1
Nov 22 23:23:39 SRVTST2 cmcld[9444]: Package SG-CFS-pkg run script exited with NO_RESTART.
Nov 22 23:23:39 SRVTST2 cmcld[9444]: Package SG-CFS-pkg run script failed.
Nov 22 23:23:39 SRVTST2 cmcld[9444]: Examine the file /etc/cmcluster/cfs/SG-CFS-pkg.log for more details.
Nov 22 23:23:39 SRVTST2 cmcld[9444]: Failed node in response to failure of package SG-CFS-pkg.
Nov 22 23:23:39 SRVTST2 cmcld[9444]: Enabled safety time to enforce FAIL FAST
Nov 22 23:23:39 SRVTST2 cmcld[9444]: Halting SRVTST2 to preserve data integrity
Nov 22 23:23:39 SRVTST2 cmcld[9444]: Reason: A crucial package failed
Nov 22 23:23:39 SRVTST2 cmcld[9444]: force: fd=18, safety_active=1, safety_enabled=1, toc_forced=1
Nov 22 23:23:39 SRVTST2 cmclconfd[9443]: The Serviceguard daemon, cmcld[9444], exited normally.
Nov 22 23:23:39 SRVTST2 cmvxd[10220]: cmvxd running (pid 10220)





/etc/cmcluster/cfs/SG-CFS-pkg.log








11/22/10 23:23:39 cmversion
11/22/10 23:23:39 ########### Node "SRVTST2": Starting package ###########
11/22/10 23:23:39 Starting service SG-CFS-vxconfigd
11/22/10 23:23:39 cmrunserv SG-CFS-vxconfigd >> /etc/cmcluster/cfs/SG-CFS-pkg.log 2>&1 /etc/cmcluster/cfs/SG-CFS-vxconfigd.sh
11/22/10 23:23:39 Cleaning up any old GAB/LLT configuration
11/22/10 23:23:39 /etc/cmcluster/cfs/vx-modules clean
11/22/10 23:23:39 Cleaning up old LLT/GAB
11/22/10 23:23:39 Monitoring vxconfigd (pid= 402) every 20 secs
11/22/10 23:23:39 ERROR: Unable to reset GAB
11/22/10 23:23:39 rm -f /etc/llttab /etc/llthosts /etc/gabtab
11/22/10 23:23:39 Starting service SG-CFS-cmvxpingd
11/22/10 23:23:39 cmrunserv SG-CFS-cmvxpingd >> /etc/cmcluster/cfs/SG-CFS-pkg.log 2>&1 /usr/lbin/cmvxpingd -t 174
11/22/10 23:23:39 rm -f /var/adm/cmcluster/cmvxd.socket
11/22/10 23:23:39 Starting service SG-CFS-cmvxd
11/22/10 23:23:39 cmrunserv SG-CFS-cmvxd >> /etc/cmcluster/cfs/SG-CFS-pkg.log 2>&1 /usr/lbin/cmvxd run -s /var/adm/cmcluster/c
mvxd.socket -t 174
11/22/10 23:23:39 /sbin/init.d/llt start (exit=1)
11/22/10 23:23:39 ERROR: Failed to start LLT
11/22/10 23:23:39 ERROR: Could not start veritas stack
11/22/10 23:23:39 ########### Node "SRVTST2": Package script failed ###########





11/22/10 23:23:39 Creating LLT configuration
11/22/10 23:23:39 mktemp -d /etc
11/22/10 23:23:39 touch /etc/010225
11/22/10 23:23:39 chmod 644 /etc/010225
11/22/10 23:23:39 chmod 444 /etc/010225
11/22/10 23:23:39 mv /etc/010225 /etc/llttab
11/22/10 23:23:39 touch -r /etc/cmcluster/cfs/.SG-CFS-pkg.ref /etc/llttab
11/22/10 23:23:39 Creating GAB configuration
11/22/10 23:23:39 mktemp -d /etc
11/22/10 23:23:39 touch /etc/010242
11/22/10 23:23:39 chmod 644 /etc/010242
11/22/10 23:23:39 chmod 444 /etc/010242
11/22/10 23:23:39 mv /etc/010242 /etc/gabtab
11/22/10 23:23:39 touch -r /etc/cmcluster/cfs/.SG-CFS-pkg.ref /etc/gabtab
11/22/10 23:23:39 chmod 544 /etc/gabtab
11/22/10 23:23:39 Creating initial LLT hosts file
11/22/10 23:23:39 mktemp -d /etc
11/22/10 23:23:39 touch /etc/010262
11/22/10 23:23:39 chmod 644 /etc/010262
11/22/10 23:23:39 chmod 444 /etc/010262
11/22/10 23:23:39 mv /etc/010262 /etc/llthosts
11/22/10 23:23:39 touch -r /etc/cmcluster/cfs/.SG-CFS-pkg.ref /etc/llthosts
11/22/10 23:23:39 Starting Veritas stack
11/22/10 23:23:39 /etc/cmcluster/cfs/vx-modules start
11/22/10 23:23:39 Starting LLT
11/22/10 23:23:39 /sbin/init.d/llt start
LLT lltconfig ERROR V-14-2-15040 node ID is already set, use -o to override




Please help

[Shibin_2 · Accepted solution]

How about your cluster status without package? Is it working properly without package?

Please go through this thread.

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1382005

[nasmel]

Thanks for your replay

Yes,My cluster is working fine with out packages.

# cmviewcl

CLUSTER STATUS
TSTCLU up

NODE STATUS STATE
SRVTST1 up running
SRVTST2 up running

MULTI_NODE_PACKAGES

PACKAGE STATUS STATE AUTO_RUN SYSTEM
SG-CFS-pkg down halted disabled yes



#
# swlist | grep -i service
ProviderSvcsBase C.03.00.06.01 Provider Services Base
T2777CB A.02.01 HP Serviceguard Cluster File em for RAC
# swlist | grep -i veritas
Base-VxFS-50 B.05.00.02 Veritas File System Bundle 5 r HP-UX
Base-VxTools-50 B.05.00.02 VERITAS Infrastructure Bundl for HP-UX
Base-VxVM-50 B.05.00.02 Base VERITAS Volume Manager e 5.0 for HP-UX
#

[Shibin_2]

Have you checked other patches for CFS ?

swlist -l patch | grep -i cfs

and

swlist -l file | grep vxfsckd

I think, it could be something to do with your patching.

[nasmel]

both servers have same set of patches installed

SRVTST1
swlist -l patch | grep -i cfs
# CIFS-CFSM A.02.04.01 HP CIFS File System Module
# CIFS-CFSM.CFSM-KRN A.02.04.01 CFSM DLKM Kernel Modules
# CIFS-CFSM.CFSM-MAN A.02.04.01 UNIX MAN pages for CFSM
# CIFS-CFSM.CFSM-RUN A.02.04.01 CFSM programs
# Cluster-CFS A.11.19.00 HP SG Cluster CFS SD Product
# Cluster-CFS.CFS-ADMIN A.11.19.00 Cluster File System Admin SD Product
# Cluster-CFS.CFS-MAN A.11.19.00 CFS MAN pages
# Package-CVM-CFS A.11.19.00 HP SG Cluster CVM CFS SD Product
# Package-CVM-CFS.CM-CVM-CFS A.11.19.00 CVM CFS Package Fileset
# Package-CVM-CFS.CM-CVM-CFS-COM A.11.19.00 CVM CFS Package Fileset


# swlist -l file | grep vxfsckd
Package-CVM-CFS.CM-CVM-CFS-COM: /etc/cmcluster/cfs/SG-CFS-vxfsckd.sh
VRTSvxfs.VXFS-RUN: /opt/VRTS/bin/vxfsckd
VRTSvxfs.VXFS-RUN: /usr/lbin/fs/vxfs5.0/vxfsckd
VRTSvxfs.VXFS-RUN: /usr/sbin/vxfsckd
#




SRVTST2



# swlist -l patch | grep -i cfs
# CIFS-CFSM A.02.04.01 HP CIFS File System Module
# CIFS-CFSM.CFSM-KRN A.02.04.01 CFSM DLKM Kernel Modules
# CIFS-CFSM.CFSM-MAN A.02.04.01 UNIX MAN pages for CFSM
# CIFS-CFSM.CFSM-RUN A.02.04.01 CFSM programs
# Cluster-CFS A.11.19.00 HP SG Cluster CFS SD Product
# Cluster-CFS.CFS-ADMIN A.11.19.00 Cluster File System Admin SD Product
# Cluster-CFS.CFS-MAN A.11.19.00 CFS MAN pages
# Package-CVM-CFS A.11.19.00 HP SG Cluster CVM CFS SD Product
# Package-CVM-CFS.CM-CVM-CFS A.11.19.00 CVM CFS Package Fileset
# Package-CVM-CFS.CM-CVM-CFS-COM A.11.19.00 CVM CFS Package Fileset
# swlist -l file | grep -i vxfsckd
Package-CVM-CFS.CM-CVM-CFS-COM: /etc/cmcluster/cfs/SG-CFS-vxfsckd.sh
VRTSvxfs.VXFS-RUN: /opt/VRTS/bin/vxfsckd
VRTSvxfs.VXFS-RUN: /usr/lbin/fs/vxfs5.0/vxfsckd
VRTSvxfs.VXFS-RUN: /usr/sbin/vxfsckd
#

[Shibin_2]

Can you verify, these listed patches are already applied or not?

http://h71028.www7.hp.com/enterprise/us/en/os/hpux11i-serviceguard-storage-management-suite-patches.html?jumpid=reg_R1002_USEN#v213

[nasmel]

HI shibin,

I just reinstalled Service guard for CFS on Both servers and then it worked.

Thanks for your help

[nasmel]

Thanks for your help