Cleuster startup Slow

[Sยภเl Kย๓คг]

I have an L2000 and N4000 cluster through 2 mirrored SC10 disarray with MC/Service guard.This cluster take long time to start the cluster,morethan 10 Mins.Can anyone help me to find out what is worng.
regarsd
SK

[Victor Fridyev]

Could you please check whether name resolution works normally. For this look at /etc/nsswitch.conf and remove all unused parts (DNS and NIS).
Additionally please attach /var/adm/syslog/syslog.log with the part of the cluster starting up.

[steven Burgess_2]

hello

check your cluster package logs in

/etc/cmcluster//

what stage is taking the time ? if you do a cmviewcl is that status reforming ? or dou have any any packages up and running ?

Steve

[Sยภเl Kย๓คг]

Victor
i am not able to find such file in my server.
regards
SK

[Sยภเl Kย๓คг]

NODE STATUS STATE
rcftrl1 up running
rcftrn1 up running

PACKAGE STATUS STATE PKG_SWITCH NODE rcfpkg startng startng enabled rcftrn1

it looks like this.But it takes sommuch time to come to the status


CLUSTER STATUS
rcf up

NODE STATUS STATE
rcftrl1 up running
rcftrn1 up running

PACKAGE STATUS STATE PKG_SWITCH NODE
rcfpkg up running enabled rcftrn1

regards
SK

[Sridhar Bhaskarla]

Hi Sunil,

I would check the following

1. How may VGs do you have that are clusteraware?. How long will it take for a normal 'ioscan' to complete. Is the ioscan output clean?.

2. What's the DNS response on *both* the servers?. Add DNS names to /etc/hosts file. Make sure all the interfaces configured with an IP in netstat -in have an entry in /etc/hosts file including heartbeats. I usually name the heartbeats as hostname-hb1, hostname-hb2.

3. Make sure the latency of the heartbeat connections is minimum.

Look for errors in /var/adm/syslog/syslog.log and see if there are any additional messages when the cluster is forming.

-Sri

[Sridhar Bhaskarla]

..and I should have added. Use cmclnodelist than .rhosts if you aren't using it already. It's faster.

-Sri

[Duncan Edmonstone · Accepted solution]

Are you saying it is like this:



NODE STATUS STATE
rcftrl1 up running
rcftrn1 up running

PACKAGE STATUS STATE PKG_SWITCH NODE
rcfpkg startng startng enabled rcftrn1



for about 10 minutes, and then eventually goes to this:

CLUSTER STATUS
rcf up

NODE STATUS STATE
rcftrl1 up running
rcftrn1 up running

PACKAGE STATUS STATE PKG_SWITCH NODE
rcfpkg up running enabled rcftrn1




Because if thats the case, then whats taking a long time to come up is not the cluster, but the package 'rcfpkg'

As others have stated - we need to see the package startup log in /etc/cmcluster//.cntl.log (or something like that) to give any further assistance.

HTH

Duncan

[Sridhar Bhaskarla]

Hi Sunil,

I didn't read your second response. As Duncan pointed out, then your cluster is starting just fine. It's the package that is coming up slow. It could be due to your application itself. You can see all the package startup activity in

/etc/cmcluster/rcfpkg/rcfpkg.*.log file. It is a general standard to put package related files under /etc/cmcluster/ directory. The package control file is usually named .cntl or .ctl. Depending on the name of the control file you will find a log file with a .log extension in that directory. While the package is starting do a 'tail -f .cntl.log' and see where it is hanging.

-Sri

[Victor Fridyev]

Hi,
If you don't have /etc/nsswitch.conf on your computers, this may cause a wrong name resolution way.
Create file with this name on all machines of the cluster and insert into the file something like this:

hosts: files

If you use dns, the file may look like this:

hosts: files [NOTFOUND=continue] dns

HTH

[melvyn burnard]

As others have pointed out, if this is the result from cmviewcl:
NODE STATUS STATE
rcftrl1 up running
rcftrn1 up running

PACKAGE STATUS STATE PKG_SWITCH NODE
rcfpkg startng startng enabled rcftrn1

Then your CLUSTER is up, but the PACKAGE rcfpkg is starting. This means your issue is NOT the cluster, but the package. You need to see what your application is doing during startup, and also check the package log file for the package on node rcftrn1.

[Brian Markus]

I had a similar problem when initially setting up. Go into the package directory and check your control script (.cntl normally but can be named anything). See what volume group itâ s activating, then do everything manually without MC/Serviceguard. Mount the file system on one node, and then start the application on that VG. See how long it takes manually. Then try this on the second node. If you cluster was setup properly, you should have a directory with your package name under /etc/cmcluster In my case I have SERVICEDESK. In that directory I have three files. My conf, control script, and log.

root@node1:/etc/cmcluster/SERVICEDESK> ls
servicedesk.cntl servicedesk.cntl.log servicedesk.conf


This is in my servicedesk.cntl script:

VGCHANGE="vgchange -a e"
VG[0]=/dev/vg10
LV[0]=/dev/vg10/lvol1; FS[0]=/opt/OV/sd; FS_MOUNT_OPT[0]="-F vxfs -o rw,largefiles"
FS_UMOUNT_COUNT=5
# START OF CUSTOMER DEFINED FUNCTIONS
/opt/OV/sd/server/bin/startserver


So I would then do the following to test manually:
vgchange -a e /dev/vg10/lvol1
mount /dev/vg10/lvol1 /opt/OV/sd
/opt/OV/sd/server/bin/startserver

If this is slow, itâ s most likely an application issue. Check name resolution on both nodes, MC/Serviceguard is very picky about that.

Once youâ ve got it working manually attempt to run the package on one node then the other while reading the package log.
tail -f /etc/cmcluster/SERVICEGUARD/servicedesk.cntl.log

Run the package on one node then the other
cmrunpkg -n node1 SERVICEDESK
cmhaltpkg -n node1 SERVICEDESK

cmrunpkg -n node2 SERVICEDESK
cmhaltpkg -n node2 SERVICEDESK


I hope this helps. Good luck.

[Sยภเl Kย๓คг]

Thank you very mush for the response.
Problem was with one disk in the cluster.One disk was faulty in one the SC10 array.
regards
SK