topic Re: Cleuster startup Slow in Operating System - HP-UX

Cleuster startup Slow

Sยภเl Kย๓คг — Sat, 11 Sep 2004 06:01:15 GMT

I have an L2000 and N4000 cluster through 2 mirrored SC10 disarray with MC/Service guard.This cluster take long time to start the cluster,morethan 10 Mins.Can anyone help me to find out what is worng.
regarsd
SK

Re: Cleuster startup Slow

Victor Fridyev — Sat, 11 Sep 2004 06:22:53 GMT

Could you please check whether name resolution works normally. For this look at /etc/nsswitch.conf and remove all unused parts (DNS and NIS).
Additionally please attach /var/adm/syslog/syslog.log with the part of the cluster starting up.

Re: Cleuster startup Slow

steven Burgess_2 — Sat, 11 Sep 2004 06:48:18 GMT

hello

check your cluster package logs in

/etc/cmcluster//

what stage is taking the time ? if you do a cmviewcl is that status reforming ? or dou have any any packages up and running ?

Steve

Re: Cleuster startup Slow

Sยภเl Kย๓คг — Sat, 11 Sep 2004 06:49:53 GMT

Victor
i am not able to find such file in my server.
regards
SK

Re: Cleuster startup Slow

Sยภเl Kย๓คг — Sat, 11 Sep 2004 07:06:20 GMT

NODE STATUS STATE
rcftrl1 up running
rcftrn1 up running

PACKAGE STATUS STATE PKG_SWITCH NODE rcfpkg startng startng enabled rcftrn1

it looks like this.But it takes sommuch time to come to the status

CLUSTER STATUS
rcf up

NODE STATUS STATE
rcftrl1 up running
rcftrn1 up running

PACKAGE STATUS STATE PKG_SWITCH NODE
rcfpkg up running enabled rcftrn1

regards
SK

Re: Cleuster startup Slow

Sridhar Bhaskarla — Sat, 11 Sep 2004 08:07:17 GMT

Hi Sunil,

I would check the following

1. How may VGs do you have that are clusteraware?. How long will it take for a normal 'ioscan' to complete. Is the ioscan output clean?.

2. What's the DNS response on *both* the servers?. Add DNS names to /etc/hosts file. Make sure all the interfaces configured with an IP in netstat -in have an entry in /etc/hosts file including heartbeats. I usually name the heartbeats as hostname-hb1, hostname-hb2.

3. Make sure the latency of the heartbeat connections is minimum.

Look for errors in /var/adm/syslog/syslog.log and see if there are any additional messages when the cluster is forming.

-Sri

Re: Cleuster startup Slow

Sridhar Bhaskarla — Sat, 11 Sep 2004 08:11:23 GMT

..and I should have added. Use cmclnodelist than .rhosts if you aren't using it already. It's faster.

-Sri

Re: Cleuster startup Slow

Duncan Edmonstone — Sat, 11 Sep 2004 12:43:32 GMT

Are you saying it is like this:

NODE STATUS STATE
rcftrl1 up running
rcftrn1 up running

PACKAGE STATUS STATE PKG_SWITCH NODE
rcfpkg startng startng enabled rcftrn1

for about 10 minutes, and then eventually goes to this:

CLUSTER STATUS
rcf up

NODE STATUS STATE
rcftrl1 up running
rcftrn1 up running

PACKAGE STATUS STATE PKG_SWITCH NODE
rcfpkg up running enabled rcftrn1

Because if thats the case, then whats taking a long time to come up is not the cluster, but the package 'rcfpkg'

As others have stated - we need to see the package startup log in /etc/cmcluster//.cntl.log (or something like that) to give any further assistance.

HTH

Duncan

Re: Cleuster startup Slow

Sridhar Bhaskarla — Sat, 11 Sep 2004 14:01:41 GMT

Hi Sunil,

I didn't read your second response. As Duncan pointed out, then your cluster is starting just fine. It's the package that is coming up slow. It could be due to your application itself. You can see all the package startup activity in

/etc/cmcluster/rcfpkg/rcfpkg.*.log file. It is a general standard to put package related files under /etc/cmcluster/ directory. The package control file is usually named .cntl or .ctl. Depending on the name of the control file you will find a log file with a .log extension in that directory. While the package is starting do a 'tail -f .cntl.log' and see where it is hanging.

-Sri

Re: Cleuster startup Slow

Victor Fridyev — Sat, 11 Sep 2004 16:04:03 GMT

Hi,
If you don't have /etc/nsswitch.conf on your computers, this may cause a wrong name resolution way.
Create file with this name on all machines of the cluster and insert into the file something like this:

hosts: files

If you use dns, the file may look like this:

hosts: files [NOTFOUND=continue] dns

HTH

Re: Cleuster startup Slow

melvyn burnard — Sat, 11 Sep 2004 16:54:35 GMT

As others have pointed out, if this is the result from cmviewcl:
NODE STATUS STATE
rcftrl1 up running
rcftrn1 up running

PACKAGE STATUS STATE PKG_SWITCH NODE
rcfpkg startng startng enabled rcftrn1

Then your CLUSTER is up, but the PACKAGE rcfpkg is starting. This means your issue is NOT the cluster, but the package. You need to see what your application is doing during startup, and also check the package log file for the package on node rcftrn1.

Re: Cleuster startup Slow

Brian Markus — Sun, 12 Sep 2004 12:20:50 GMT

I had a similar problem when initially setting up. Go into the package directory and check your control script (.cntl normally but can be named anything). See what volume group itâ s activating, then do everything manually without MC/Serviceguard. Mount the file system on one node, and then start the application on that VG. See how long it takes manually. Then try this on the second node. If you cluster was setup properly, you should have a directory with your package name under /etc/cmcluster In my case I have SERVICEDESK. In that directory I have three files. My conf, control script, and log.

root@node1:/etc/cmcluster/SERVICEDESK> ls
servicedesk.cntl servicedesk.cntl.log servicedesk.conf

This is in my servicedesk.cntl script:

VGCHANGE="vgchange -a e"
VG[0]=/dev/vg10
LV[0]=/dev/vg10/lvol1; FS[0]=/opt/OV/sd; FS_MOUNT_OPT[0]="-F vxfs -o rw,largefiles"
FS_UMOUNT_COUNT=5
# START OF CUSTOMER DEFINED FUNCTIONS
/opt/OV/sd/server/bin/startserver

So I would then do the following to test manually:
vgchange -a e /dev/vg10/lvol1
mount /dev/vg10/lvol1 /opt/OV/sd
/opt/OV/sd/server/bin/startserver

If this is slow, itâ s most likely an application issue. Check name resolution on both nodes, MC/Serviceguard is very picky about that.

Once youâ ve got it working manually attempt to run the package on one node then the other while reading the package log.
tail -f /etc/cmcluster/SERVICEGUARD/servicedesk.cntl.log

Run the package on one node then the other
cmrunpkg -n node1 SERVICEDESK
cmhaltpkg -n node1 SERVICEDESK

cmrunpkg -n node2 SERVICEDESK
cmhaltpkg -n node2 SERVICEDESK

I hope this helps. Good luck.

Re: Cleuster startup Slow

Sยภเl Kย๓คг — Sun, 12 Sep 2004 23:13:14 GMT

Thank you very mush for the response.
Problem was with one disk in the cluster.One disk was faulty in one the SC10 array.
regards
SK