Serviceguard

[Kenrick Sy]

Hi,

I have two HP-UX servers with Serviceguard installed and configured. SERVER1 is the primary while SERVER2 act as the failover server. It was running smoothly for the past 3 years. Lately it was experiencing unexpected behavior. When SERVER1 is shutdown, it will do a failover on SERVER2. However, the uptime on SERVER2 after failover is only 30 minutes. Also trying single mode on SERVER2, uptime is also only 30 minutes.

What do I need to check?

Thanks.

Kenrick

[A. Clay Stephenson · Accepted solution]

Do the failures occur after exactly 30 minutes or approximately 30 minutes? If it is exactly 30 minutes then that strongly suggests something in software. Are these failures TOC's? Have you examined any of the SG logs or the system logs. You really don't provide enough data to begin to analyze your problem.

[Kapil Jha]

what do you mean by
>>>>the uptime on SERVER2 after failover is only 30 minutes
does it mean after failover server2 lasts for 30 minutes only.
have you checked logs what is happeing in these 30 minutes.If same thing is happening everytime or smthin else.....
As said above please provide more info.
Kapil

[whiteknight]

Ken,

here is a troubleshooting approach you can refer to :-

Serviceguard Troubleshooting Approaches
http://docs.hp.com/en/B3936-90079/ch08s07.html


WK

[Kenrick Sy]

Hi,

What are the specific log files needed to further troubleshoot this problem?

Approximately 30 minutes uptime on SERVER2 after failover or in single mode.

Thanks.

Kenrick

[Kapil Jha]

Hi Kenrick,
Can you please send the syslog file when it is about to go down.
Have you seen some other problems in this half an hour....u must have monitored system for at least these 30 minutes....
>>>Also trying single mode on SERVER2
have you tried server2 alone check if its running for more than 30 minutes.
BR,
Kapil

[Kenrick Sy]

Hi,

Attached is the syslog file.

Thanks.

Kenrick

[Kenrick Sy]

Hi,

I checked on the HP-UX server and found out that the root filesystem is 97% used. I think that my problem on MC Serviceguard is related to the 97% usage on the root filesystem.

Hope you can help me on my problem.

Thanks in advance.

Kenrick

[Mridul Shrivastava]

Try finding out the largest files under root and if they are not important take a backup of those files and remove them from root to create space.
Execute this command to find out the large files:

find . -type f -xdev -size +10000000c -exec ll {} \;

If above o/p shows lots of files then add one more Zero in the command and execute again.

[Kenrick Sy]

Hi Kapil,

First I run bdf on SERVER1
# bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 143360 135265 7779 95% /
/dev/vg00/lvol1 83733 52868 22491 70% /stand
/dev/vg00/lvol8 1105920 296679 758996 28% /var
/dev/vg00/lvol7 1179648 521940 616610 46% /usr
/dev/vg00/u01 4096000 1567092 2371055 40% /u01
/dev/vg00/lvol4 65536 12023 50203 19% /tmp
/dev/vg00/lvol6 536576 425757 103939 80% /opt
/dev/vg00/lvol5 20480 5648 13935 29% /home
/dev/vgdb/u02 2048000 1554689 462484 77% /u02
/dev/vgdb/u03 1536000 375122 1088329 26% /u03
/dev/vgdb/u04 10240000 1643575 8060965 17% /u04
/dev/vgdb1/u05 10240000 2234660 7755178 22% /u05
/dev/vgdb2/u6 15360000 14032994 1285542 92% /u06
/dev/vgdb2/u7 15360000 5730492 9329002 38% /u07

Then I do a "cmhaltnode SERVER1" to test the cluster shifting to SERVER2, running bdf on SERVER2 results in the absence of u02, u03, u04, u05, u06 and u07.

What are the possible adjustments in the settings? This has been running for the past 3 years.

Thanks.

Kenrick

[yassyboy]

Hi Kenrick,

Looking at your BDF it seems you have couple of file systems with over 75% usage.

I am sure you have thought of this but if the files that are residing are important then
1. Backup the System and those files

2. Extend the volumes (If there is free space)or add bigger disks.

3. Because your system is running for over 3 years, I am assuming the system is still using the default HDDrives that came with it.

Upgrading all hardrives will save you alot of admin process, more over the system does require upgrade so this is your golden chance to advise the upper management to replace the HDD with larger ones.

4. Also check if some applications are not producing unnecessary logs.
Some times this tends to grow very rapidly with huge sizes and consume system resources.

Hope that helps....

Kind Regards...

[Rita C Workman]

To try and get root "/" down.

Try looking for old logfiles and trim them. Example might be your package logfiles that are generally /etc/cmcluster/packages//

Any other software that keeps logs in /etc

Multiple old un-needed (backup) copies of /etc/hosts; /etc/group; /etc/lvmtab, ...etc.

Look and see if there are any bogus files created under /dev. I usually find /dev/rmt/ to be the culprit when we have had new folks learning HPUX.

Look for any files that obviously just plain don't belong out there and either clean them up or move them over.

Just a couple thoughts, but actually I don't think that is why your failing in around 30 minutes.

Rgrds,
Rita

[Rita C Workman]

Let me retract......./dev/root shows as full in syslog that I just noticed was attached.

So, since the logfiles for your package is sitting under the /etc directory (i.e. under "/") if it can't write to the logfile, that could stop your package.

/rcw

[yassyboy]

Hi All,

I agree with you all.

We have a spare HP-UX server and want to use this as a testbed for applications, simulate problems, troubleshooting and training.

We need to configure as a single node cluster with MC Service Guard and the toolkit (Same version as mentioned above.) I was wondering if thats possible by downloading from HP site. As this is not supported anymore is it freely available for download for training.

KR....