best way to test clustering

[Ahmed_58]

Hi,
I'm new on clsutering and I need help on best way to test running cluster afetr adding new disks, VG's etc., shall I shutdown the server to see how failover works? or?

Regards,
Ahmed

[A. Clay Stephenson]

No, shutting down is far too nice. First, you yank ONE thing. ie. 1 LAN cable, 1 disk drive, 1 SCSI cable, 1 ethernet switch, etc. MC/SG is designed to handle single point of failure -- thus limit yourself to one thing at a time. Most failures are handled by LVM or LAN failovers before MC/SG ever comes into play. To test the loss of a node, the correct procedure is to yank the power cord(s) on that host.

If you feel that yanking a power cord is too harsh then you haven't made your cluster robust enough.

[Patrice Le Guyader]

Hi Ahmed,

What we do on production's server is to plan a switch of the packages (MC/SG), Service Group (Veritas Cluster) concerned and look at the definition of the VG and of the LVols. We are even looking at the good repartition of the Physical extends when we have a mirror accross several arrays for controling that vgimport is good(No cross array definition).

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1009128

Regards
Pat

[Kent Ostby]

Shutting one box down with a reboot will give you one kind of test and it's not a bad way to go since systems do panic.

It will at least test your cluster lock disk or quorem server.

The pulls of the various cables will work well and wiping out one subnet in the area to the two machines would be another good one.

[Ahmed_58]

Thanks for usfull info,

Can I use cm commands to test failover?
And how can I get back to primary server after the test?
Bascilly is there anywhere I can follow the steps of checking?

Regards,
Ahmed

[A. Clay Stephenson]

You can use the cmhaltnode, for example, to halt a node but again, that is much too gentle, to be a real test. Ask yourself, when you have a node failure, is it going to issue a cmhaltnode and then die gracefully or is it just going to quit? When I said yank power cords, I meant exactly that because that is how your server is going to die.

There are really no "cookie-cutter" test plans because all clusters are different. Almost no one will have your mix of servers, network cards, disks, disk arrays, SAN's, i/o cards, LAN cards, backup power, and applications. You really have to develop and test your own test plans. The idea is that you do your best to break the cluster but you have to play fair --- break ONE thing then fix it and break ONE other thing ... and so on.

[melvyn burnard]

Take a read of Chapter 8 of the Managing Serviceguard manual, but essentially nothiong beats "deestructive" tests, as in forcing real life scenarios like removing lan cables, kill the cmcld process, pulling power on a node etc.

[Ahmed_58]

Thanks for your very usfull advice,
appreiciated.

Regards.

[Stephen Doud]

You mentioned testing after adding "new disks, VG's".
I have to assume you want to test Serviceguards' ability to deal with disk and LVM changes.

Unless you have a specific EMS Resource configured to monitor /vg/vg/pv_summary, Serviceguard will be blind to path failures or mech failures to disk devices.

As for a testing suite, check this document:
Title: Serviceguard Test Suite Whitepaper
Document ID: UMCSGKBAN00001166

[Ahmed_58]

Thank you all. and wish good luck.
Ahmed