Service Guard testing

[Quin Hammes]

We have 2 L 2000s mirrored over fiber to 2 FC 60 disk arrays. This is working just fine right now. We are now going to install service guard. The client wants detailed testing to prove that everything works.
One test they want to do is to disconnect the network cable and see how the NIC card fails over to the standby. Once we do this, what is the easiest way to get it back on the primary nic card (or does this happen automatically?).
They would also like to test fiber disconnects. If the we disconnect the primary fiber cable and it starts using the alternate HW path, how do we get it back (plug the cable back in and do an ioscan?).
Another test they want to do is to power off the primary disk array. I assume that we should just resync the mirror after it is complete.
The last major step is to test a failure of the primary L2000. What is the best way to do this without hurting the box? We could just unplug all three power cables to the L2000 but I assume that is not a good thing to do. Should we just kill the cluster processes or do a shutdown of the box?
Any input on the best way to test and prove to a client that Service Guard is running correctly would be appreciated (hint hint points galore). Thanks in advance.

[James R. Ferguson · Accepted solution]

Hi:

Take a look at Knowledge Base document #UXSGLVKBAN00000470 ("How do I test MC/ServiceGuard?"). I think you will find this quite useful.

...JRF...

[Christopher Caldwell]

We've jerked network cables (yup, Service Guard works ;-) ).

It's been a while, but I believe you define failover/failback policies in the package. For instance, we never automatically fail a package back to a node the becomes available; I'd rather do that by hand.

Not sure on you fibre cable; I've never actually done it. When I challenged the HP folks about having to shut down FC60s to rev fibre switch firmware, they conceded that it would be possible to disable a port in the switch; that may be preferrable to a disconnect.

I'm not sure what happens when the path becomes available again. Keep in mind, that to maintain availability, on loss of path, you might fail host A to host B, then reboot/rescan to get the dual paths back might not be a bad proposition. Your mileage may vary.

For the box itself, don't forget to test the mirror of the internal drives (yeah, go ahead and jerk the drive). The mirror stuff really works! You need to verify that you can boot from the alternate root, and that the box keeps running when you pull an internal root drive.

[Be VERY careful with servers that deal with things like databases that can become corrupt]. The power kill test is a valid test. Probably wouldn't want to do it every day, but ya gotta test.

If you're using EMS events to cause package failovers, you might look at trying to simulate other problems.

[Quin Hammes]

how do you get to the knowledge base? which knowledge base?

[James R. Ferguson]

Hi:

To get to the "Knowledge Base", make sure you are logged-in to the ITRC for full access privileges; click Maintenance and Support -> Search Technical Knowledge Base; Choose either "Search by Keyword" or "Search by Doc ID" from the dropdown; and enter a keyword, or in the case of my previous post, enter the document ID I cited.

Regards!

...JRF...

[Quin Hammes]

Nope, I tried that and the document is not found when I search. Is this doc only for HP internal?

[James R. Ferguson]

Hi again:

OK, see attached.

...JRF...