MSA2012fc "flapping" its paths during heavy usage - how to trace problem?

[Paul_637]

Hi all.

I have an MSA2012fc with a direct-attached server 2008 cluster pair (via two FC cards on each node).

One of the volumes is giving trouble under fairly heavy load (B2D usage, whilst spooling from disk to tape it seems) whereby we get constant MPIO errors (Event ID's 16 & 17) once or twice per second or so and the console hangs whilst logging in.

The two event log errors flapping are various combinations of:

(16) A fail-over on \Device\MPIODisk1 occurred.
(17) \Device\MPIODisk1 is currently in a degraded state. One or more paths have failed, though the process is now complete.

This time around I decided (rather than power-cycling the server) to kill one of the paths to the MSA (via volume management) and the events stopped immediately with the server becoming responsive once again.

Can anyone point me to any suitable trouble-shooting info so I can work out what's going on? I'm currently downloading SANSurfer in the hope it might reveal something but could do with a pointer in the meantime...

Paul

[hvhari]

Its good to check the events on MSA to check for any kind of HW errors .

[Paul_637]

Indeed, was the first place I looked but couldn't see anything related.

One thing that has occurred to me is the "oddities" of the MSA2000 series and their Active/Active status being somewhat "different" to other similar arrays.

Given a config with a dual controller & two direct-attach hosts, if correctly cabled, should you be able to see your drives via *both* HBAs, or will the drive only be accessible via the alternate path if the MSA controller actually fails (and thus fails over)?

[Bhavnish Kamboj (BK)]

Paul,

As you said if all connected correctly then definately server will see the disks from all active and aletrnate path too but it will only use the path to access it as per your MPIO policy.

If you go to the MPIO Manager in My Computer -> Manage then there you can see the all available paths for your disks.

[Johan Guldmyr]

You can go into each vdisk/volume in the SMU to see which controller owns that vdisk.

Your MPIO - it's not setup to roundrobin or something like that is it?

What did sansurfer say?

Do you need both HBAs?

Can you maybe discern from the MSA event log which of the two paths (channels) that are flapping? Maybe one of the HBAs is causing this - checked cabling - etc?

Don't forget to check firmwares of especially the MSA.