MSA P2000 G3 SAS Network Connection Issues

[phlegmer]

Hardware

24 Disk MSA P2000 G3 SAS

x2 SAS Controller AW592A

Bundle Firmware

TS251P006-02 on both controllers

A couple weeks ago I had disconnected the Cat5 cable from one of the controllers doing some rearanging in the back of the rack.  When I plugged it back in, the connection has been flaky ever since.  What I see happening on the controller is the link light will flash slowly for a while, then suddenly it will stay solid and the activity light will randomly flicker.  After a few moments of that, it goes back just the link light flashing slowly....rinse and repeat.  When I try to connect to that controller, it works for a little bit then connection is dropped.  Up, down, up, down....

I have tried all of the following and still can't get this to stay connected (just for reference, the bottom slot is the one with the issue):

1.   Swapped top and bottom controller Cat5 cables (swapping ports on the switch)
2.   Tried at lest 2 different cables
3.   Shutdown the controller, pulled it from the chasses (wait 30 sec), then reinserted the controller
4.   Assigned a new IP address to the bottom controller
5.   Set aside down time and power cycled the whole MSA
6.   Even purchased a factory refurbished controller

Tried all this and the problem still stays with that location (the bottom slot)

Please advise!

Thanks

[AnkitM]

If this is covered under warranty then I would advise you to open a case with Support as it requires store.log analysis.

If not then can you attach output of #show configuration and show events (grep it to 300 events) ?

[phlegmer]

Show Config and Event attached

Thanks!

[AnkitM]

The issue which you are facing appears to be an issue with either faulty cabling or a faulty ethernet switch.

Can you swap the Ethernet cables between the two controllers. This will show us whether the problem is outside or inside the storage system.

If fault follows the cable then
1. Replace Cable

2. Still issue persist, Change the switch Port.

3. Still issue persist, we might need to replace controller module.

[phlegmer]

As stated in my initial posting, we have already tried your three steps.  The issue seems to stay with the slot.  Perhaps something wrong with the chasis?

Thanks

[AnkitM]

I doubt it. I would re-try changing cable and the switch port again before I go for Chassis replacement.

Even the official MSA Event Description Guide refers to the same. Its more of an external issue.

If there is an issue with the Management Port then from MSA end Controller Replacement should take care of it.

[phlegmer]

Problem RESOLVED!!

Just for fun, I plugged that controller into a totally different managed switch that I had on hand.  The link stayed connected!  Plugged it back into the server switch, same up and down up and down....  Swapped ports (for like the 10th time I think) still the same and stayed with that controller.

Got in touch with our network group (we don't manage our own networking hardware) and had them look into the issue.  From what they saw, that controller kept on wanting to only use 10Mbps / half?!  I go into the web interface and don't see anything about network speed settings.  SSH into the box, do a 'show network' and pow! there it was.  That controller was set to 10Mbps/half and auto negotiation was disabled.  While Controller A was enabled and set correctly.  However, I could not force it to change.  Came across another thread that talked about this same thing.  It wasn't until I actually sshed into controller B and made the change that the correct settings actually stuck.

Now all appears well.

Post-analysis:

Unplugged the controller NIC

Plugged in a tape library into the switch port that was used by the MSA

Did the firmware upgrade I needed to do to the tape library

Unplugged tape library and plugged the MSA back

Controller somehow switched itself to disable auto negotiate 10/half

I just tried to reproduce the above (sans firmware upgrade) but wasn't able.  The tape library also uses auto negotiate 100/full so who knows how/why it happened.