msa2000 - controllers do not work

[Andrea Cicchelli]

Hello,

My msa works but I can't access it (MSA2312sa) with network from both controllers. I can acces using controller B via Com - telnet - cli. So this is the configuration :

# show controllers
Controllers
-----------
Controller ID: A
Serial Number: N/A
Hardware Version:
CPLD Version:
MAC Address: 00:00:00:00:00:00
WWNN: 500C0FFDA8C92000
IP Address: 0.0.0.0
IP Subnet Mask: 0.0.0.0
IP Gateway: 0.0.0.0
Disks: 12
Vdisks: 0
Cache Memory Size (MB): 1024
Host Ports: 4
Disk Channels: 2
Disk Bus Type: SAS
Status: Down
Failed Over: No
Fail Over Reason: Not applicable

Controller ID: B
Serial Number: 3CL947S584
Hardware Version: 54
CPLD Version: 8
MAC Address: 00:C0:FF:DA:52:F9
WWNN: 500C0FFDA8C92000
IP Address: 192.100.1.100
IP Subnet Mask: 255.255.255.0
IP Gateway: 192.100.1.253
Disks: 12
Vdisks: 2
Cache Memory Size (MB): 1024
Host Ports: 4
Disk Channels: 2
Disk Bus Type: SAS
Status: Running
Failed Over: Yes
Fail Over Reason: PCIE link recovery failed

As you can see, the Controller A has no data. I can not configure its network via cli. I mean, I can send the command and the command is successfully as you can see

set network-parameters ip 192.100.1.101 netmask 255.255.255.0 gateway 192.100.
1.253 controller a
Success: Command completed successfully. - Network parameters were changed.

... but in the controller a, no change.

I can config network in Controller B but the network is down : i can't not ping that ip and from cli using command  "ping" I can't ping the external devices.

I can not restart mc a :

# restart mc a
During the restart process you will briefly lose communication with the specifie
d Management Controller(s).
Continue? yes
Error: The request cannot be completed because the other controller is not up.

Instead, I can restart sc a, but there is no change.

At this moment only controller b is linked to host (hp dl380).

So what should I do ?

is the a controller broken?

is the b controller broken?

do I need to upgrade the firmware (Storage Controller Code Version: M110R21) ? How can I upgrade it if i can't access via network ?

Thanks

Andrea

[AnkitM]

If you have physical access to the MSA then could you check on the backpanel of the MSA Controller A, if it has any amber LED lit?

If yes, then it is possible that Controller A has failed.

You may try below steps:

1. Reseat Controller A and verify if it shows an IP Address

2. If not, Can you attach output of #show config and #show events ?

[Andrea Cicchelli]

Hi,

No led on in back panel of controller a. I can see only the two led of the ethernet port.

How can I reset controller A  ? I did not find the reset command via cli.

Andrea

[AnkitM]

Reseat the Controller NOT Reset.

1. Disconnect cables connected to the module. Label each cable to facilitate reconnection.

2. Turn the thumbscrews until the screws disengage from the module (1) and rotate both latches downward to disengage the module from the internal connector.

3. Pull the module straight out of the enclosure.

Wait for 3-5mins.

1. With the latches in the open position, slide the module into the enclosure as far as it will go. If necessary, press lightly on the top-center of the module to facilitate insertion. 

2. Rotate the latches upward to engage the module with the internal connector and turn the thumbscrews finger-tight.

3. Reconnect the cables.

[Andrea Cicchelli]

Thanks AnkitM and I’m sorry for the error.

Here the situation before reseat of controller A :

******************************************************

Controller ID: A

Serial Number: N/A

Hardware Version:

CPLD Version:

MAC Address: 00:00:00:00:00:00

WWNN: 500C0FFDA8C92000

IP Address: 0.0.0.0

IP Subnet Mask: 0.0.0.0

IP Gateway: 0.0.0.0

Disks: 12

Vdisks: 0

Cache Memory Size (MB): 1024

Host Ports: 4

Disk Channels: 2

Disk Bus Type: SAS

Status: Down

Failed Over: No

Fail Over Reason: Not applicable

Controller ID: B

Serial Number: 3CL947S584

Hardware Version: 54

CPLD Version: 8

MAC Address: 00:C0:FF:DA:52:F9

WWNN: 500C0FFDA8C92000

IP Address: 192.100.1.100

IP Subnet Mask: 255.255.255.0

IP Gateway: 192.100.1.253

Disks: 12

Vdisks: 2

Cache Memory Size (MB): 1024

Host Ports: 4

Disk Channels: 2

Disk Bus Type: SAS

Status: Running

Failed Over: Yes

Fail Over Reason: PCIE link recovery failed

Controller A Versions

---------------------

Storage Controller CPU Type: Not Present

Storage Controller Code Version: Not Present

Memory Controller FPGA Code Version: Not Present

Storage Controller Loader Code Version: Not Present

Management Controller Code Version: Not Present

Management Controller Loader Code Version: Not Present

Expander Controller Code Version: Not Present

CPLD Code Version: Not Present

Hardware Version: Not Present

Host Interface Module Version: Not Present

Host Interface Module Model: Not Present

Controller B Versions

---------------------

Storage Controller CPU Type: Athlon 2600+ 1600MHz

Storage Controller Code Version: M110R21

Memory Controller FPGA Code Version: F300R22

Storage Controller Loader Code Version: 19.009

Management Controller Code Version: W441R06

Management Controller Loader Code Version: 12.015

Expander Controller Code Version: 1103

CPLD Code Version: 8

Hardware Version: 54

Host Interface Module Version: 50

Host Interface Module Model: 0

*********************************************

I removed and put back in the controller A. Here the new config :

Controllers

-----------

Controller ID: A

Serial Number: 3CL948R438

Hardware Version: 54

CPLD Version: 8

MAC Address: 00:C0:FF:DA:4D:3E

WWNN: 500C0FFDA8C92000

IP Address: 192.100.1.35

IP Subnet Mask: 255.255.255.0

IP Gateway: 192.100.1.103

Disks: 12

Vdisks: 1

Cache Memory Size (MB): 1024

Host Ports: 4

Disk Channels: 2

Disk Bus Type: SAS

Status: Running

Failed Over: No

Fail Over Reason: Not applicable

Controller ID: B

Serial Number: 3CL947S584

Hardware Version: 54

CPLD Version: 8

MAC Address: 00:C0:FF:DA:52:F9

WWNN: 500C0FFDA8C92000

IP Address: 192.100.1.100

IP Subnet Mask: 255.255.255.0

IP Gateway: 192.100.1.253

Disks: 12

Vdisks: 1

Cache Memory Size (MB): 1024

Host Ports: 4

Disk Channels: 2

Disk Bus Type: SAS

Status: Running

Failed Over: No

Fail Over Reason: Not applicable

Controller A Versions

---------------------

Storage Controller CPU Type: Athlon 2600+ 1600MHz

Storage Controller Code Version: M110R21

Memory Controller FPGA Code Version: F300R22

Storage Controller Loader Code Version: 19.009

Management Controller Code Version: W441R06

Management Controller Loader Code Version: 12.015

Expander Controller Code Version: 1103

CPLD Code Version: 8

Hardware Version: 54

Host Interface Module Version: 50

Host Interface Module Model: 0

Controller B Versions

---------------------

Storage Controller CPU Type: Athlon 2600+ 1600MHz

Storage Controller Code Version: M110R21

Memory Controller FPGA Code Version: F300R22

Storage Controller Loader Code Version: 19.009

Management Controller Code Version: W441R06

Management Controller Loader Code Version: 12.015

Expander Controller Code Version: 1103

CPLD Code Version: 8

Hardware Version: 54

Host Interface Module Version: 50

Host Interface Module Model: 0

So after your suggestion, The system can see the controller A and there are no error (Failed Over) on link of Controller B

But I can’t ping both controller.

I tried changing the settings of the controller a’s network card through cli and both controllers have returned to the previous situation: A controller does not have the data and B controller has an error.

Controller ID: A

Serial Number: N/A

Hardware Version:

CPLD Version:

MAC Address: 00:00:00:00:00:00

WWNN: 500C0FFDA8C92000

IP Address: 0.0.0.0

IP Subnet Mask: 0.0.0.0

IP Gateway: 0.0.0.0

Disks: 12

Vdisks: 0

Cache Memory Size (MB): 1024

Host Ports: 4

Disk Channels: 2

Disk Bus Type: SAS

Status: Down

Failed Over: No

Fail Over Reason: Not applicable

Controller ID: B

Serial Number: 3CL947S584

Hardware Version: 54

CPLD Version: 8

MAC Address: 00:C0:FF:DA:52:F9

WWNN: 500C0FFDA8C92000

IP Address: 192.100.1.100

IP Subnet Mask: 255.255.255.0

IP Gateway: 192.100.1.253

Disks: 12

Vdisks: 2

Cache Memory Size (MB): 1024

Host Ports: 4

Disk Channels: 2

Disk Bus Type: SAS

Status: Running

Failed Over: Yes

Fail Over Reason: PCIE link recovery failed

These events after my operation :

2015-06-12 14:16:24 [310] #B34535: MSA2312sa Array SN#00C0FFDA52F9 Controller B

INFORMATIONAL Discovery and initialization of enclosure data was completed follo

wing a rescan.

2015-06-12 14:16:08 [19] #B34534: MSA2312sa Array SN#00C0FFDA52F9 Controller B I

NFORMATIONAL A rescan-bus operation was done. (number of disks that were found:

12, number of enclosures that were found: 1) (rescan reason code: 24)

2015-06-12 14:16:03 [211] #B34533: MSA2312sa Array SN#00C0FFDA52F9 Controller B

INFORMATIONAL The SAS topology has changed (components were added or removed). (

Channel: 0, number of elements: 69, expanders: 1, native levels: 1, partner leve

ls: 0, device PHYs: 13)

2015-06-12 14:16:03 [211] #B34532: MSA2312sa Array SN#00C0FFDA52F9 Controller B

INFORMATIONAL The SAS topology has changed (components were added or removed). (

Channel: 1, number of elements: 5, expanders: 0, native levels: 0, partner level

s: 0, device PHYs: 0)

2015-06-12 14:15:57 [114] #B34531: MSA2312sa Array SN#00C0FFDA52F9 Controller B

INFORMATIONAL Disk link down. (Channel: 1)

2015-06-12 14:15:57 [114] #B34530: MSA2312sa Array SN#00C0FFDA52F9 Controller B

INFORMATIONAL Disk link down. (Channel: 0)

2015-06-12 14:15:57 [71] #B34529: MSA2312sa Array SN#00C0FFDA52F9 Controller B I

NFORMATIONAL Failover initiated, failover set A

2015-06-12 14:15:57 [194] #B34528: MSA2312sa Array SN#00C0FFDA52F9 Controller B

INFORMATIONAL Auto-write-through trigger event: partner processor down.

2015-06-12 14:15:57 [84] #B34527: MSA2312sa Array SN#00C0FFDA52F9 Controller B W

ARNING Killed partner controller. (reason: PCIE link recovery failed [failover r

eason code: 29])

2015-06-12 14:15:45 [175] #B34526: MSA2312sa Array SN#00C0FFDA52F9 Controller B

INFORMATIONAL The Ethernet link is down for controller B.

2015-06-12 14:15:45 [181] #B34525: MSA2312sa Array SN#00C0FFDA52F9 Controller B

INFORMATIONAL LAN configuration parameters have been set.

2015-06-12 14:14:06 [181] #A6284: MSA2312sa Array SN#00C0FFDA52F9 Controller A I

NFORMATIONAL LAN configuration parameters have been set.

2015-06-12 14:14:06 [28] #A6283: MSA2312sa Array SN#00C0FFDA52F9 Controller A IN

FORMATIONAL Controller configuration parameters have been changed.

2015-06-12 14:14:05 [141] #A6282: MSA2312sa Array SN#00C0FFDA52F9 Controller A I

NFORMATIONAL The Management Controller IP address has changed. (new IP address:

IP changed to: 192.100.1.191)

********************************************************************

For completeness, the only controller B is connected to the host through the sas ports.

Andrea

[AnkitM]

1. Reseat Controller A again and try to change the IP Address via CLI to the same subnet and then check if it works or not?

2. If it doesn't Power Down MSA, Unseat Controller B, Power On MSA only with Controller A and check if you can ping Controller A after assigning valid IP (if not already), If yes then Insert Controller B and verify both the Controllers are pingable or not.

3. If not, then either Midplane Chassis or Contoller A has a fault.