MSA P2000 G3 iSCSI PCIE link recovery failed

[CrazyDTosser]

Hi everybody!

I have this error on controller A. Controller B take the ownership and can not reboot A. Then i reboot it manualy. If i dont reboot it the same error will be on controller B and storage are down.

I have latest firmware on all controller and disks.

After firmware update i made factory reset from cli.

2015-04-17 12:39:16

There is a problem with a FRU. (FRU type: Controller module A, enclosure: 1, product ID: , SN: , version: , related event serial number: B3890, related event code: 313)

2015-04-17 12:39:16

RAID controller A failed, reason PCIE link recovery failed. Product ID , S/N )

2015-04-17 12:38:21

Critical Error: Fault Type: Page Fault p1: 0x3FBEDD88, p2: 0x0010002, p3: 0x0000000, p4: 0x0000000 CThr: Cache

2015-04-17 12:38:01

Killed partner controller. (reason: PCIE link recovery failed)

May be i need to downgrade firmware?

Component

Controller A

Controller B

Bundle Version

TS251P006-02

TS251P006-02

Storage Controller Code Version

T251P08-01

T251P08-01

Storage Controller Loader Code Version

23.008

23.008

Memory Controller FPGA Code Version

F400R02

F400R02

Management Controller Code Version

L251R005-11

L251R005-11

Management Controller Loader Code Version

2.5

2.5

Expander Controller Code Version

2028

2028

CPLD Code Version

22

22

THX.

P.S. This thread has been moved from Disk Array to Storage Area Networks (SAN) (Small and Medium Business). - Hp Forum moderator

[AnkitM]

Can you do a show events and tail it to appx 250events and attach the file?

[CrazyDTosser]

 Hello.

Yes,file in attach

[CrazyDTosser]

Now on my MSA both controllers are stop responding. Logs in attach

[Torsten.]

Looks like an issue with controller A:

ERROR There is a problem with a FRU. (FRU type: Controller module A

It appears A is starting again and again but crashing everytime.

Does the array work if you remove A?

I would try to shut the array down, remove A and start it up.

[CrazyDTosser]

Okey, I try this. But when i reboot controller A manualy, array work properly.

[AnkitM]

2015-04-21 11:45:17 [081] #B3944: P2000 G3 iSCSI Array SN#00C0FF128AC2 Controller B INFORMATIONAL Kill was released (that is, the partner controller was allowed to boot up), unknown reason.
2015-04-21 08:18:20 [107] #A7555: P2000 G3 iSCSI Array SN#00C0FF128AC2 Controller A ERROR Critical Error: Fault Type: Gen Prot Fault p1: 0x3FBEDD82, p2: 0x0182F37, p3: 0x0000000, p4: 0x0000000 CThr: Cache
2015-04-21 08:17:40 [314] #B3943: P2000 G3 iSCSI Array SN#00C0FF128AC2 Controller B ERROR There is a problem with a FRU. (FRU type: Controller module A, enclosure: 1, product ID: , SN: , version: , related event serial number: B3942, related event code: 313)
2015-04-21 08:17:40 [313] #B3942: P2000 G3 iSCSI Array SN#00C0FF128AC2 Controller B ERROR RAID controller A failed, reason PCIE link recovery failed. Product ID , S/N )

Reboot may be a temporary fix.

Replace Controller A.

[CrazyDTosser]

Hello everyone. I have a bad news!

After shutdown and detach controller A,  my msa stop responding again after some time. Events in attach.

[AnkitM]

Controller B is Operational. 

Who owns the Vdisk? Contoller A or B. I do see Failover completed.

You can check by #show vdisks

You can check status of the Controller by #show controllers

[CrazyDTosser]

Owner of all vdisk is B.

Now i change "Controller Redundancy Mode" to Single coontroller.