MSA Storage
cancel
Showing results for 
Search instead for 
Did you mean: 

MSA P2000 G3 iSCSI PCIE link recovery failed

 
SOLVED
Go to solution
CrazyDTosser
Occasional Advisor

MSA P2000 G3 iSCSI PCIE link recovery failed

Hi everybody!

I have this error on controller A. Controller B take the ownership and can not reboot A. Then i reboot it manualy. If i dont reboot it the same error will be on controller B and storage are down.

 

I have latest firmware on all controller and disks.

After firmware update i made factory reset from cli.

 

 

2015-04-17 12:39:16

There is a problem with a FRU. (FRU type: Controller module A, enclosure: 1, product ID: , SN: , version: , related event serial number: B3890, related event code: 313)

 

2015-04-17 12:39:16

RAID controller A failed, reason PCIE link recovery failed. Product ID , S/N )

 

2015-04-17 12:38:21

Critical Error: Fault Type: Page Fault p1: 0x3FBEDD88, p2: 0x0010002, p3: 0x0000000, p4: 0x0000000 CThr: Cache

 

2015-04-17 12:38:01

Killed partner controller. (reason: PCIE link recovery failed)

 

May be i need to downgrade firmware?

 

Component

Controller A

Controller B

 

Bundle Version

TS251P006-02

TS251P006-02

 

Storage Controller Code Version

T251P08-01

T251P08-01

 

Storage Controller Loader Code Version

23.008

23.008

 

Memory Controller FPGA Code Version

F400R02

F400R02

 

Management Controller Code Version

L251R005-11

L251R005-11

 

Management Controller Loader Code Version

2.5

2.5

 

Expander Controller Code Version

2028

2028

 

CPLD Code Version

22

22

 

THX.

 

 

P.S. This thread has been moved from Disk Array to Storage Area Networks (SAN) (Small and Medium Business). - Hp Forum moderator

16 REPLIES 16
AnkitM
Trusted Contributor

Re: MSA P2000 G3 iSCSI PCIE link recovery failed

Can you do a show events and tail it to appx 250events and attach the file?

Was your question answered correctly? If so, please remember to mark your question Answered when you get the correct answer and award KUDOS! to the person providing the answer. This helps others searching for a similar issue.
CrazyDTosser
Occasional Advisor

Re: MSA P2000 G3 iSCSI PCIE link recovery failed

 Hello.

Yes,file in attach

CrazyDTosser
Occasional Advisor

Re: MSA P2000 G3 iSCSI PCIE link recovery failed

Now on my MSA both controllers are stop responding. Logs in attach

Torsten.
Acclaimed Contributor

Re: MSA P2000 G3 iSCSI PCIE link recovery failed

Looks like an issue with controller A:

 

ERROR There is a problem with a FRU. (FRU type: Controller module A

 

It appears A is starting again and again but crashing everytime.

 

Does the array work if you remove A?

 

I would try to shut the array down, remove A and start it up.


Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
CrazyDTosser
Occasional Advisor

Re: MSA P2000 G3 iSCSI PCIE link recovery failed

Okey, I try this. But when i reboot controller A manualy, array work properly.

AnkitM
Trusted Contributor

Re: MSA P2000 G3 iSCSI PCIE link recovery failed

2015-04-21 11:45:17 [081] #B3944: P2000 G3 iSCSI Array SN#00C0FF128AC2 Controller B INFORMATIONAL Kill was released (that is, the partner controller was allowed to boot up), unknown reason.
2015-04-21 08:18:20 [107] #A7555: P2000 G3 iSCSI Array SN#00C0FF128AC2 Controller A ERROR Critical Error: Fault Type: Gen Prot Fault p1: 0x3FBEDD82, p2: 0x0182F37, p3: 0x0000000, p4: 0x0000000 CThr: Cache
2015-04-21 08:17:40 [314] #B3943: P2000 G3 iSCSI Array SN#00C0FF128AC2 Controller B ERROR There is a problem with a FRU. (FRU type: Controller module A, enclosure: 1, product ID: , SN: , version: , related event serial number: B3942, related event code: 313)
2015-04-21 08:17:40 [313] #B3942: P2000 G3 iSCSI Array SN#00C0FF128AC2 Controller B ERROR RAID controller A failed, reason PCIE link recovery failed. Product ID , S/N )

 

Reboot may be a temporary fix.

 

Replace Controller A.

Was your question answered correctly? If so, please remember to mark your question Answered when you get the correct answer and award KUDOS! to the person providing the answer. This helps others searching for a similar issue.
CrazyDTosser
Occasional Advisor

Re: MSA P2000 G3 iSCSI PCIE link recovery failed

Hello everyone. I have a bad news!

After shutdown and detach controller A,  my msa stop responding again after some time. Events in attach.

AnkitM
Trusted Contributor

Re: MSA P2000 G3 iSCSI PCIE link recovery failed

Controller B is Operational. 

 

Who owns the Vdisk? Contoller A or B. I do see Failover completed.

You can check by #show vdisks

You can check status of the Controller by #show controllers

 

Was your question answered correctly? If so, please remember to mark your question Answered when you get the correct answer and award KUDOS! to the person providing the answer. This helps others searching for a similar issue.
CrazyDTosser
Occasional Advisor

Re: MSA P2000 G3 iSCSI PCIE link recovery failed

Owner of all vdisk is B.

Now i change "Controller Redundancy Mode" to Single coontroller.

AnkitM
Trusted Contributor

Re: MSA P2000 G3 iSCSI PCIE link recovery failed

It should work then. Verify if the Volumes are still mapped, if not you can remap it. 

You can also try unmapping and remapping if you are unable to see the shared storage from the host.

 

If you would need Redundancy then you will need to replace Controller A eventually...sooner the better.

Was your question answered correctly? If so, please remember to mark your question Answered when you get the correct answer and award KUDOS! to the person providing the answer. This helps others searching for a similar issue.
CrazyDTosser
Occasional Advisor

Re: MSA P2000 G3 iSCSI PCIE link recovery failed

Hello everyone!

My MSA still working with one controller without reboots since 23 april. Thanks to all.

CrazyDTosser
Occasional Advisor

Re: MSA P2000 G3 iSCSI PCIE link recovery failed

Too early to celebrate...

MSA is going down. Log in attach. Pls help...

AnkitM
Trusted Contributor

Re: MSA P2000 G3 iSCSI PCIE link recovery failed

It seems Controller B went down:

2015-04-28 11:43:04 [056] #B4085: P2000 G3 iSCSI Array SN#00C0FF128CBA Controller B INFORMATIONAL Storage Controller booted up (cold boot - power up). SC firmware version: T251P08-01
2015-04-28 11:38:40 [107] #B4084: P2000 G3 iSCSI Array SN#00C0FF128CBA Controller B ERROR Critical Error: Fault Type: Gen Prot Fault p1: 0x3FBEDD82, p2: 0x0000000, p3: 0x0000000, p4:

 

ESXi Hosts are connected to this MSA?! If yes, Can you check if VAAI is enabled or disabled?

 

Was your question answered correctly? If so, please remember to mark your question Answered when you get the correct answer and award KUDOS! to the person providing the answer. This helps others searching for a similar issue.
CrazyDTosser
Occasional Advisor

Re: MSA P2000 G3 iSCSI PCIE link recovery failed

My MSA connected with two ESXi host version 5.5.

And i disable VAAI now.

AnkitM
Trusted Contributor
Solution

Re: MSA P2000 G3 iSCSI PCIE link recovery failed

Yes, you can disable VAAI by following below VMware KB Article: kb.vmware.com/kb/1033665

Was your question answered correctly? If so, please remember to mark your question Answered when you get the correct answer and award KUDOS! to the person providing the answer. This helps others searching for a similar issue.
CrazyDTosser
Occasional Advisor

Re: MSA P2000 G3 iSCSI PCIE link recovery failed

This is it!  My MSA now work good. Thanks!