MSA2040 qla2xxx abort commands

[matthias_]

HI,

we have here 2  MSA 2040 FC dual controller with different disk configuration

both systems run the latest GL210R004 firmware.

We have the strange effect that from time to time (mostly once or twice a day) some FC paths are failing and the whole storage is not accessable for 30-60 seconds . There are no logged errors on the storage itself.

We have it on different Systems (Ubuntu 14.04.2, XEN Server 6.2, Xen Server 6.5 ....)

The rest of the storages we have (some lsi devices HP MSA P2000 etc.) are running fine connected to the same servers in the same SANs

The messages from the qla2xxx drivers are more or less the same on all Servers:

[2399570.386292] qla2xxx [0000:05:00.0]-8802:1: Aborting from RISC nexus=1:1:2 sp=ffff8801504a3d40 cmd=ffff880a480b1e00
[2399570.386447] qla2xxx [0000:05:00.0]-3822:1: FCP command status: 0x5-0x0 (0x80000) nexus=1:1:2 portid=0b0a00 oxid=0x292 cdb=8a0000000008aefc18a8 len=0x1000 rsp_info=0x0 resid=0x0 fw_resid=0x0.
[2399570.386474] qla2xxx [0000:05:00.0]-8804:1: Abort command mbx success cmd=ffff880a480b1e00.
[2399570.386481] qla2xxx [0000:05:00.0]-801c:1: Abort command issued nexus=1:1:2 --  1 2002.
[2399570.386485] qla2xxx [0000:05:00.0]-8802:1: Aborting from RISC nexus=1:1:2 sp=ffff8803fa456e40 cmd=ffff8801713f6000
[2399570.386628] qla2xxx [0000:05:00.0]-3822:1: FCP command status: 0x5-0x0 (0x80000) nexus=1:1:2 portid=0b0a00 oxid=0x2a1 cdb=00000000000000000000 len=0x0 rsp_info=0x0 resid=0x0 fw_resid=0x0.
[2399570.386659] qla2xxx [0000:05:00.0]-8804:1: Abort command mbx success cmd=ffff8801713f6000.
[2399570.386664] qla2xxx [0000:05:00.0]-801c:1: Abort command issued nexus=1:1:2 --  1 2002.
[2399570.387581] device-mapper: multipath: Failing path 8:176.

on the xen servers we get:

[1672725.840070] qla2xxx [0000:42:00.0]-8802:7: Aborting from RISC nexus=7:0:1 sp=ffff88012455d2c0 cmd=ffff880160b3e400
[1672725.840220] qla2xxx [0000:42:00.0]-3822:7: FCP command status: 0x5-0x0 (0x80000) nexus=7:0:1 portid=0c1400 oxid=0x135 cdb=28000000780000000100 len=0x200 rsp_info=0x0 resid=0x0 fw_resid=0x0.
[1672725.840222] qla2xxx [0000:42:00.0]-8804:7: Abort command mbx success cmd=ffff880160b3e400.
[1672725.840233] qla2xxx [0000:42:00.0]-801c:7: Abort command issued nexus=7:0:1 -- 1 2002.
[1701372.489615] qla2xxx [0000:42:00.0]-8802:7: Aborting from RISC nexus=7:0:1 sp=ffff88011ae0ea00 cmd=ffff880111699a00
[1701372.489741] qla2xxx [0000:42:00.0]-3822:7: FCP command status: 0x5-0x0 (0x80000) nexus=7:0:1 portid=0c1400 oxid=0xc9 cdb=00000000000000000000 len=0x0 rsp_info=0x0 resid=0x0 fw_resid=0x0.
[1701372.489747] qla2xxx [0000:42:00.0]-8804:7: Abort command mbx success cmd=ffff880111699a00.
[1701372.489752] qla2xxx [0000:42:00.0]-801c:7: Abort command issued nexus=7:0:1 -- 1 2002.
[1701372.489754] qla2xxx [0000:42:00.0]-8802:7: Aborting from RISC nexus=7:0:1 sp=ffff88012455ca00 cmd=ffff880171721000
[1701372.489836] qla2xxx [0000:42:00.0]-3822:7: FCP command status: 0x5-0x0 (0x80000) nexus=7:0:1 portid=0c1400 oxid=0xcb cdb=28000000780000000100 len=0x200 rsp_info=0x0 resid=0x0 fw_resid=0x0.
[1701372.489840] qla2xxx [0000:42:00.0]-8804:7: Abort command mbx success cmd=ffff880171721000.
[1701372.489844] qla2xxx [0000:42:00.0]-801c:7: Abort command issued nexus=7:0:1 -- 1 2002.
[1701372.489846] qla2xxx [0000:42:00.0]-8802:7: Aborting from RISC nexus=7:0:2 sp=ffff88011ae0f7c0 cmd=ffff880111698700
[1701372.489925] qla2xxx [0000:42:00.0]-3822:7: FCP command status: 0x5-0x0 (0x80000) nexus=7:0:2 portid=0c1400 oxid=0xca cdb=00000000000000000000 len=0x0 rsp_info=0x0 resid=0x0 fw_resid=0x0.
[1701372.489930] qla2xxx [0000:42:00.0]-8804:7: Abort command mbx success cmd=ffff880111698700.
[1701372.489932] qla2xxx [0000:42:00.0]-801c:7: Abort command issued nexus=7:0:2 -- 1 2002.
[1701372.489934] qla2xxx [0000:42:00.0]-8802:7: Aborting from RISC nexus=7:0:2 sp=ffff88012455c500 cmd=ffff880160b2d200
[1701372.490016] qla2xxx [0000:42:00.0]-3822:7: FCP command status: 0x5-0x0 (0x80000) nexus=7:0:2 portid=0c1400 oxid=0xcc cdb=8a0000000000032e2d88 len=0x1000 rsp_info=0x0 resid=0x0 fw_resid=0x0.
[1701372.490019] qla2xxx [0000:42:00.0]-8804:7: Abort command mbx success cmd=ffff880160b2d200.
[1701372.490023] qla2xxx [0000:42:00.0]-801c:7: Abort command issued nexus=7:0:2 -- 1 2002.
[1701372.490025] qla2xxx [0000:42:00.0]-8802:7: Aborting from RISC nexus=7:0:1 sp=ffff880119e68f00 cmd=ffff88010d07e200
[1701372.490104] qla2xxx [0000:42:00.0]-3822:7: FCP command status: 0x5-0x0 (0x80000) nexus=7:0:1 portid=0c1400 oxid=0xcd cdb=35000000000000000000 len=0x0 rsp_info=0x0 resid=0x0 fw_resid=0x0.
[1701372.490109] qla2xxx [0000:42:00.0]-8804:7: Abort command mbx success cmd=ffff88010d07e200.
[1701372.490111] qla2xxx [0000:42:00.0]-801c:7: Abort command issued nexus=7:0:1 -- 1 2002.
[1701372.757769] device-mapper: multipath: Failing path 8:16.
[1701372.758947] device-mapper: multipath: Failing path 8:32.
[1728468.258835] qla2xxx [0000:42:00.0]-8802:7: Aborting from RISC nexus=7:0:1 sp=ffff88016e0e5540 cmd=ffff8801609f4100
[1728468.259221] qla2xxx [0000:42:00.0]-3822:7: FCP command status: 0x5-0x0 (0x80000) nexus=7:0:1 portid=0c1400 oxid=0x7af cdb=2a000000780800000200 len=0x400 rsp_info=0x0 resid=0x0 fw_resid=0x0.
[1728468.259248] qla2xxx [0000:42:00.0]-8804:7: Abort command mbx success cmd=ffff8801609f4100.
[1728468.259257] qla2xxx [0000:42:00.0]-801c:7: Abort command issued nexus=7:0:1 -- 1 2002.

I don`t think that is a Hardware issue, becaue these are 2 MSAs 5 different Servers on 4 different Switches .

Any Idea what could be the Problem?

Many Thanks

Matthias

[matthias_]

Nobody the same issues?

In the meantime I updated the qla2xxx driver on the xen machines to the latest version, but no luck.

Here is some additional output of the driver, maybe it helps:

[35743.827614] qla2xxx [0000:42:00.0]-8802:7: Aborting from RISC nexus=7:0:2 sp=ffff880156afd680 cmd=ffff88015a98ed00 handle=376
[35743.827619] qla2xxx [0000:42:00.0]-188c:7: Entered qla24xx_abort_command.
[35743.827624] qla2xxx [0000:42:00.0]-1838:7: Entered qla2x00_issue_iocb_timeout.
[35743.827626] qla2xxx [0000:42:00.0]-1800:7: Entered qla2x00_mailbox_command.
[35743.827627] qla2xxx [0000:42:00.0]-1806:7: Prepare to issue mbox cmd=0x54.
[35743.827629] qla2xxx [0000:42:00.0]-1911:7: Mailbox registers (OUT):
[35743.827630] qla2xxx [0000:42:00.0]-1912:7: mbox[0]<-0x0054
[35743.827632] qla2xxx [0000:42:00.0]-1912:7: mbox[1]<-0x0000
[35743.827633] qla2xxx [0000:42:00.0]-1912:7: mbox[2]<-0x715d
[35743.827634] qla2xxx [0000:42:00.0]-1912:7: mbox[3]<-0xd100
[35743.827636] qla2xxx [0000:42:00.0]-1912:7: mbox[6]<-0x0000
[35743.827637] qla2xxx [0000:42:00.0]-1912:7: mbox[7]<-0x0010
[35743.827638] qla2xxx [0000:42:00.0]-1917:7: I/O Address = ffffc9000006a0c0.
[35743.827640] qla2xxx [0000:42:00.0]-180f:7: Going to unlock irq & waiting for interrupts. jiffies=100872880.
[35743.828048] qla2xxx [0000:42:00.0]-3822:7: FCP command status: 0x5-0x0 (0x80000) nexus=7:0:2 portid=0c1400 oxid=0x850 cdb=8a0000000000065bfd68 len=0x3000 rsp_info=0x0 resid=0x0 fw_resid=0x0 sp=ffff880156afd680 cp=ffff88015a98ed00.
[35743.828056] qla2xxx [0000:42:00.0]-1814:7: Cmd=54 completed.
[35743.828057] qla2xxx [0000:42:00.0]-1913:7: Mailbox registers (IN):
[35743.828059] qla2xxx [0000:42:00.0]-1914:7: mbox[0]->0x4000
[35743.828060] qla2xxx [0000:42:00.0]-1914:7: mbox[2]->0x715d
[35743.828061] qla2xxx [0000:42:00.0]-1821:7: Done qla2x00_mailbox_command.
[35743.828063] qla2xxx [0000:42:00.0]-183a:7: Done qla2x00_issue_iocb_timeout.
[35743.828064] qla2xxx [0000:42:00.0]-1891:7: Done qla24xx_abort_command.
[35743.828066] qla2xxx [0000:42:00.0]-8804:7: Abort command mbx success cmd=ffff88015a98ed00.
[35743.828071] qla2xxx [0000:42:00.0]-801c:7: Abort command issued nexus=7:0:2 --  1 2002.
[35743.828074] qla2xxx [0000:42:00.0]-8802:7: Aborting from RISC nexus=7:0:2 sp=ffff880156afdcc0 cmd=ffff8801633e2e00 handle=377
[35743.828075] qla2xxx [0000:42:00.0]-188c:7: Entered qla24xx_abort_command.
[35743.828077] qla2xxx [0000:42:00.0]-1838:7: Entered qla2x00_issue_iocb_timeout.
[35743.828078] qla2xxx [0000:42:00.0]-1800:7: Entered qla2x00_mailbox_command.
[35743.828080] qla2xxx [0000:42:00.0]-1806:7: Prepare to issue mbox cmd=0x54.
[35743.828081] qla2xxx [0000:42:00.0]-1911:7: Mailbox registers (OUT):
[35743.828082] qla2xxx [0000:42:00.0]-1912:7: mbox[0]<-0x0054
[35743.828084] qla2xxx [0000:42:00.0]-1912:7: mbox[1]<-0x0000
[35743.828085] qla2xxx [0000:42:00.0]-1912:7: mbox[2]<-0x715d
[35743.828087] qla2xxx [0000:42:00.0]-1912:7: mbox[3]<-0xd100
[35743.828088] qla2xxx [0000:42:00.0]-1912:7: mbox[6]<-0x0000
[35743.828089] qla2xxx [0000:42:00.0]-1912:7: mbox[7]<-0x0010
[35743.828091] qla2xxx [0000:42:00.0]-1917:7: I/O Address = ffffc9000006a0c0.
[35743.828092] qla2xxx [0000:42:00.0]-180f:7: Going to unlock irq & waiting for interrupts. jiffies=100872880.
[35743.828443] qla2xxx [0000:42:00.0]-3822:7: FCP command status: 0x5-0x0 (0x80000) nexus=7:0:2 portid=0c1400 oxid=0x851 cdb=8a0000000000065c0c20 len=0x22000 rsp_info=0x0 resid=0x0 fw_resid=0x0 sp=ffff880156afdcc0 cp=ffff8801633e2e00.
[35743.828466] qla2xxx [0000:42:00.0]-1814:7: Cmd=54 completed.
[35743.828468] qla2xxx [0000:42:00.0]-1913:7: Mailbox registers (IN):
[35743.828469] qla2xxx [0000:42:00.0]-1914:7: mbox[0]->0x4000
[35743.828470] qla2xxx [0000:42:00.0]-1914:7: mbox[2]->0x715d
[35743.828472] qla2xxx [0000:42:00.0]-1821:7: Done qla2x00_mailbox_command.
[35743.828473] qla2xxx [0000:42:00.0]-183a:7: Done qla2x00_issue_iocb_timeout.
[35743.828475] qla2xxx [0000:42:00.0]-1891:7: Done qla24xx_abort_command.
[35743.828476] qla2xxx [0000:42:00.0]-8804:7: Abort command mbx success cmd=ffff8801633e2e00.
[35743.828484] qla2xxx [0000:42:00.0]-801c:7: Abort command issued nexus=7:0:2 --  1 2002.
[35743.828487] qla2xxx [0000:42:00.0]-8802:7: Aborting from RISC nexus=7:0:2 sp=ffff880156afc140 cmd=ffff88015a843700 handle=378
[35743.828489] qla2xxx [0000:42:00.0]-188c:7: Entered qla24xx_abort_command.
[35743.828491] qla2xxx [0000:42:00.0]-1838:7: Entered qla2x00_issue_iocb_timeout.
[35743.828492] qla2xxx [0000:42:00.0]-1800:7: Entered qla2x00_mailbox_command.
[35743.828494] qla2xxx [0000:42:00.0]-1806:7: Prepare to issue mbox cmd=0x54.
[35743.828495] qla2xxx [0000:42:00.0]-1911:7: Mailbox registers (OUT):
[35743.828496] qla2xxx [0000:42:00.0]-1912:7: mbox[0]<-0x0054
[35743.828497] qla2xxx [0000:42:00.0]-1912:7: mbox[1]<-0x0000
[35743.828499] qla2xxx [0000:42:00.0]-1912:7: mbox[2]<-0x715d
[35743.828500] qla2xxx [0000:42:00.0]-1912:7: mbox[3]<-0xd100
[35743.828502] qla2xxx [0000:42:00.0]-1912:7: mbox[6]<-0x0000
[35743.828503] qla2xxx [0000:42:00.0]-1912:7: mbox[7]<-0x0010
[35743.828504] qla2xxx [0000:42:00.0]-1917:7: I/O Address = ffffc9000006a0c0.
[35743.828506] qla2xxx [0000:42:00.0]-180f:7: Going to unlock irq & waiting for interrupts. jiffies=100872880.
[35743.828864] qla2xxx [0000:42:00.0]-3822:7: FCP command status: 0x5-0x0 (0x80000) nexus=7:0:2 portid=0c1400 oxid=0x852 cdb=8a0000000000065c1be8 len=0x22000 rsp_info=0x0 resid=0x0 fw_resid=0x0 sp=ffff880156afc140 cp=ffff88015a843700.
[35743.828887] qla2xxx [0000:42:00.0]-1814:7: Cmd=54 completed.
[35743.828888] qla2xxx [0000:42:00.0]-1913:7: Mailbox registers (IN):
[35743.828889] qla2xxx [0000:42:00.0]-1914:7: mbox[0]->0x4000
[35743.828891] qla2xxx [0000:42:00.0]-1914:7: mbox[2]->0x715d
[35743.828892] qla2xxx [0000:42:00.0]-1821:7: Done qla2x00_mailbox_command.
[35743.828894] qla2xxx [0000:42:00.0]-183a:7: Done qla2x00_issue_iocb_timeout.
[35743.828895] qla2xxx [0000:42:00.0]-1891:7: Done qla24xx_abort_command.
[35743.828897] qla2xxx [0000:42:00.0]-8804:7: Abort command mbx success cmd=ffff88015a843700.
[35743.828907] qla2xxx [0000:42:00.0]-801c:7: Abort command issued nexus=7:0:2 --  1 2002.
[35743.828909] qla2xxx [0000:42:00.0]-8802:7: Aborting from RISC nexus=7:0:2 sp=ffff880156afde00 cmd=ffff88015a843400 handle=379
[35743.828911] qla2xxx [0000:42:00.0]-188c:7: Entered qla24xx_abort_command.
[35743.828913] qla2xxx [0000:42:00.0]-1838:7: Entered qla2x00_issue_iocb_timeout.
[35743.828914] qla2xxx [0000:42:00.0]-1800:7: Entered qla2x00_mailbox_command.
[35743.828916] qla2xxx [0000:42:00.0]-1806:7: Prepare to issue mbox cmd=0x54.
[35743.828917] qla2xxx [0000:42:00.0]-1911:7: Mailbox registers (OUT):
[35743.828918] qla2xxx [0000:42:00.0]-1912:7: mbox[0]<-0x0054
[35743.828920] qla2xxx [0000:42:00.0]-1912:7: mbox[1]<-0x0000
[35743.828921] qla2xxx [0000:42:00.0]-1912:7: mbox[2]<-0x715d
[35743.828922] qla2xxx [0000:42:00.0]-1912:7: mbox[3]<-0xd100
[35743.828924] qla2xxx [0000:42:00.0]-1912:7: mbox[6]<-0x0000
[35743.828925] qla2xxx [0000:42:00.0]-1912:7: mbox[7]<-0x0010
[35743.828926] qla2xxx [0000:42:00.0]-1917:7: I/O Address = ffffc9000006a0c0.
[35743.828928] qla2xxx [0000:42:00.0]-180f:7: Going to unlock irq & waiting for interrupts. jiffies=100872880.
[35743.829252] qla2xxx [0000:42:00.0]-3822:7: FCP command status: 0x5-0x0 (0x80000) nexus=7:0:2 portid=0c1400 oxid=0x853 cdb=8a0000000000065d1d98 len=0x13000 rsp_info=0x0 resid=0x0 fw_resid=0x0 sp=ffff880156afde00 cp=ffff88015a843400.
[35743.829279] qla2xxx [0000:42:00.0]-1814:7: Cmd=54 completed.
[35743.829280] qla2xxx [0000:42:00.0]-1913:7: Mailbox registers (IN):
[35743.829281] qla2xxx [0000:42:00.0]-1914:7: mbox[0]->0x4000
[35743.829283] qla2xxx [0000:42:00.0]-1914:7: mbox[2]->0x715d
[35743.829284] qla2xxx [0000:42:00.0]-1821:7: Done qla2x00_mailbox_command.
[35743.829285] qla2xxx [0000:42:00.0]-183a:7: Done qla2x00_issue_iocb_timeout.
[35743.829287] qla2xxx [0000:42:00.0]-1891:7: Done qla24xx_abort_command.
[35743.829288] qla2xxx [0000:42:00.0]-8804:7: Abort command mbx success cmd=ffff88015a843400.
[35743.829299] qla2xxx [0000:42:00.0]-801c:7: Abort command issued nexus=7:0:2 --  1 2002.
[35743.829301] qla2xxx [0000:42:00.0]-8802:7: Aborting from RISC nexus=7:0:2 sp=ffff880156afda40 cmd=ffff880171f2e700 handle=37a
[35743.829302] qla2xxx [0000:42:00.0]-188c:7: Entered qla24xx_abort_command.
[35743.829304] qla2xxx [0000:42:00.0]-1838:7: Entered qla2x00_issue_iocb_timeout.
[35743.829306] qla2xxx [0000:42:00.0]-1800:7: Entered qla2x00_mailbox_command.
[35743.829307] qla2xxx [0000:42:00.0]-1806:7: Prepare to issue mbox cmd=0x54.
[35743.829308] qla2xxx [0000:42:00.0]-1911:7: Mailbox registers (OUT):
[35743.829309] qla2xxx [0000:42:00.0]-1912:7: mbox[0]<-0x0054
[35743.829311] qla2xxx [0000:42:00.0]-1912:7: mbox[1]<-0x0000
[35743.829312] qla2xxx [0000:42:00.0]-1912:7: mbox[2]<-0x715d
[35743.829314] qla2xxx [0000:42:00.0]-1912:7: mbox[3]<-0xd100
[35743.829315] qla2xxx [0000:42:00.0]-1912:7: mbox[6]<-0x0000
[35743.829316] qla2xxx [0000:42:00.0]-1912:7: mbox[7]<-0x0010
[35743.829318] qla2xxx [0000:42:00.0]-1917:7: I/O Address = ffffc9000006a0c0.
[35743.829319] qla2xxx [0000:42:00.0]-180f:7: Going to unlock irq & waiting for interrupts. jiffies=100872880.
[35743.829646] qla2xxx [0000:42:00.0]-3822:7: FCP command status: 0x5-0x0 (0x80000) nexus=7:0:2 portid=0c1400 oxid=0x854 cdb=8a0000000000065d1e90 len=0x22000 rsp_info=0x0 resid=0x0 fw_resid=0x0 sp=ffff880156afda40 cp=ffff880171f2e700.
[35743.829672] qla2xxx [0000:42:00.0]-1814:7: Cmd=54 completed.
[35743.829673] qla2xxx [0000:42:00.0]-1913:7: Mailbox registers (IN):
[35743.829675] qla2xxx [0000:42:00.0]-1914:7: mbox[0]->0x4000
[35743.829676] qla2xxx [0000:42:00.0]-1914:7: mbox[2]->0x715d
[35743.829677] qla2xxx [0000:42:00.0]-1821:7: Done qla2x00_mailbox_command.
[35743.829679] qla2xxx [0000:42:00.0]-183a:7: Done qla2x00_issue_iocb_timeout.
[35743.829680] qla2xxx [0000:42:00.0]-1891:7: Done qla24xx_abort_command.
[35743.829681] qla2xxx [0000:42:00.0]-8804:7: Abort command mbx success cmd=ffff880171f2e700.
[35743.829690] qla2xxx [0000:42:00.0]-801c:7: Abort command issued nexus=7:0:2 --  1 2002.
[35743.829692] qla2xxx [0000:42:00.0]-8802:7: Aborting from RISC nexus=7:0:2 sp=ffff880156afd2c0 cmd=ffff88015210fa00 handle=37b
[35743.829694] qla2xxx [0000:42:00.0]-188c:7: Entered qla24xx_abort_command.
[35743.829696] qla2xxx [0000:42:00.0]-1838:7: Entered qla2x00_issue_iocb_timeout.
[35743.829697] qla2xxx [0000:42:00.0]-1800:7: Entered qla2x00_mailbox_command.
[35743.829698] qla2xxx [0000:42:00.0]-1806:7: Prepare to issue mbox cmd=0x54.
[35743.829699] qla2xxx [0000:42:00.0]-1911:7: Mailbox registers (OUT):
[35743.829701] qla2xxx [0000:42:00.0]-1912:7: mbox[0]<-0x0054
[35743.829702] qla2xxx [0000:42:00.0]-1912:7: mbox[1]<-0x0000
[35743.829703] qla2xxx [0000:42:00.0]-1912:7: mbox[2]<-0x715d
[35743.829705] qla2xxx [0000:42:00.0]-1912:7: mbox[3]<-0xd100
[35743.829706] qla2xxx [0000:42:00.0]-1912:7: mbox[6]<-0x0000
[35743.829707] qla2xxx [0000:42:00.0]-1912:7: mbox[7]<-0x0010
[35743.829709] qla2xxx [0000:42:00.0]-1917:7: I/O Address = ffffc9000006a0c0.
[35743.829710] qla2xxx [0000:42:00.0]-180f:7: Going to unlock irq & waiting for interrupts. jiffies=100872880.
[35743.830034] qla2xxx [0000:42:00.0]-3822:7: FCP command status: 0x5-0x0 (0x80000) nexus=7:0:2 portid=0c1400 oxid=0x855 cdb=8a0000000000065d2dd0 len=0x22000 rsp_info=0x0 resid=0x0 fw_resid=0x0 sp=ffff880156afd2c0 cp=ffff88015210fa00.
[35743.830062] qla2xxx [0000:42:00.0]-1814:7: Cmd=54 completed.
[35743.830063] qla2xxx [0000:42:00.0]-1913:7: Mailbox registers (IN):
[35743.830064] qla2xxx [0000:42:00.0]-1914:7: mbox[0]->0x4000
[35743.830066] qla2xxx [0000:42:00.0]-1914:7: mbox[2]->0x715d
[35743.830067] qla2xxx [0000:42:00.0]-1821:7: Done qla2x00_mailbox_command.
[35743.830068] qla2xxx [0000:42:00.0]-183a:7: Done qla2x00_issue_iocb_timeout.
[35743.830070] qla2xxx [0000:42:00.0]-1891:7: Done qla24xx_abort_command.
[35743.830072] qla2xxx [0000:42:00.0]-8804:7: Abort command mbx success cmd=ffff88015210fa00.
[35743.830080] qla2xxx [0000:42:00.0]-801c:7: Abort command issued nexus=7:0:2 --  1 2002.
[35743.830083] qla2xxx [0000:42:00.0]-8802:7: Aborting from RISC nexus=7:0:2 sp=ffff880156afc000 cmd=ffff8801631c7b00 handle=37c
[35743.830084] qla2xxx [0000:42:00.0]-188c:7: Entered qla24xx_abort_command.
[35743.830086] qla2xxx [0000:42:00.0]-1838:7: Entered qla2x00_issue_iocb_timeout.
[35743.830087] qla2xxx [0000:42:00.0]-1800:7: Entered qla2x00_mailbox_command.
[35743.830089] qla2xxx [0000:42:00.0]-1806:7: Prepare to issue mbox cmd=0x54.
[35743.830090] qla2xxx [0000:42:00.0]-1911:7: Mailbox registers (OUT):
[35743.830091] qla2xxx [0000:42:00.0]-1912:7: mbox[0]<-0x0054
[35743.830092] qla2xxx [0000:42:00.0]-1912:7: mbox[1]<-0x0000
[35743.830094] qla2xxx [0000:42:00.0]-1912:7: mbox[2]<-0x715d
[35743.830095] qla2xxx [0000:42:00.0]-1912:7: mbox[3]<-0xd100
[35743.830096] qla2xxx [0000:42:00.0]-1912:7: mbox[6]<-0x0000
[35743.830098] qla2xxx [0000:42:00.0]-1912:7: mbox[7]<-0x0010
[35743.830099] qla2xxx [0000:42:00.0]-1917:7: I/O Address = ffffc9000006a0c0.
[35743.830100] qla2xxx [0000:42:00.0]-180f:7: Going to unlock irq & waiting for interrupts. jiffies=100872880.
[35743.830454] qla2xxx [0000:42:00.0]-3822:7: FCP command status: 0x5-0x0 (0x80000) nexus=7:0:2 portid=0c1400 oxid=0x856 cdb=8a0000000000065d2ff0 len=0xd000 rsp_info=0x0 resid=0x0 fw_resid=0x0 sp=ffff880156afc000 cp=ffff8801631c7b00.
[35743.830457] qla2xxx [0000:42:00.0]-1814:7: Cmd=54 completed.
[35743.830458] qla2xxx [0000:42:00.0]-1913:7: Mailbox registers (IN):
[35743.830459] qla2xxx [0000:42:00.0]-1914:7: mbox[0]->0x4000
[35743.830461] qla2xxx [0000:42:00.0]-1914:7: mbox[2]->0x715d
[35743.830462] qla2xxx [0000:42:00.0]-1821:7: Done qla2x00_mailbox_command.
[35743.830463] qla2xxx [0000:42:00.0]-183a:7: Done qla2x00_issue_iocb_timeout.
[35743.830465] qla2xxx [0000:42:00.0]-1891:7: Done qla24xx_abort_command.
[35743.830466] qla2xxx [0000:42:00.0]-8804:7: Abort command mbx success cmd=ffff8801631c7b00.
[35743.830470] qla2xxx [0000:42:00.0]-801c:7: Abort command issued nexus=7:0:2 --  1 2002.
[35743.830472] qla2xxx [0000:42:00.0]-8802:7: Aborting from RISC nexus=7:0:2 sp=ffff880156afd040 cmd=ffff880160512a00 handle=37d
[35743.830474] qla2xxx [0000:42:00.0]-188c:7: Entered qla24xx_abort_command.
[35743.830476] qla2xxx [0000:42:00.0]-1838:7: Entered qla2x00_issue_iocb_timeout.
[35743.830477] qla2xxx [0000:42:00.0]-1800:7: Entered qla2x00_mailbox_command.
[35743.830478] qla2xxx [0000:42:00.0]-1806:7: Prepare to issue mbox cmd=0x54.
[35743.830479] qla2xxx [0000:42:00.0]-1911:7: Mailbox registers (OUT):
[35743.830481] qla2xxx [0000:42:00.0]-1912:7: mbox[0]<-0x0054
[35743.830482] qla2xxx [0000:42:00.0]-1912:7: mbox[1]<-0x0000
[35743.830483] qla2xxx [0000:42:00.0]-1912:7: mbox[2]<-0x715d
[35743.830485] qla2xxx [0000:42:00.0]-1912:7: mbox[3]<-0xd100
[35743.830486] qla2xxx [0000:42:00.0]-1912:7: mbox[6]<-0x0000
[35743.830487] qla2xxx [0000:42:00.0]-1912:7: mbox[7]<-0x0010
[35743.830489] qla2xxx [0000:42:00.0]-1917:7: I/O Address = ffffc9000006a0c0.
[35743.830490] qla2xxx [0000:42:00.0]-180f:7: Going to unlock irq & waiting for interrupts. jiffies=100872880.
[35743.830846] qla2xxx [0000:42:00.0]-3822:7: FCP command status: 0x5-0x0 (0x80000) nexus=7:0:2 portid=0c1400 oxid=0x857 cdb=8a000000000008122900 len=0x1000 rsp_info=0x0 resid=0x0 fw_resid=0x0 sp=ffff880156afd040 cp=ffff880160512a00.
[35743.830850] qla2xxx [0000:42:00.0]-1814:7: Cmd=54 completed.
[35743.830851] qla2xxx [0000:42:00.0]-1913:7: Mailbox registers (IN):
[35743.830852] qla2xxx [0000:42:00.0]-1914:7: mbox[0]->0x4000
[35743.830853] qla2xxx [0000:42:00.0]-1914:7: mbox[2]->0x715d
[35743.830855] qla2xxx [0000:42:00.0]-1821:7: Done qla2x00_mailbox_command.
[35743.830856] qla2xxx [0000:42:00.0]-183a:7: Done qla2x00_issue_iocb_timeout.
[35743.830858] qla2xxx [0000:42:00.0]-1891:7: Done qla24xx_abort_command.
[35743.830859] qla2xxx [0000:42:00.0]-8804:7: Abort command mbx success cmd=ffff880160512a00.
[35743.830865] qla2xxx [0000:42:00.0]-801c:7: Abort command issued nexus=7:0:2 --  1 2002.
[35743.830867] qla2xxx [0000:42:00.0]-8802:7: Aborting from RISC nexus=7:0:2 sp=ffff880156afcf00 cmd=ffff88015a8e1500 handle=37e
[35743.830868] qla2xxx [0000:42:00.0]-188c:7: Entered qla24xx_abort_command.
[35743.830870] qla2xxx [0000:42:00.0]-1838:7: Entered qla2x00_issue_iocb_timeout.
[35743.830872] qla2xxx [0000:42:00.0]-1800:7: Entered qla2x00_mailbox_command.
[35743.830873] qla2xxx [0000:42:00.0]-1806:7: Prepare to issue mbox cmd=0x54.
[35743.830874] qla2xxx [0000:42:00.0]-1911:7: Mailbox registers (OUT):
[35743.830875] qla2xxx [0000:42:00.0]-1912:7: mbox[0]<-0x0054
[35743.830877] qla2xxx [0000:42:00.0]-1912:7: mbox[1]<-0x0000
[35743.830878] qla2xxx [0000:42:00.0]-1912:7: mbox[2]<-0x715d
[35743.830879] qla2xxx [0000:42:00.0]-1912:7: mbox[3]<-0xd100
[35743.830881] qla2xxx [0000:42:00.0]-1912:7: mbox[6]<-0x0000
[35743.830882] qla2xxx [0000:42:00.0]-1912:7: mbox[7]<-0x0010
[35743.830883] qla2xxx [0000:42:00.0]-1917:7: I/O Address = ffffc9000006a0c0.
[35743.830885] qla2xxx [0000:42:00.0]-180f:7: Going to unlock irq & waiting for interrupts. jiffies=100872880.
[35743.831240] qla2xxx [0000:42:00.0]-3822:7: FCP command status: 0x5-0x0 (0x80000) nexus=7:0:2 portid=0c1400 oxid=0x858 cdb=8a000000000003bbe398 len=0x3000 rsp_info=0x0 resid=0x0 fw_resid=0x0 sp=ffff880156afcf00 cp=ffff88015a8e1500.
[35743.831245] qla2xxx [0000:42:00.0]-1814:7: Cmd=54 completed.
[35743.831246] qla2xxx [0000:42:00.0]-1913:7: Mailbox registers (IN):
[35743.831248] qla2xxx [0000:42:00.0]-1914:7: mbox[0]->0x4000
[35743.831249] qla2xxx [0000:42:00.0]-1914:7: mbox[2]->0x715d
[35743.831250] qla2xxx [0000:42:00.0]-1821:7: Done qla2x00_mailbox_command.
[35743.831252] qla2xxx [0000:42:00.0]-183a:7: Done qla2x00_issue_iocb_timeout.
[35743.831253] qla2xxx [0000:42:00.0]-1891:7: Done qla24xx_abort_command.
[35743.831254] qla2xxx [0000:42:00.0]-8804:7: Abort command mbx success cmd=ffff88015a8e1500.
[35743.831257] qla2xxx [0000:42:00.0]-801c:7: Abort command issued nexus=7:0:2 --  1 2002.
[35743.831259] qla2xxx [0000:42:00.0]-8802:7: Aborting from RISC nexus=7:0:2 sp=ffff880156ae6c80 cmd=ffff88016d8fd100 handle=388
[35743.831261] qla2xxx [0000:42:00.0]-188c:7: Entered qla24xx_abort_command.
[35743.831263] qla2xxx [0000:42:00.0]-1838:7: Entered qla2x00_issue_iocb_timeout.
[35743.831264] qla2xxx [0000:42:00.0]-1800:7: Entered qla2x00_mailbox_command.
[35743.831265] qla2xxx [0000:42:00.0]-1806:7: Prepare to issue mbox cmd=0x54.
[35743.831266] qla2xxx [0000:42:00.0]-1911:7: Mailbox registers (OUT):
[35743.831268] qla2xxx [0000:42:00.0]-1912:7: mbox[0]<-0x0054
[35743.831269] qla2xxx [0000:42:00.0]-1912:7: mbox[1]<-0x0000
[35743.831270] qla2xxx [0000:42:00.0]-1912:7: mbox[2]<-0x715d
[35743.831272] qla2xxx [0000:42:00.0]-1912:7: mbox[3]<-0xd100
[35743.831273] qla2xxx [0000:42:00.0]-1912:7: mbox[6]<-0x0000
[35743.831274] qla2xxx [0000:42:00.0]-1912:7: mbox[7]<-0x0010
[35743.831276] qla2xxx [0000:42:00.0]-1917:7: I/O Address = ffffc9000006a0c0.
[35743.831277] qla2xxx [0000:42:00.0]-180f:7: Going to unlock irq & waiting for interrupts. jiffies=100872880.
[35743.831633] qla2xxx [0000:42:00.0]-3822:7: FCP command status: 0x5-0x0 (0x80000) nexus=7:0:2 portid=0c1400 oxid=0x862 cdb=00000000000000000000 len=0x0 rsp_info=0x0 resid=0x0 fw_resid=0x0 sp=ffff880156ae6c80 cp=ffff88016d8fd100.
[35743.831663] qla2xxx [0000:42:00.0]-1814:7: Cmd=54 completed.
[35743.831666] qla2xxx [0000:42:00.0]-1913:7: Mailbox registers (IN):
[35743.831667] qla2xxx [0000:42:00.0]-1914:7: mbox[0]->0x4000
[35743.831668] qla2xxx [0000:42:00.0]-1914:7: mbox[2]->0x715d
[35743.831670] qla2xxx [0000:42:00.0]-1821:7: Done qla2x00_mailbox_command.
[35743.831671] qla2xxx [0000:42:00.0]-183a:7: Done qla2x00_issue_iocb_timeout.
[35743.831673] qla2xxx [0000:42:00.0]-1891:7: Done qla24xx_abort_command.
[35743.831674] qla2xxx [0000:42:00.0]-8804:7: Abort command mbx success cmd=ffff88016d8fd100.
[35743.831678] qla2xxx [0000:42:00.0]-801c:7: Abort command issued nexus=7:0:2 --  1 2002.
[35743.831680] qla2xxx [0000:42:00.0]-8802:7: Aborting from RISC nexus=7:0:1 sp=ffff880156ae63c0 cmd=ffff88016d8fc500 handle=387
[35743.831682] qla2xxx [0000:42:00.0]-188c:7: Entered qla24xx_abort_command.
[35743.831684] qla2xxx [0000:42:00.0]-1838:7: Entered qla2x00_issue_iocb_timeout.
[35743.831686] qla2xxx [0000:42:00.0]-1800:7: Entered qla2x00_mailbox_command.
[35743.831687] qla2xxx [0000:42:00.0]-1806:7: Prepare to issue mbox cmd=0x54.
[35743.831688] qla2xxx [0000:42:00.0]-1911:7: Mailbox registers (OUT):
[35743.831689] qla2xxx [0000:42:00.0]-1912:7: mbox[0]<-0x0054
[35743.831691] qla2xxx [0000:42:00.0]-1912:7: mbox[1]<-0x0000
[35743.831692] qla2xxx [0000:42:00.0]-1912:7: mbox[2]<-0x715d
[35743.831694] qla2xxx [0000:42:00.0]-1912:7: mbox[3]<-0xd100
[35743.831695] qla2xxx [0000:42:00.0]-1912:7: mbox[6]<-0x0000
[35743.831696] qla2xxx [0000:42:00.0]-1912:7: mbox[7]<-0x0010
[35743.831698] qla2xxx [0000:42:00.0]-1917:7: I/O Address = ffffc9000006a0c0.
[35743.831699] qla2xxx [0000:42:00.0]-180f:7: Going to unlock irq & waiting for interrupts. jiffies=100872881.
[35743.832056] qla2xxx [0000:42:00.0]-3822:7: FCP command status: 0x5-0x0 (0x80000) nexus=7:0:1 portid=0c1400 oxid=0x861 cdb=00000000000000000000 len=0x0 rsp_info=0x0 resid=0x0 fw_resid=0x0 sp=ffff880156ae63c0 cp=ffff88016d8fc500.
[35743.832087] qla2xxx [0000:42:00.0]-1814:7: Cmd=54 completed.
[35743.832089] qla2xxx [0000:42:00.0]-1913:7: Mailbox registers (IN):
[35743.832091] qla2xxx [0000:42:00.0]-1914:7: mbox[0]->0x4000
[35743.832092] qla2xxx [0000:42:00.0]-1914:7: mbox[2]->0x715d
[35743.832094] qla2xxx [0000:42:00.0]-1821:7: Done qla2x00_mailbox_command.
[35743.832095] qla2xxx [0000:42:00.0]-183a:7: Done qla2x00_issue_iocb_timeout.
[35743.832097] qla2xxx [0000:42:00.0]-1891:7: Done qla24xx_abort_command.
[35743.832098] qla2xxx [0000:42:00.0]-8804:7: Abort command mbx success cmd=ffff88016d8fc500.
[35743.832101] qla2xxx [0000:42:00.0]-801c:7: Abort command issued nexus=7:0:1 --  1 2002.
[35744.505845] device-mapper: multipath: Failing path 8:16.
[35744.508716] device-mapper: multipath: Failing path 8:32.

[nd1]

Hello,

We are observing the same issues at present. We've had the entire MSA 2040 replaced by now and the issue persists.

Have you tried the latest firmware for your HBA? I'm guessing we're using the same HBA:

HP StoreFabric SN1000Q 16GB 2-port PCIe Fibre Channel Host Bus Adapter
(QW972A)

We have not seen these issues with other MSA and HBA combinations.

[matthias_]

We have different Controllers (qle2462, qle2562)   , but all Qlogic qla2xxx from dell.

I tried to upgrade and load the latest firmware (8.01.02) on the qla2562 card, but the result is the same.

The other cards are running a 7.x firmware

So no fix yet. Did You solved the Problem somehow?

[nd1]

Hello,

I'm afraid we have no fix yet either :(