MSA2312i vdisk with 2 LEFTOVR disks after upgrade

[Luca]

Greetings,

   after updating this old but still functional MSA2312i with 12 1TB SATA disks, once completing the 2nd controller update the unit failed 2 disks part of a RAID 10 vdisk (6) with LEFTOVR condition. The update was from firmware M113R11 to M114P01. Both controllers updated fine (partner update disabled, updated first one, waited for it to come back online then updated the other) but after controller B restarted a vdisk was first quarantined, then 2 disks were tagged as LEFTOVR after which the vdisk was dequarantined automatically. The first vdisk is a RAID 5 array and already had a failed disk before the upgrade.  As an additional note the automatic scrubbing was enabled and scrubbing was apparently ongoing as a vscrub command error was logged.

Chronologically speaking Controller A logged 3 disk down errors (error ID 8)  with "Unknown reason", 2 contemporarily and one after 13 seconds which then magically recovered without log entry(!). Shortly after the vscrub error (ID 207) and after 3 seconds the vdisk was quarantined (ID 172) by Controller A.  Controller B then after 8 seconds dequarantined the same vdisk (it was the owner and it has just came back up after it's firmware update). Controller B shortly after also logged a vscrub job failure (ID 207). At this point the MSA had 2 before perfectly working disks now in LEFTOVR mode (one per mirror couple in the RAID 10 set fortunately).

Deleted metadata to enable RCON; however one was automatically assigned to the first RAID 5 with failed disk array (ID 0-5, failed disk in 4, reassigned disk in ID 10 slot 11 so now it is our of position, will have to move it later).

          I don't know why this happened, it's not the first time I updated this MSA though I have had some surprises with other MSAs but when replacing failed disks - never a "false failure" like this after a firmware update. Luckily (for now) the LEFTOVR disks that were marked as such were part of seperate mirrors of the same RAID 10 volume leaving the possibility to reconstruct them. There was no I/O activity during this firmware update as it the device is used as backup target and no job was running. After Controller A update, all the disks were as before and device logged no errors which is why I proceeded in updating controller B. Apparently these problems appeared as soon as controller B came back online and requested the ownership of it's vdisk (the RAID 10 vdisk). The disks were all 1TB SATA (6x ST1000NM0011 rev SN02 in the RAID 10 and mixed ST1000NM0033/NM0011/340NS in the RAID 5 vdisk)

   I know the MSA2312i is pretty old, but I just wanted to leave a warning here that there may be a problem when upgrading to this latest firmware from M113. Maybe it is due to a combination of factors (Failed disk in the RAID 5 vdisk + a RAID 10 vdisk, disk firmwares/models) or maybe not, in any case, beware.

[Navaneetha1]

Hello Luca , 

I understand your concerns. 

I do see a lot of fixes related to quarantine issues in latest firmware.
However, till we reach the latest firmware we cant rule out such issues from occuring .  So the safest way would be to schedule a down time and then update the firmware. 

I am an HPE Employee