msa1000 strangeness after upgrading to fw 4.48

[richard stovall]

The FE said that you need to contact him about obtaining the part. I am happy to give you his name, email, phone, etc., but I am reluctant to provide this information in the public forum. How would you like me to send it to you?

We did not make any changes last night. We decided to try and make it until this weekend when I've got more time to recover from a worst-case scenario.

Thanks,

RS

[richard stovall]

Update - 7/3/06

None of the HP-supplied parts had any effect. Individually, I replaced the EMU, backplane, MSA1000 controller with the old cache installed, and a new cache module in the old controller. That is, each part was tried by itself without combining more than one new part at a time.

In each case the array came up and displayed the same odd behavior.

Since the drive in bay 13 was known to be the one displaying:

Disk1255 1,255 1,13 72.8 GB 160 MB/s 0, 1, 2, 3

in the show disks command, I shut down the msa1000 and replaced it with a new drive. Well, things only got weirder. Instead of replacing the missing drive and rebuilding, the old disk was still being seen as inserted, but obviously with a failed status. The new physical disk showed up, but couldn't be added to the array since it was the '15th' drive. Examples of show disks and show unit below:
CLI> show disks
box,bay bus,ID Size Speed Units
Disk101 1,01 0,00 72.8 GB 160 MB/s 0, 1, 2, 3
Disk102 1,02 0,01 72.8 GB 160 MB/s 0, 1, 2, 3
Disk103 1,03 0,02 72.8 GB 160 MB/s 0, 1, 2, 3
Disk104 1,04 0,03 72.8 GB 160 MB/s 0, 1, 2, 3
Disk105 1,05 0,04 72.8 GB 160 MB/s 0, 1, 2, 3
Disk107 1,07 0,08 72.8 GB 160 MB/s 0, 1, 2, 3
Disk112 1,12 0,13 72.8 GB 160 MB/s 0, 1, 2, 3
Disk108 1,08 1,00 72.8 GB 160 MB/s 0, 1, 2, 3
Disk109 1,09 1,01 72.8 GB 160 MB/s 0, 1, 2, 3
Disk110 1,10 1,02 72.8 GB 160 MB/s 0, 1, 2, 3
Disk111 1,11 1,03 72.8 GB 160 MB/s 0, 1, 2, 3
Disk112 1,12 1,04 72.8 GB 160 MB/s 0, 1, 2, 3
Disk113 1,13 1,05 72.8 GB 160 MB/s none
Disk114 1,14 1,08 72.8 GB 160 MB/s 0, 1, 2, 3

CLI> show units

Unit 0:
In PDLA mode, Unit 0 is Lun 1; In VSA mode, Unit 0 is Lun 0.
Unit Identifier :
Device Identifier : 600805F3-000D4F00-A91B235C-AC1E0013
Cache Status : Enabled
Max Boot Partition: Disabled
Volume Status : VOLUME USING REGENERATE
Parity Init Status: Complete
14 Data Disk(s) used by lun 0:
Disk101: Box 1, Bay 01, (SCSI bus 0, SCSI id 0)
Disk102: Box 1, Bay 02, (SCSI bus 0, SCSI id 1)
Disk103: Box 1, Bay 03, (SCSI bus 0, SCSI id 2)
Disk104: Box 1, Bay 04, (SCSI bus 0, SCSI id 3)
Disk105: Box 1, Bay 05, (SCSI bus 0, SCSI id 4)
Disk112: Box 1, Bay 12, (SCSI bus 0, SCSI id 13)
Disk107: Box 1, Bay 07, (SCSI bus 0, SCSI id 8)
Disk108: Box 1, Bay 08, (SCSI bus 1, SCSI id 0)
Disk109: Box 1, Bay 09, (SCSI bus 1, SCSI id 1)
Disk110: Box 1, Bay 10, (SCSI bus 1, SCSI id 2)
Disk111: Box 1, Bay 11, (SCSI bus 1, SCSI id 3)
Disk112: Box 1, Bay 12, (SCSI bus 1, SCSI id 4)
Disk1255: Box 1, Bay 255, (SCSI bus 1, SCSI id 13) DRIVE FAILED!
Disk114: Box 1, Bay 14, (SCSI bus 1, SCSI id 8)
Spare Disk(s) used by lun 0:
No spare drive is designated.
Logical Volume Raid Level: DISTRIBUTED PARITY FAULT TOLERANCE (Raid 5)
stripe_size=16kB
Logical Volume Capacity : 498MB

Some very friendly HP support folks had me replace the new drive with the original and the array rebuilt itself. We still have two drives with failed indications, but the MSA1000 is performing just fine.

My thought now is that the problem is in one or more of the physical drives, but which one(s)?

[John Kufrovich]

What does ACU say about your system. Does it show the disk correctly?

[richard stovall]

Yes, but it appeared outside the array. The 'failed' drive (the one that was removed) was listed as still being part of the array.

[John Kufrovich]

So, ACU show DISK106 and DISK113. While the MSA cli, doesn't report those disk.

Does ACU report you LUN as failed?

[richard stovall]

>So, ACU show DISK106 and DISK113. While the MSA cli, doesn't report those disk.

>Does ACU report you LUN as failed?

Currently everything is back to the way it was. When I had a new drive in bay 13 I was able to see it both in ACU and from the MSA CLI. It showed up as disk 113. Disk 1255 did not appear in 'show disks' but it did appear in all the units, and in each unit it presented as failed (since it wasn't present).

The ACU showed the new disk as present, but not part of any unit. I can't remember if the units showed up as failed or degraded. They were functional, however, albeit with no protection.

[John Kufrovich · Accepted solution]

Richard, it appears everything is working fine now. What did you do to correct the problem.

[richard stovall]

Thank you for following up.

The resolution was to destroy the configuration and start from scratch. Interestingly, the two drives which were acting up refused (absolutely refused) to be seen as available to any newly configure arrays. I replaced them with spares and all is well again.

So in the end it was the drives after all...

Thanks again for all the help and counsel.

rs