MSA 2050 disks issues

[Support111]

Hi,

We have a MSA 2050 with 8 failed hard drives, we have opened a ticket but unfortunately the assigned support takes a long time to respond and is not precise with her answers.

The support has told us that we should place the hard drives that we have as "Available" in Global Spares, so there are doubts because a few weeks ago in one of the enclosures we had an "Available" hard drive but another hard drive failed in that same enclosure and the disk that was "Available" is no longer "Available" so we assume that the "Dynamic spare" option is enabled and the hard disk that was previously "Available" replaced the failed one.

So, having "Available" hard drives with the same size and speed capacity, is it necessary to add them to Global Spares?

How can I confirm that the "Dynamic spare" option is enabled?

In case "Dynamic spare" option is enabled, can I change hard drives without worry? All hard drives in the computer are SAS and have the same speed and size.

We also have Controller A to IO Module A of Enclosure 2 degraded. The support says we need to replace this cable but hasn't told us if it can be done without shutting down the storage. 

https://ibb.co/PW4Y6cM Please see the image, the cable to change is the one highlighted in red. Is it necessary to turn off the storage?

Regards,

[rakesh_93]

Hello,

I am surprised 8 drives failed!. Was it over a period of time or all at once?

It is possible that the drives could have gone into a LEFTOVER state and not really failed since you also mentioned IO module/path issues to the enclosure.

The dynamic sparing option is enabled by default. So no need to manually assign replaced/AVAIL disks as spares.

You may verify the setting in GUI - Action > Advanced Settings > Disk

OR CLI - show advanced-settings

You can also run your log here for a health check report - msa.ext.hpe.com

Check if you are behind in disk firmware since you seem to have an unusually high number of failures.

Regards

Rakesh S

[Support111]

Thanks for your quick response @rakesh_93.

It was over a period of time. And we only have one hard drive "LEFTOVER".

If we have "Dynamic spare" enabled, can we replace the hard drives in "FAILED" state?

Can we change the SAS cable without turning off the storage?

Thanks, I'm going to analyze the log in msa.ext.hpe.com. I did not know that page.

[support_s]

System recommended content:

1. HPE MSA 2050 Storage - Troubleshooting

2. HPE MSA 2050 SAN Storage - Operations

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[Shawn_K]

@Support111 

It is hard to answer your question about replacing a failed cable while using the storage completely without knowing the error messages you have recieved. The diagram you provided has redundant pathing - which is good. However, is the IOM A of enclosure 2 showing a degraded amber LED or is the cable port showing an amber LED?

If the port is showing as amber, I would suggest that during a period of low IO, with no rebuild occurring, and no drive replacement, you try to unplug the cable from the enclousre 2 IOM and reinsert the connector. Sometimes they come lose due to being bumped or jossled. This might solve your issue.

If the IOM is amber you will likely need to reseat the IOM to see if that resolves the problem or replace the IOM. 

Running the logs through the MSA Health Check tool will help with this problem.

If you have the Dynamic Spares option enabled you can replace the hard drives at any time with similar drives of the same size and speed. Once the replacement is done you should check the drive firmware to confirm you are running the latest version. Replacing failed drives can be done at any time but it will cause a scan of the array's backend. I suggest not doing the drive replacement while working on the enclosure 2 IOM or that cable.

I work for Hewlett Packard Enterprise. The comments in this post are my own and do not represent an official reply from HPE. No warranty or guarantees of any kind are expressed in my reply.

Cheers,
Shawn

[rakesh_93]

Hello 

• It was over a period of time. And we only have one hard drive "LEFTOVER".

Ok. The reason I asked was if they were all in LEFTOVER status, the cause could be something else ( firmware related/ power outages /incorrect power ON sequence etc ) and all that may have been required was clearing the metadata if the disk-groups were still degraded and not faulty. If it was over a period of time, then it would be good to check the firmware and also leave additional disks in the enclosure as spares to prevent RAID failure.

• If we have "Dynamic spare" enabled, can we replace the hard drives in "FAILED" state?

Yes the failed drive can be replaced when you have dynamic sparing enabled. But it would be a better idea to fix the cabling issue you mentioned before kicking off multiple rebuilds.

• Can we change the SAS cable without turning off the storage?

I understand you/support already narrowed down issue to a cable. I checked the cabling diagram you attached and that cable can be replaced online, however it is important to verify that the 'B' path is functional and none of the drives are reporting any data path to disk is not redundant errors. 
If we are not sure, I'd recommend to check with engineer again or arrange a maintenance window.

Regards

Rakesh S

[Support111]

Thanks for your quick response @Shawn_K.

All LEDs are green, there are no amber LEDs.

With the report I just saw the following:

Please see the image: https://ibb.co/0ZBHpbv 

[Support111]

Thanks for the information @rakesh_93 !

[Shawn_K]

@Support111 

From the MSA Health Check report is shows the controllers need a firmware upgrade. Your enclosure IOMs are all on current version, while you do need a drive frimware upgrade. However, this is your firmware version and does not indicate or review cable issues.

What indication is there for the Controller A to IO Module of Enclosure 2 degraded? Is there an event or error message present in the logs?

I work for Hewlett Packard Enterprise. The comments in this post are my own and do not represent an official reply from HPE. No warranty or guarantees of any kind are expressed in my reply.

Cheers,
Shawn

[Support111]

Hi @Shawn_K, thanks again for you reply.

This is the information related to the cables: 

Enclosure 1, Controller A, Expander , Out Expansion Port   The enclosure expansion port is degraded; some, but not all, of the PHY lanes can support activity.   - Check that the SAS expansion cable for the indicated port is present, connected between the correct ports, and fully seated in its connectors with its latches locked. Replace any faulty cable. Check that none of the PHYs have been manually disabled. - If this does not correct the problem, check that the modules at the ends of the cable are fully seated in their slots with their latches locked and have good health. Replace any faulty controller module or expansion module.

  Enclosure 2, Expansion Module A, Expander , In Expansion Port   The enclosure expansion port is degraded; some, but not all, of the PHY lanes can support activity.   - Check that the SAS expansion cable for the indicated port is present, connected between the correct ports, and fully seated in its connectors with its latches locked. Replace any faulty cable. Check that none of the PHYs have been manually disabled. - If this does not correct the problem, check that the modules at the ends of the cable are fully seated in their slots with their latches locked and have good health. Replace any faulty controller module or expansion module.

As I mentioned before, all the "link" lights are green.

[Shawn_K]

@Support111 

During a period of low host IO activity, I would try reseating both SAS connectors of that cable. As I mentioned it is easy to bump those on the back of the array. Be sure there is no other rebuild, disk activity, or drive replacement occurring during this time. If that does not clear the errors, then replacement of the cable will be necessary.

I work for Hewlett Packard Enterprise. The comments in this post are my own and do not represent an official reply from HPE. No warranty or guarantees of any kind are expressed in my reply.

Cheers,
Shawn