MSA 2040 SLOW after it crashed

[moboter]

My Storage Pool on the MSA ran out of space so i added more disks.

It synchronised the pools fine but now the connection speed through the fibre is extremly slow. Rebooting  the controllers and the MSA didnt help my current transfer speed is 50 MB max

Any one here who can help a storage newbie?

There are no errors appearing on the MSA Log

[ArunKKR]

Hi ,

I believe that the MSA controller firmware would be the latest version GL225P002.
I would suggest having a data back up before performing any actions on the array as its an end of support life product.
Execute the commands listed below from a putty SSH session to verify over all array health status and cache status:

show system
show cache-parameters

Write Back Status would usually list status as Enabled

If its a switch connected configuration you could check the switch logs for any fibre channel path issues like faulty cable, SFP etc.

If its a direct connect configuration, it would be difficult to identify the cause without reviewing debug logs which requires logging an HPE support case.

If the host server operating system is windows, you could check for event id 153 which usually indicates fibre channel path issues.

Event ID 153

The IO operation at logical block address 123456 for Disk 2 was retried.

To rule out fibre channel path issues, you could shut down one storage controller at a time through SMU/SSH CLI for around 10 minutes and check whether it helps improve performance.

Check the MSA events through SMU for any repeated errors from any of the hard drives.
Ensure that the Pool has at least 200GB free space.

[ArunKKR]

Hello @moboter , 
 
Let us know if you were able to resolve the issue.
 
If you have no further query and you are satisfied with the answer then kindly mark the topic as Solved so that it is helpful for all community members.

[moboter]

Operation Mode: Aktiv-Aktiv-ULP

Controller A Cache Parameters
-----------------------------
Write Back Status: Aktiviert
CompactFlash Status: Installiert
CompactFlash Health: OK
Cache Flush: Aktiviert

Controller B Cache Parameters
-----------------------------
Write Back Status: Aktiviert
CompactFlash Status: Installiert
CompactFlash Health: OK
Cache Flush: Aktiviert

The client Log is full with 153 events

I ordered a new tranceiver and will aee if that helps

The only thing that worries me that 4 Tranceivers would have gone faulty at the same time

[ArunKKR]

Hi,

It would usually be a single SFP module that would be causing the issues.

It could be either on MSA controller ports or on the server FC HBA card.

If its a switch connected configuration switch logs will help identify the cause.

[moboter]

No switch is involved
I will Test with a new transceiver with only one Server Connected
Von meinem iPhone gesendet

[ArunKKR]

Hi, You could also test by shutting down one storage controller at a time and check whether host server is reporting event id 153 repeatedly. When one of the storage controllers is shut down the other controller will take over the load and handle traffic. This way you can isolate which controller path has the faulty SFP. I believe that only one server would report event id 153 if its a direct connect configuration. If multiple servers are reporting event id 153 it might not be an SFP issue or FC path issue.

[moboter]

Restarted MC and SC from CLI

The management controller retrieves data from the storage controller

a short while later

The Administrative Controller is not ready.

Thats what I get when I type show system from either controller

Any advice?

Cold reboot?

[ArunKKR]

Hi,  

You could try a graceful power cycle first.

Please check whether the below command works:

shutdown both

Allow 5-7 minutes for controllers to shut down

Remove power to controller enclosure 1 and then attached disk enclosures.

Provide power to disk enclosures, allow 3-5 minutes for all disks to spin up and then provide power to controller enclosure 1.

If the shutdown command doesnt work the only option would be to directly remove power to controller enclosure 1 and follow rest of the steps. 

Please have a back-up of the data in MSA volumes (if they are still accessible by host servers) before trying the suggested steps.