EVA Storage
cancel
Showing results for 
Search instead for 
Did you mean: 

EVA 6400 "I/O Module B Stuck in "Loading Firmware"

 
SOLVED
Go to solution
Highlighted
AlexBriner
Advisor

EVA 6400 "I/O Module B Stuck in "Loading Firmware"

Hello All,

First of all, thank you for taking your time reading this new topic,.

I have a problem with a newly replaced I/O Module B on Disk Enclosure 2, I have performed a replacement of a damaged Module I/O for the Storage EVA 6400 5 days ago more or less and I noticed that the Operational Status at thate time was "Loading Firmware", Today we can see that the same Status is is showing in "Operational State" as you can see in the image: 

Captura.PNG

 

We have just removed and installed the same module to see if the Operational Status changed but is still showing the same, the new I/O module has a Firmware Version 0105 when the rest of the modules are in Firmware version 0085, could this triggering the constant "Loading Firmware"  ? 

By experience we have performed before a I/O Module replacement for a EVA 4400 and it took 4 days to see if the Operational Status changed and it Did to "Good" the main difference it was that the Firmware version of the Spare Module was lower than the other Modules already installed on all Disk enclosures.

 

Thank you very much for any help :)

Regards,

Alexander Briner 

12 REPLIES 12
vikashk
Advisor

Re: EVA 6400 "I/O Module B Stuck in "Loading Firmware"

Dear Alexander,

Greetings!

I see that replacement IO module is stuck in FW sync state.

This should not take this much time and you already have reseated it.

Are there any failed/failing drive(s) in system or any loop issues?

Could you please share the controller event log and controller configuration dump log from CV?

For CV version 10.x:
https://support.hpe.com/hpsc/doc/public/display?docId=mmr_sf-EN_US000012256

For CV version 9.x:
https://support.hpe.com/hpsc/doc/public/display?docId=mmr_sf-EN_US000012606&docLocale=en_US

I will await update.

Regards,
Vikash



I am an HPE employee!
AlexBriner
Advisor

Re: EVA 6400 "I/O Module B Stuck in "Loading Firmware"

Hello Vikash,

Thank you for the help :),  You are right, there another Warning in the Controller A:

Sin título.png

 

I will resquest a visit to get the event log and controller configuration dump :)

Many thanks and regards,

Alexander Briner.

 

AlexBriner
Advisor

Re: EVA 6400 "I/O Module B Stuck in "Loading Firmware"

Hello Vikash,

I have attached the logs mentioned before for review,

https://drive.google.com/file/d/15dzhhCCIk9nKI8d014MAzw6x95gtUujB/view?usp=sharing

Thank you very much ! :)1

Best Regards,
Alexander Briner

 

 

AlexBriner
Advisor

Re: EVA 6400 "I/O Module B Stuck in "Loading Firmware"

Hello Vikash,

Today we have visited the customer to see the State of the module I/O, unfortunately is still in "Loading Firmware" there has been 1 week since I performed a reset of the module.

I have decided to replace the module I/O to see what happens, As soon as I have the spare part I will let you know about the Operational State.

Regards,

Alexander Briner

vikashk
Advisor

Re: EVA 6400 "I/O Module B Stuck in "Loading Firmware"

Dear Alexander,

Thank you for the log file!

My sincere apologies for reaching out late to you.

Analysis:

1. All the DGs and vdisks are online.
2. Both the controllers are online running inactive code - XCS 95340000.
3. The DGs statuses are:
DG 'DRFATA' in attention due to failed drives and loop issues on drives..
DG 'DRFC' nearing set occupancy threshold.

4. The loop 1B is disrupted causing single active path on all drives.
5. The encl 2 IO module is good and seems to be in hung state due to firmware mismatch.
6. The failed drives requiring replacement are:
E2 B2
E1 B6 > 300GB > seems to be replaced recently if array dates to Apr'18 and could be reseated for re-use.
E4 B4

E1 B10
E1 B11
E4 B10

Suggestions:

1. It would be good idea to have latest backup taken as pre-cautionary measure.
2. Esnure to keep occupancy below 92% for enough space to be available for EVA maintanenace activity and drive faults.

3. The IO module on encl 2 is good. The firmware sync is to maintain same level of firmware on both IO modules which is by design.

The activity seems to be hung due to loop issues inturn drives with single active path only.

We do not need to replace this IO module.

4. All the PSUs 2 (right looking at the ports/power cord) on all disk enclosures showing as 'Unsupported'. Could you share the LED status on these PSUs?

5. Regarding loop 1B disrupted, to avoid any confusion later, may I have all disk enclosures IO module health LEDs along with P1-P2 ports LEDs details?

**I see encl 6, IO module B disappeared. However, I would await below details to share relevant steps.

1A path: Left Controller DP1A (LED) > Encl-X P1 (LED), P2 (LED) > Encl-Y P1 (LED), P2 (LED) > ………Encl-Z P1 (LED), P2 (LED) > Right controller DP1A (LED).

1B path: Left Controller DP1B (LED) > Encl-X P1 (LED), P2 (LED) > Encl-Y P1 (LED), P2 (LED) > ………Encl-Z P1 (LED), P2 (LED) > Right controller DP1B (LED).

Left/Right looking at the controllers' ports.

Once loop is up and drives are active on both paths, the firmware sync should complete within couple of minutes.

6. Is array showing date in Apr'18?
7. The soft diagnostics error on controller might require reboot (once loop 1B is up). It is usual behaviour due to failed drives or loop issues on array.

***Please do not restart any controller at any point; I would suggest once all are up.

Thanks and regards,
Vikash



I am an HPE employee!
AlexBriner
Advisor

Re: EVA 6400 "I/O Module B Stuck in "Loading Firmware"

Hello Vikash,

Thank you very much for the analisys!! :)

I have arrenged a visit to the customer for tomorrow to get the information needed to proceed and I also will recomend to the customer to get a updated backup.

I will not replace the I/O Module.. as soon as I get the information I will post it in this thread.

Many thanks and regards :D,

Alexander Briner

AlexBriner
Advisor

Re: EVA 6400 "I/O Module B Stuck in "Loading Firmware"

Hello @vikashk,

We have visited the customer to see the Storage, Below the result:

1. It would be good idea to have latest backup taken as pre-cautionary measure.

R. They have an updated backup, today they will perform a new one.

2. Esnure to keep occupancy below 92% for enough space to be available for EVA maintanenace activity and drive faults.

R.- Currently is set up upto 90%.

4. All the PSUs 2 (right looking at the ports/power cord) on all disk enclosures showing as 'Unsupported'. Could you share the LED status on these PSUs?

R.- All the LED status from the PSUs  are in Green Colour

. Regarding loop 1B disrupted, to avoid any confusion later, may I have all disk enclosures IO module health LEDs along with P1-P2 ports LEDs details?

R.- All the LED status from behind the Storage are in Green Colour

6. Is array showing date in Apr'18?

R.- The arrays show 20 Dec 2018

I have sent to you via private messaje screenshots and pictures from the Command View and the storage from the front and rear of the Storage.

I would be good idea to replace the failed drives requiring replacement ? (even if they are marked with the Green Led Status? to see if the Loading Firmware status dessapears?

I really appreciate your help, thank you very much :)

Best regards,

Alexander Briner

 

 

vikashk
Advisor

Re: EVA 6400 "I/O Module B Stuck in "Loading Firmware"

Dear Alexander,

Merry Christmas!

There are two different logs shared earlier and one dated 20Dec.

Thank you for screenshots; I could see the same details from EVA6400 log file.

Earlier logs and analysis were for EVA4400:
Name: EVA4400_HOSMIL_ST2
WWN: 5001-4380-04C7-F080

Latest logs dated 20 Dec are for EVA6400:
Name: EVA6400_HOSMIL_ST1
WWN: 5000-1FE1-501C-E510

Analysis and suggestions are:

*****EVA 4400:
Name: EVA4400_HOSMIL_ST2
WWN: 5001-4380-04C7-F080

Loop 1B disrupted. Could be due to encl 6, IO module B which could be reseated and swapped. However, we will need loop diagram and P1-P2 LEDs status on 1B path to determine which one is culprit.

**The loop errors propagate along entire path so erraneous status could be reported anywhere or for good module.

1B path: Left Controller DP1B (LED) > Encl-X P1 (LED), P2 (LED) > Encl-Y P1 (LED), P2 (LED) > ………Encl-Z P1 (LED), P2 (LED) > Right controller DP1B (LED).

Following drives are failed requiring replacement:
E1 B10, E1 B11, E2 B2, E4 B4, E4 B10


*****EVA 6400:
Name: EVA6400_HOSMIL_ST1
WWN: 5000-1FE1-501C-E510

Both loops up.

Controller A (not sure top or bottom, could be checked from OCP for WWN), WWN 5001-4380-05de-884b has 'Soft Diagnostics error'. Hence, degraded condition.

2B IOM module still showing firmware sync. The IO modules have mismatched firmware.
name = modulea
firmware = 0085

name = moduleb
firmware = 0105

I would suggest:

For EVA6400: EVA6400_HOSMIL_ST1 / 5000-1FE1-501C-E510

1. Reseat the drive (within 20 sec) - encl 3, bay 8.

2. Wait for 10 minutes and check if FW sync is complete, dis-engage encl 2, IO module B 2-3 inches out in its slot, wait for 4-5 minutes and then reseat it.

3. Wait for 10 minutes and check the status. If you would like, you may swap the IO modules B between encl 1 & encl 2 (to check if status migrates).

4. Wait for some time and refresh CV to verify updated status, click on controller A & B to determine which one has WWN 5001-4380-05de-884b and still having soft diag error.

Reboot above controller (CV > Controller > click tab 'shut down' > reboot/restart ), wait for 10 minutes, refresh CV and check if degraded condition disappears.

**It would be good idea to perform this activity during low IOPs hrs so that surviving controller could take load efficiently.

If not, please reboot the controller again.

The soft diagnostics erorr appear if controller found some errors while booting up. Sometimes, we need to reboot controllers twice or thrice to get them POST (initial diagnostics) cleanly.

Once above is done, please share the latest logs (controller event log, controller configuration dump log and controller terination log).

Thanks again and best regards,
Vikash



I am an HPE employee!
AlexBriner
Advisor

Re: EVA 6400 "I/O Module B Stuck in "Loading Firmware"

Hello Vikashk ,

Merry Christmas for you too!

I would like to thank you for taking your time and for reviewing this topic in this special day.

I am terrible sorry about the EVA4400, It was not my intention to upload send it to you for review :(

I will arrange a visit to proceed with your instruciontons and also starting the activity in a low Iops hrs. I have one question about the procedure (just to to be sure 100%), when you mean to reset the drive encl 3, bay 8. , that means to Remove the drive a few inches and then re inserted it within 20 sec?

Thank you very much !!

Best Regards,
Alexander Briner