Nimble HF40 rebot every 1 minute

[George_Shashin]

Hi!

I have a Nimble that has been lying around in a warehouse for a long time. I want to use it for training colleagues and internal tasks.
I know its IP addresses and admin password.

Model: HF40
Version: 5.1.4.0-683149-opt

There are the following problems with it now:

1) as soon as I log in, exactly 1 minute passes (I measured it) and the controller reboots.
This happens in any case - whether I use the graphical interface or connect via the COM port
I also tried leaving only 1 controller - it does not help.

2) the login is always accepted not the first time. Through the console cable, you need to try to enter the correct password 3-4 times and only then can I log in

3) I tried to perform Sanitize Booth, and the system did not ask for any passwords. It wrote that it started to perform Sanitize on the controller. But after 24 hours nothing has changed.

Unfortunately, I have no way to contact Nimble support to create a case.
Tell me, how can I fix this nimble? The data on it is not important.

[buzzsubash]

Hello @George_Shashin 
Really tough to give a clear answer without seeing the logs/console on what is happening before reboot. Does both controllers reboot after every minute? 
You have mentioned that up on checking using one controller it still happens. Have you tried pulling out one controller and test using the other ?

Also, Were you able to see any message from console or UI?  Was there any alerts around NVDIMM ?

[support_s]

System recommended content:

1. PCIe Card Replacement - HPE Alletra 5000 HPE Nimble Storage AFxx, HFxx | Replace a PCIe Card (HPE Alletra 5000, Nimble Storage AFxx, HFxx)

2. Controller Upgrade - HPE Alletra 5000 HPE Nimble Storage AFxx, HFxx | Upgrade Controllers (Alletra 5010, Nimble Storage AF20, HF20)

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[giladzzz]

Hi

you need to assitance from support because what you describe is a hardware software problem

without support  you can not upgrade or bypass the problem.

Regards

[George_Shashin]

Hi! @buzzsubash 
Only the controller we are logged in to is rebooted. That is, the active controller. This behavior is observed in both slots for both controllers (including if the controllers are swapped with each other). No output is observed in the console before rebooting, the ability to enter anything is simply lost and the controller reboots.

>>Have you tried pulling out one controller and test using the other ?
Yes

>>Really tough to give a clear answer without seeing the logs
Just give me a set of commands that you would like to see and I will try to collect them.

>>Was there any alerts around NVDIMM ?
I thought about it. Can you tell me how to check it?
I will only have 1 minute for the test.

[George_Shashin]

Hi!  @giladzzz 
Unfortunately, I don't have access to support. But I do have access to spare parts, for example, I have the same nimble that can be taken apart for spare parts.

[George_Shashin]

@buzzsubash @giladzzz 
I forgot an important fact! At the moment, there are only 2 SSDs out of 6 in the nimble (someone pulled them out for their own needs)
Can this affect reboots?
Should I add 4 disks before continuing?


Nimble OS $ disk --list
------+--------------------+----+---------+-------+---------------+---------+-----
Slot # Serial # Type Disk Size Disk RAID Shelf Shelf
(GB) State Status Serial Loca-
-tion
------+--------------------+----+---------+-------+---------------+---------+-----
1.A 50026B7282DBBA2D SSD 480.10 in use N/A AF-205694 A.0
1.B 50026B7282DBBF29 SSD 480.10 in use N/A AF-205694 A.0
2.A N/A N/A N/A absent N/A AF-205694 A.0
3.A N/A N/A N/A absent N/A AF-205694 A.0
4 ZC22MMKG0000C913GS6D HDD 2000.40 in use okay AF-205694 A.0
5 ZC22MJKK0000C913NXNF HDD 2000.40 in use okay AF-205694 A.0
6 ZC22MJ4F0000C9141WQA HDD 2000.40 in use okay AF-205694 A.0
7 ZC22NAVC0000C913L2JW HDD 2000.40 in use okay AF-205694 A.0
8 ZC22MM340000C912F69A HDD 2000.40 in use okay AF-205694 A.0
9 ZC22MGRW0000C913NT1X HDD 2000.40 in use okay AF-205694 A.0
10 ZC22MJL40000C9141W8S HDD 2000.40 in use okay AF-205694 A.0
11 ZC22MJ3Y0000C913NS0F HDD 2000.40 in use okay AF-205694 A.0
12 ZC22MHRN0000C9141WEV HDD 2000.40 in use okay AF-205694 A.0
13 ZC22MGVR0000C913NRMH HDD 2000.40 in use okay AF-205694 A.0
14 ZC22MHZW0000C913NX3R HDD 2000.40 in use okay AF-205694 A.0
15 ZC22MHTV0000C913NWW6 HDD 2000.40 in use okay AF-205694 A.0
16 ZC22MJE60000C9141WGP HDD 2000.40 in use okay AF-205694 A.0
17 ZC22MJ1Z0000C913NWUD HDD 2000.40 in use okay AF-205694 A.0
18 ZC22MJ160000C913NXK2 HDD 2000.40 in use okay AF-205694 A.0
19 ZC22MGME0000C913NWMZ HDD 2000.40 in use okay AF-205694 A.0
20 ZC22MHRV0000C913NXK1 HDD 2000.40 in use okay AF-205694 A.0
21 ZC22MGPJ0000C913NV8M HDD 2000.40 in use okay AF-205694 A.0
22 ZC22MCXY0000C912N50N HDD 2000.40 in use okay AF-205694 A.0
23 ZC22MJH40000C913NUB1 HDD 2000.40 in use okay AF-205694 A.0
24 ZC22MGE40000C913NSYU HDD 2000.40 in use okay AF-205694 A.0

[giladzzz]

Hi

this a problem that I had with one of my customer below is support answer

"

Unfortunately, this Nimble HF20 Array AF-XXXXX  has encountered software defect AS-160377 related to certain NVDIMMs.

The NVDIMM firmware contains a configuration that will result in failing to ARM after a threshold of 5 years of total power-on time has been exceeded on a given NVDIMM. Once the threshold has been crossed, services will fail to start following a planned or unplanned reboot of an affected controller."

as you see this is a hardware problem that is solved in later software versions so you can try another controller if it does not have the same problem it should work but your goal should be to upgrade the Nimble version.

Regards

Give a KUDO if this helps

[George_Shashin]

I know about this bug. This is a completely different case.

[buzzsubash]

@George_Shashin  Yes, SSD imbalance could be one of the issue. It is a must to add the SSD, so that the RAID set becomes stable and DSD, the core DSD service responsible for handling I/O. 

Once it is done, please do check if controllers are rebooting.

And if it is still happening, we will have to check if it is due to NVDIMM bug AS-160377 mentioned by @giladzzz, then please do log a case with support team. Will have someone take a look at it.

This is a zero-day bug and we do assist in fixing it even if the array is out of support contract. Please note, support will be provided only if it is due to AS-160377.  No part replacements will be provided.

[George_Shashin]

Got it, thanks for the advice!
I'll add the SSD and come back with the results.

[support_s]

Hello,

Let us know if you were able to resolve the issue.

If you are satisfied with the answers then kindly click the "Accept As Solution" button for the most helpful response so that it is beneficial to all community members.

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

[buzzsubash]

Hello @George_Shashin 
Were you able to test the controllers after inserting SSDs? Are they still rebooting?

[George_Shashin]

Hi! @buzzsubash  
I couldn't find the original bundle with 480GB disks to fill the empty slots.
But I found this one: R0P05A HPE Nimble Storage HF40/60 Adaptive Array 17.28TB (3x3840GB and 3x1920GB) FIO Cache Bundle
I installed it.
Unfortunately, rebooting the nimble every 1 minute did not change.
It's strange, I noticed that despite both controllers are inserted, each does not see the other.
for example: A active B none
it seems like the controllers cannot unite into a cluster or something like that.
I collected the output of console commands from each controller. Including "alert --list" (only from B), maybe this data will be useful.

https://drive.google.com/file/d/1Wgzy9iOwi7A2xRQBf1ux2IXvlJIveNjk/view?usp=sharing

https://drive.google.com/file/d/1Qu1LnhWtjBvtMSR8PY6dny3rO-FsZxM4/view?usp=sharing

[buzzsubash]

Thanks for the logs,. 
Unfortunately, we need system logs that can be only accessed using root password. And is only accessibly by support team.

I am sorry, without taking a look at them, can;t suggest any workaround in this case. 

[George_Shashin]

Thank you! later i try to create support case for this nimble.