Nimble AF7000 fails to boot up after replacing both controllers and fan chassis

[guraymon]

For the last couple of days, we have been trying to restore our Nimble AF7000 after the main fan chassis failed. Our third party vendor has replaced the fan chassis and the 2 controllers which seemed to be damaged from overheating. When the controllers try to come up, they complain of the /scratch folder and File System being full.
Is there a way to access this file system without Root access to see what is happening? Has anyone run into this issue before and what did you need to do to resolve it?

Thanks.

[Satish04]

Hi guraymon,

Please follow the below procedure to diagnose the issue.

1. Nimble arrays have built-in diagnostic tools and logs accessible through the management interface.
Check if there are any system logs, health reports, or diagnostic information available that could shed light on the /scratch folder issue.

2. If possible, monitor the array's resource utilization, including storage usage, CPU, memory, etc.
This monitoring might reveal what is causing the /scratch folder to fill up and help identify any abnormal behavior.

3. If you can access some non-root directories, look for any unnecessary or temporary files that can be safely deleted to free up space in the /scratch folder. Be cautious not to delete essential system files.

4. Check if any configuration changes were made during the replacement of the fan chassis and controllers. A misconfiguration might contribute to the /scratch folder filling up.

5. Collaborate closely with the third-party vendor who performed the replacement. They might have encountered similar issues before and may have insights on how to resolve the problem.

6. Resolving the issue may require root access, which is typically restricted on storage arrays for security reasons.

If the third-party vendor cannot resolve the issue without root access, it is recommended to reach out to HPE Tech support or HPE Nimble support to resolve the issue.

Always ensure that you have proper backups of critical data before attempting any changes to the storage array.

Hope this helps.

Regards,
Satish

[guraymon]

None of these options are available as the Controllers never come up so we have no access to the arrays or the GUI to view anything.

[Mahesh202]

Hi guraymon

It is very unfortunate that the controllers are not coming up, and you don't have access to the arrays or the GUI, the situation may be more critical. In this case, it's important to follow these steps:

Contact HPE Nimble Support: Since the controllers are not coming up, you should immediately contact HPE Nimble Support for urgent assistance. They have the expertise to troubleshoot hardware and software issues and guide you through the recovery process.

Review your organization's emergency response plan for critical system failures. Ensure that you follow any prescribed procedures to mitigate risks and address issues effectively.

While waiting for support, verify that all hardware connections are correct and secure. Ensure power sources, network connections, and cables are appropriately connected.

Regards
Mahesh

[Sunitha_Mod]

Hello @guraymon,

Let us know if you were able to resolve the issue.

If you have no further query and you are satisfied with the answer then kindly mark the topic as Solved so that it is helpful for all community members.

[guraymon]

So we had to renew our contract with HPE to have them look at the issue. Turns out we ran into a software bug that was known and after HPE cleared out the NVRAM, the controllers came back up. We could not do this ourselves because we did not have root access to the File System.