rx2660 VOLTAGE FAULT in System Log by BMC

[Bo Midberg]

Several VOLTAGE FAULT has been logged in system log. At a few occations there are also VOLTAGE OK messages logged, but the next VOLTAGE FAULT is logged about 22 hrs later.

Cannot see any other error messages, except at two occations some months ago, but seem not to be related. Message in those cases has been BOOT NOT DETECTED and OS had stopped. In each case a hard reset restarted the server. In those cases the VFP have displayed similar codes:

System         Int Health     Ext Health     Power

OFF            Flash Amber    On Green       On Green

and

Flash Red      Flash Amber    On Green       On Green

Have not been able to detect the root of the VOLTAGE FAULT - Any idea?

[Bill Hassell]

If you look at the front of the server, the status lights will tell you what component is bad. 
Here's the service manual -- see page 26 for details.

http://h20628.www2.hp.com/km-ext/kmcsdirect/emr_na-c01163996-16.pdf

[Bo Midberg]

Have not observed any lamp turned on for any component, but I do not know if these are turned off when investigating the system log through the iLO.

Had seen a comment about the [system?] battery in another case regarding another type of server, but assume that in those cases the settings may be lost. As far as I can see in the manuals I've found, there is no status lamp or system log message regarding the battery.

Of course, it may be that the voltage from the battery during long periods is low. Not sure when the voltage fault message started to log. It was present 7 months ago just after the logs were purged last time and system rebooted (have no anotation on this), but I cannot remember that this message has been logged earlier.

Viewing the BOARD INFO of the Socket System Board, it is of a version 4642, which says that it was shipped without a battery clip.

[Bill Hassell]

The battery (used to retain firmware settings) doesn't have a status light.
Do you have manually choose a boot path or does it remember the boot paths?

When you use the MP (console) to examine the SL logs, this does indeed turn off the warn/err lights. 

When you look at the Event logs in the SL menu, do you type T (for text mode) to get more useful messages?
Do you have both power suplies installed? The CM menu hasa a power status (PS) command to show all fans and power.

[Bo Midberg]

The server remember the boot path and the boot configuration settings.

Yes, the server has double power supplies, one PS to UPS and the other to normal electric power.

No, I had not used the Text view mode in SL, was not aware of it. It givs:

Log Entry 239: 05 Jan 2018 07:32:02
Alert Level 3: Warning
Keyword: VOLTAGE_FAULT
Voltage Fault - Check all boards with this voltage.
Logged by: Baseboard Management Controller;
Sensor: Voltage - System Battery
Data1: Performance Lags
0x205A4F29F20210E0 FFFF0106D5020300

I suppose this means that the system battery does not provide the expected voltage.

The state off all power supplies is normal according to PS in CM.

[Bo Midberg · Accepted solution]

Replaced the System and iLO/MP batteries. Had first to make backup off system settings. The NVRAMBKP tool was a great aid for both saving and restoring the system settings. Just in case I did also document the settings on paper.

No such possibilities was available for the iLO/MP settings. Had to write it down or copy from a Web SSL connection to the iLO/MP. The problem with the Web SSL connection is that it requires very old combinations of web browser (MSIE 8) and JAVA (V6). Found a Windows 2008 server with MSIE 9 and JAVA 6 where it was possible to connect to the iLO/MP on the server where to replace the batteries.

After replacing the batteries the System settings was restored by the NVRAMBKP tool, but the iLO/MP settings had to be entered manually before connection worked as earlier.

As Network settings was auto negotiated, that was not a problem, but several settings had to be entered under Administration:

> Disable Telnet

> Generate New Key Pair for SSH (to activate ssh connections)

> Generate New Key Pair for Web SSL (after entering the certificate information)

> Enter the Advanced iLO/MP Licence Key

> Set password for all users

Afterwards I tested the batteries in a tester that only works for 1.5 volts batteries. One battery was totally empty and the other seemed fully loaded, but as the batteries are of 3 volts it will not be a confident test. Just for curiosity. Was just amazed that no bigger problem was observed with one battery totally exhausted.

[Bo Midberg]

No additional messages have been logged on voltage fault since the batteries was changed.

Before changing the batteries it is adviceable to record the iLO/MP settings as the NVRAMBKP tool do not save those. I found the following commands useful (in the MP):

cm

ls

uc

sa -nc

so -nc

lm -nc

dns -nc

q

Also, after you have generated a new ssh certificate, you will get these messages at connecting with ssh from any combination of host and user where you already had made connections to that iLO/MP from ssh:

WARNING: HOST IDENTIFICATION HAS CHANGED!

IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the host key has just been changed.

To get rid of that you must replace the certificate for each user receiving that message on that host:

$ delete [.ssh2.hostkeys]key_22_snmp.pub;

$ ssh "Admin@snmp"

where snmp is the dns name to the iLO/MP on that server