RX2660 BMC glitches, server won't start

[Johnny_Cage]

Recently, I turned on RX2660 server that was sitting offline for about 4 years and it didn't start: power button turns green and fans are spinning at maximum speed, but all other LEDs on front panel are off.

Replaced both batteries with fresh ones. Tried to disconnect power(with few minutes rest, with and without batteries). System board has DIAG2 LED blinking, which corresponds to "ETO fetch" status - what's on earth is that anyway?

Connected to MP via LAN (putty/telnet) and looks like all things related to BMC/IPMI are extremly unstable, so in 20-50 attempts, BMC-related commands succeed just once, and fail for the rest. For example, I had to execute "LOC -ON" command like 20 times before it finally turned UID LED on. Same for "PC -ON" command - it takes 20-30 attempts before server finally turns power on. And after issuing any command there is a delay of approx. 5 seconds before displaying the results.

I tried to issue RB (reset BMC) command, but since it returns immediately without returning success/failure I don't know if it really worked. So I executed it 50 times to be sure.

All logs are empty, except system event (3 irrelevant events) and iLO event (completely irrelevant entries). VFP says "Communication problem with BMC".

Below are the results of executing all relevant commands in MP. I'm providing 2 versions for those commands: one for when BMC fails (happens most of the time) and one for when BMC succeeds (lucky me).

Spoiler

MP:CM> DF
-> No FRU information available.

MP:CM> SS
The query of the System Processors' State failed.

MP:CM> SS
System Processor Status:
System Power is Off. No processor information is available.

MP:CM> sysrev
Current firmware revisions
MP FW : F.02.23
BMC FW : revision not available
EFI FW : revision not available
System FW : ROM A 03.01, ROM B 04.11, Boot ROM B

MP:CM> sysrev
Current firmware revisions
MP FW : F.02.23
BMC FW : 05.24
EFI FW : ROM A 06.20, ROM B 07.14
System FW : ROM A 03.01, ROM B 04.11, Boot ROM B
UCIO FW : 03.0b
PRS FW : 00.08 UpSeqRev: 02, DownSeqRev: 01

MP:CM> XD -i
-> I2C access test (get BMC Device ID record)
Confirm? (Y/[N]): y
-> Test result: FAIL
-> Command successful.

MP:CM> XD -i
-> I2C access test (get BMC Device ID record)
Confirm? (Y/[N]): y
-> Test result: FAIL
-> Command successful.

MP:CM> XD -i
-> I2C access test (get BMC Device ID record)
Confirm? (Y/[N]): y
Entire contents of BMC Device ID Record (in hex):
devId: 32
devRev: 81
majorFwRev: 05
minorFwRev: 24
ipmiVersion: 01
devSupport: 3f
mfgId: 00000b
productId: 0611
-> Test result: PASS
-> Command successful.

MP> SL -event
# Location|Alert| Encoded Field | Data Field | Keyword / Timestamp
-------------------------------------------------------------------------------
0 0x60800A7C00000010 0000000000000000 IPMI Type-00 Event
1 0 *3 0x6B000A0000E00020 0100000000000004 IPMI Type-E0 Event
2 0x6B000A7C00000040 0100000000000000 IPMI Type-00 Event

MP> VFP
Welcome to the Virtual Front Panel (VFP).
Use Ctrl-B to exit.
System state Activity # of logs since boot
--------------- -------- --------------------
Unknown Unknown 1

LEDs | LOCATOR | SYSTEM | INT. HEALTH | EXT. HEALTH | POWER
-------------------------------------------------------------------------------
| UNKNOWN | UNKNOWN | UNKNOWN | UNKNOWN | UNKNOWN
-----------------------------------------------------------------------------
Status | Communication problem with BMC
-----------------------------------------------------------------------------

[jsm6]

Hi Johnny,

You do not have a MP firmware package available for this server.

All you have is ISO/EFI/OS package.

https://support.hpe.com/hpesc/public/km/product/5102364/HPE-Integrity-rx2660-Servers?ismnp=0&l5oid=3346452#t=DriversandSoftware&sort=relevancy&layout=table&f:@kmswsoftwaretypekey=[swt8000193]

Looks to me like you would need a SYS BD replacement.

Thanks and Regards.

[Johnny_Cage]

I've bought another server of identical model. Seller said it was working and even provided a video of working unit before shipping. But upon arrival it exposed the very same problem - now I have two dead servers.

What seller did is removed the CPU, RAM and PCIe extension bracket. I've installed my CPU and RAM which are identical SKUs to those  which were present in a "working" unit. Could that be the problem? I tried removing and re-sitting components many times. Components are identical to those that were in the server before. What could possibly be wrong here?

[JenzaDavid]

I realizize this is an old post, but did you ever find a resolution? I have just taken some old Itanium rx2660s out of storgage, and have discovered the same issue you described. I am confident they were working when moved to storage two or three years ago.. Ahanging the batteries only let me save the LAN values. 

[Johnny_Cage]

Nope. Two dead servers in my posession. It might be firmware/hardware design bug. Like the one with HP SAS drivers where firmware irrecoverably damaged the drive after some period of time (some counter register overflow).