rx7620 firmware update problems

[Robert_Jewell]

I have installed a new cell board in an rx7620 and naturally the firmware did not match the other one installed. However when I use either the DFW command or the FW utility I am having problems.

I use the DFW command to try to copy PDHC and SFW firmware from cell 1 to cell 0 and I get the following response:

Flash Current
Number Cabinet Name Partition Handle Firmware Version Comments
------ --------- -------- --------- -------- ------------------ --------
1 0 PDHC 0A 0 256 A.000.000 Current
2 0 PDHC 0B 0 264 A.000.000 Old
3 0 SFW 0A 0 320 0.000 Pri PA
4 0 SFW 0B 0 328 0.000 Sec PA
5 0 PDHC 1A 0 257 A.003.027 Current
6 0 PDHC 1B 0 265 A.003.027 Old
7 0 SFW 1A 0 321 22.002 Pri PA
8 0 SFW 1B 0 329 3.088 Sec IA

Note: You can only duplicate one firmware type at a time.


Enter the Entities to be updated (EX: 7,8) : 4
Enter the source entity for the SFW image: 8
Are you sure that you want to continue(Y/N): y

**** Updating device SFW 0B ****
Erasing Flash(es). This may take several minutes.
Flash(es) erased successfully.
Programming.
Flashes failed to program : status code 10.

DFW completed with errors.


So, I tried to use the FW command to update firmware over the MP LAN. The utility does a great job in first erasing the contents before reporting the following:

**** Firmware Updating PDHC 0 ****
Ftp Connection Successful
Erasing Flash(es). This may take several minutes.

Link layer timed out - Retry 1

Link layer timed out - Retry 2

Link layer timed out - Retry 3

Link layer failed retries - Aborting update.
Erase of flash failed
**** Failed to burn entity PDHC 0 ****


The connection to the FTP server is solid and confirmed the MP LAN settings and they are at 10MB Half. I followed the instructions on the release 4.1 PDF instructions. Before updating the firmware I shut down the 48V to the cabinet.

Something else to note is that I did succesfully update the MP using the FW utility and I was able to update the PDHC FPGA using the OSP utility (which both use the FTP method).

Below is a recent SYSREV:

[mp-rx7620] MP:CM> sysrev

Cabinet firmware revision report

PROGRAMMABLE HARDWARE :

System Backplane : GPM FM OSP
------- ------- -------
1.002 1.002 1.002

PCI-X Backplane : LPM HS
------- -------
2.000 1.000

Core IO : Master Slave
-------- -------
2.010 2.010

LPM PDHC
------- -------
Cell 0 : 0.000 0.000
Cell 1 : 1.002 1.010


FIRMWARE:

Core IO
Master : A.007.008
Event Dict. : 1.015
Slave : A.007.008
Event Dict. : 1.015

Cell 0
PDHC : A.000.000
Pri SFW : 0.000
Sec SFW : 0.000

Cell 1
PDHC : A.003.027
Pri SFW : 22.002 (PA)
Sec SFW : 3.088 (IA)


The current release installed is 4.10. The release that was on the cell board was 4.00 I am pretty sure. We used it from another server. Right now I am afraid to cycle the AC line cords but fear that the PROM's are toast at this point anyway. Before moving forward I would like to see if anyone else has experienced these types of problems.

Thanks and regards,
Bob

[Sameer_Nirmal]

I would first doubt at proper sitting of cell 0. Check if the cell is installed correctly and properly latched and verify it through it's status LEDs.

Secondly I would doubt about missing steps like 2 and 6 ( cell at BIB ) as mentioned the firmware upgrade instructions for both DFW and FW updates.

Looking at the recent "sysrev" output, I think the server is safe to power cycle and then cell PDHC,SFW upgrade could be done with proper steps. However I would suggest to contact HP support before doing anything as a safe side.

Note that with FW PDHC 0 update, the flash erase is failed and with DFW SFW 0B update, cell SFW image( Sec PA ) is erased. Until firmware version 4.0 , MP Lan was fixed at 10Mb HD. But with 4.1, it could be 100MB FD.

[Stefan Stechemesser]

Hi Bob,

firmware up- and downgrade on this server type is a complicated procedure and the release notes for the specific firmware(s) have the be followed exactly.
The problem is that the server has not only the PDHC and PDC Firmware. There is also firmware on the MP card and firmware on the "field programmable gate arrays" (FPGAs)
-PDHC FPGA (Cell board)
-LPM FPGA (Cell Local Power Monitor)
-GPM FPGA (Global Power Monitor)
-Core I/O FPGA (MP board)
-some more that are not that important.

All firmware versions must match, otherwise strange things may happen like in you case.
The worst case could be a damaged cell board.
The correct procedure would have been:

Install New/Replacement cell board
==================================
DO NOT POWER ON CELL BOARD! (MP:CM> PE)
1.) MP:CM> OSP upgrade PDHC to 1.010
2.) MP:CM> DFW or FW to upgrade SFW and PDH
-If DFW is used:
Using a source cell that has the 4.1 SFW and PDH
-if IA 4.1 SFW to 3.088, PDHC to A.003.027

3.) Remove HSKP
-If more then one partition you can Pull Cell from Chassis or
If only one partition you can pull AC cords
4.) Restore power to enable upgraded Firmware
-Insert Cell board or Plug AC cords
5.) MP:CM> sysrev Verify Cell Firmware matches rest of complex


You only tried to upgrade the Cell PDC and PDHC firmware, but you did not try to update also the PDHC FPGA firmware.
Could you initially see the firmware version of the new cellboard in sysrev ?
What you can try is to reflash the PDHC FPGA firmware on that cellboard to 1.010
with the "pdhc_1.10.0" file from the 4.1 or 4.2 bundle with the "osp" command on the MP (cell should be turned off during upgrade). This should fix the situation.
If not: Call HP support (always a good idea).

best regards

Stefan

[Robert_Jewell]

Sorry for the delay. That was due to some vacation time and other things.

Anyway, thank you for the replies. I thought I had made some progress on this. I was able to get the cell board's firmware recognized by removing it, bringing up the partition on cell 1 and then inserting it back into the chassis. I was able to see the firmware and noted the mismatch. At that point I was able to use the DFW and FW commands properly. I got the boards the same and all was good.

For a little while anyway. I have then wanted to downrev to 4.0. This is a dev box and I wanted to simulate upgrading the firmware from 4.0 to 4.2. The "downrev" seemed to go smooth and all updates applied looking normal. I followed the 4.0 instructions and updated the FPGA's first and then the FW images. After all was done I reset the box and the cell baords never powered back on.

Checking the PS output I get the following on the cells:

HW status for Cell 1 : FAILURE DETECTED

Power status : NOT VIABLE, no fault
Boot is blocked
PDH memory is not shared
Processor Compatibility : FAULT
RIO cable status : UNDEFINED
RIO cable connection physical location : cannot be determined
Core cell is INVALID
Attention Led is off

PDHC status Leds : ----

CPU Module Slot 0 1 2 3
Populated
Local 48V Good
Power Enabled
Power Good

(* - True, P - Processor, T - Terminator)



DIMMs populated:
0 . . . 4 . . . 8 . . .12 . . .


1 1 1 1 1 1
VRM's 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
Present :
Enabled :
Pwr Good :


Front Side Bus Freq. : 250 MHz
CPU Core Freq. : 0 MHz
CPU Part Number :

System Boot Rom (SFW) firmware rev 0.000
PDH controller (PDHC) firmware rev 0.000, built THU JAN 01 00:00:00 1970
MICE revision is 0.0

And when trying to power it on:


Cell board power disabled - incompatible processors/cell.

Error attempting to power ON : RtnCodeFailure


Now, I know for a fact that the processors are correct. I have an mx2 1.1ghx processor and a Terminator installed in slots 0 and 2. Plus, note that these were working prior to the firmware downgrade attempt.

I have another cell board I can install to see if I can get the board seen as before, but this is getting really weird.

-Bob

[Stefan Stechemesser]

Hi Bob,

Is it possible that you did not fully downgrade to version 4.0 ?
It is not supported to run a firmware mix of 4.0 cells and 4.1 cells/MP on a rx7620 and some firmware mixes are known to cause errors.
For example: If you have MP Core I/O FPGA version 2.010 (you have it acc. to your sysrev) then the PDHC FPGA version MUST be 1.010, otherwise you have exactly the symptom described by you.
To fix this situation, you should first update the PDHC FPGA to 1.010 with the osp command, then powercycle the cell (unplugging of cellboard or disconnect power cables) to make the PDHC version active.
After that you can continue with the rest of the upgrade with DFW or fw (cellboard 0 should be powered off with pe during firmware update).

good luck

Stefan

[Robert_Jewell]

Thanks Stefan,
That does bring up a point. When I had completed the downgrade and reset the system, I had realized that I may not have completely updated the firmware images since there are two locations for the PDHC and SFW on each of the cell boards. If I understand it right, the FW update should update the oldest rev between the two images. The system should boot up on the most recent rev. One of the cells I know had a secondary rev for the PA-RISC processors. Could it be it is confused on what it is supposed to use?

I checked the FPGA revs and they do match up. I tried reflashing them anyway, but with no changes.

Is there anything I can do to the PDH Riser board to clear out these images to a default or anything similar. I doubt it, but it seems that nothing I do can get these boards to power on. I have also noted that the LED cluster on the center of the cell can tell me if the board comes on line or not. If the board comes up the FAB LED lights along with the flashing HEARTBEAT. If the board is "dead" only the HEARTBEAT LED flashes.

-Bob

[Stefan Stechemesser]

Hi Bob,

I'm sorry, but if reflashing the PDHC (with powercycling after that by unplugging the cables) did not help, then your cellboard may be defect.
Call the HP support and let them analyze the problem.
There is no way to reset the firmware flashes to a "default". You could only clear the cells NVRAM by unplugging the battery on the PDHC daughter card, but this will not affect any firmware revision.
Note that firmware up- and downgrads are possible when the cell is turned off ...

[Robert_Jewell]

Well in the end I was able to revive one of the cell boards. What I thought did the trick was to remove the processors from the board and then insert it into the chassis. When I did this the board was seen and I was able to re-flash the firmware. I re-installed the processors, powered on the server and all was good.

I tried this on the other cell and got nowhere. I still have doubts that the firmware was corrupted during the update as twice I have been able to get the board to come online.

I really do not have any final answers here. Perhaps going down to an older revision confused the MP's somehow and they see the cell boards as they should not and do not allow them to join the fabric.

At any rate, I have replaced that final cell board as I have spent way too much time on this anyway. Thanks for all the help.

Regards,
Bob