Got a failure trying to install 11i v2 on a rx2600 server

[Steven E. Protter]

Certied for the OS.

Got the same error trying to restore its Ignite make_net_recovery image from 11.22.

We'd like some help to know what to investigate. We suspect hardware but can't figure out anything more specific than that.

************* SYSTEM ALERT ************** SYSTEM NAME: simba

04 May 2006 14:34:14
Alert Level 7: Fatal
Keyword: Type-02 137001 1273857
Machine Check initiated
Logged by: SFW-redundant; Sensor: Critical Interrupt
Data2: OEM Code2: 0x00
0xC1445A10E6024000 003FA17000130300

A: ack read of this entry - X: Disable all future alert messages Anything else skip redisplay the log entry
->Choice:Timeout!
*****************************************

************* SYSTEM ALERT ************** SYSTEM NAME: simba

04 May 2006 14:34:14
Alert Level 7: Fatal
Keyword: MC_INITIATED
Machine Check initiated
Logged by: System Firmware 0
Data: Implementation dependent data field
0xF680009800E05AA5 0000000000000000

A: ack read of this entry - X: Disable all future alert messages Anything else skip redisplay the log entry
->Choice:Timeout!
*****************************************

************* SYSTEM ALERT ************** SYSTEM NAME: simba

04 May 2006 14:34:14
Alert Level 3: Warning
Keyword: BOOT_SEL_FULL
IPMI System Event Log (SEL) is full
Logged by: System Firmware 0
Data: Data field unused
0x6080005700E05AA5 0000000000000000

A: ack read of this entry - X: Disable all future alert messages Anything else skip redisplay the log entry
->Choice:Timeout!
*****************************************

************* SYSTEM ALERT ************** SYSTEM NAME: simba


Alert Level 3: Warning
Keyword: HP-UX_DUMP_ERROR
OS dump, error (EFxx)
Logged by: HP-UX Kernel 0
Data: Legacy 20-bit PA HEX chassis code
0x3F00033900E00000 00000000000AEFFF

A: ack read of this entry - X: Disable all future alert messages Anything else skip redisplay the log entry
->Choice:Timeout!
*****************************************

********** VIRTUAL FRONT PANEL ********** System Boot detected
*****************************************
LEDs: SYSTEM POWER
FLASH YELLOW ON
LED State: Running non-OS code. Non-critical error detected.
Check Events and Console Logs for error messages.


************* SYSTEM ALERT ************** SYSTEM NAME: simba

04 May 2006 14:54:12
Alert Level 7: Fatal
Keyword: Type-02 137001 1273857
Machine Check initiated
Logged by: SFW-redundant; Sensor: Critical Interrupt
Data2: OEM Code2: 0x00
0xC1445A1594024000 003FA17000130300

A: ack read of this entry - X: Disable all future alert messages Anything else skip redisplay the log entry
->Choice:Timeout!
*****************************************

************* SYSTEM ALERT ************** SYSTEM NAME: simba

04 May 2006 14:54:12
Alert Level 7: Fatal
Keyword: MC_INITIATED
Machine Check initiated
Logged by: System Firmware 0
Data: Implementation dependent data field
0xF680009800E05AA5 0000000000000000

A: ack read of this entry - X: Disable all future alert messages Anything else skip redisplay the log entry
->Choice:
*****************************************

************* SYSTEM ALERT ************** SYSTEM NAME: simba

04 May 2006 14:54:12
Alert Level 7: Fatal
Keyword: MC_INITIATED
Machine Check initiated
Logged by: System Firmware 0
Data: Implementation dependent data field
0xF680009800E05AA5 0000000000000000

A: ack read of this entry - X: Disable all future alert messages Anything else skip redisplay the log entry
->Choice:A
*****************************************

************* SYSTEM ALERT ************** SYSTEM NAME: simba


Alert Level 3: Warning
Keyword: HP-UX_DUMP_ERROR
OS dump, error (EFxx)
Logged by: HP-UX Kernel 0
Data: Legacy 20-bit PA HEX chassis code
0x3F00033900E00000 00000000000AEFFF

A: ack read of this entry - X: Disable all future alert messages Anything else skip redisplay the log entry
->Choice:
*****************************************

************* SYSTEM ALERT ************** SYSTEM NAME: simba


Alert Level 3: Warning
Keyword: HP-UX_DUMP_ERROR
OS dump, error (EFxx)
Logged by: HP-UX Kernel 0
Data: Legacy 20-bit PA HEX chassis code
0x3F00033900E00000 00000000000AEFFF

A: ack read of this entry - X: Disable all future alert messages Anything else skip redisplay the log entry
->Choice:A

SEP

[Enrico P.]

Hi,
is your led attention blinking?
try to read the error log from MP and reistall

Enrico

[AIMC]

Hi SEP,

The logs are not giving enough info, could you stop the server at EFI & run


Shell> errdump mca 'filename'

I can decode that a bit better, the only problem is getting the errdump off the box.. usb key is usually the best if EFI recognises the file system on it.

Aidan

[Chan 007]

Steven,

I assume this is a H/W problem. I just said based on the pdf on on my experience. So I may be wrong.

Download this pdf from the following link

www.digital.co.hu/techinfo/ Documents/rx4610_tech_wp_06_2001.pdf

Chan

[Steven E. Protter]

Yes, we assume hardware as well and are looking for clues.

We shut down the system power so there are currently no lights lit.

SEP

[Eric Antunes]

Hi SEP,

It must be a hardware issue, like in this following thread:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=701308

Best Regards,

Eric Antunes

[TwoProc]

SEP, without a decode of the chassis code that you posted above - I think you're down to stripping down the computer to its bare essentials and seeing if it comes up. One cell board, minimal memory would be one way, but first - I'd try to see if an I/O card is holding me up (mainly because this is just so easy to rule out compared to the former). So, I'd pull all I/O cards except what I need to get to the tape drive and see if Ignite will come up. If not, then put all the cards back and then start pulling memory, if that doesn't work, then start pulling cell boards. Not many fun options though!

Good Luck

[Steven E. Protter]

I plan to do the decode this week if my management feels its worth the time.

There are other folks that can do this and I'll distribute the link to them in case they have time.

We start our workweek Sunday morning.

SEP

[Darrel Louis]

Steven,

I assume you've checked your firmware level.
Is this a clean install or was there already a 11iv2 version installed on the server?

Darrel

[Steven E. Protter]

Not a clean install.
11.22 was on it. This also refused to install from Ignite server.

SEP

[Steven E. Protter]

Look mom, we spent almost a full man day and got this:

Cache Identification:
Cache Level Instruction(KBytes) Data(KBytes)
0 16 16
1 N/A 256
2 N/A 1536
Configuring I/O...
Please wait.

Looking for SCSI devices via LSI SPT device driver.....
** 238: ERROR: pc_ec_7 = 1
7 0 0x00006B 0x0000000000000018 unexpected trap
** 238: ERROR: pc_ec_7 = 1
7 0 0x000066 0x0000000000000018 trap taken, number in ext PE
** 238: ERROR: pc_ec_7 = 1
7 0 0x00003C 0x0000000000005400 trap taken, offset in ext PE

Unexpected TRAP...no handler installed...!

Platform Event = 0x00000000000000cd
Vector Number = 0x0000000000000018
IVT offset = 0x0000000000005400
State Save Address = 0x80000000ff44a068
iip = 0x000000003fcb67e0
ipsr = 0x0000100000006038
isr = 0x0000000400000030
ifa = 0x000000000000000f
itir = 0x0000000000000040
iipa = 0x000000003fcb67d0
ifs = 0x800000000000050f
iim = 0x000000000008401f
iha = 0x0000000000000000

************* SYSTEM ALERT **************
SYSTEM NAME: simba

05 Jul 2006 14:50:17
Alert Level 7: Fatal
Keyword: Type-02 0f7000 1011712
System firmware error
Logged by: SFW-redundant; Sensor: System Firmware Progress
Data2: PRV State: 0x3f
0xC144ABD1A90202A0 FF3F4070000F0300

A: ack read of this entry - X: Disable all future alert messages
Anything else skip redisplay the log entry
->Choice:


We attempted to skip the messy diagnostics and build a bootable CD to update the firmware but that hung right after the HPOFM prompt.

We can boot the system off a recovery shell and even figured out how to get the new firmware on it via tftp.

Question: Where is the EFI partition and how to we copy files to it.

efi_cp is a utility that comes with HP-UX 11.23 IA64.

I don't understand this examnple.


Copy bootloader into /EFI/HPUX/ of the EFI file system on the device
specified in the EFI_PARTITION environment variable and rename it to
hpux.efi:

efi_cp bootloader /EFI/HPUX/hpux.efi


If you have no disk to boot from, where is the EFI partition?

SEP

[Torsten. · Accepted solution]

Steven,

the EFI partition is on the disk only.

Please post your current version of firmware

MP:CM>sysrev

Depending on your current version, you need perhaps to another version prior to go to the newest. Also it is important to follow the instructions (I'm sure you did ;-))

Last time I updated an small Integrity server I noticed that I have to clear the logs (SL -> c) to make the update work.

[Torsten.]

From the release notes:

1. If the currently installed MP firmware is older than E.02.10 then updating to MP firmware revision E.02.26 requires updating to MP E.02.10 first.

2. Upgrades to MP firmware revision E.02.29 will require updating to MP firmware revision E.02.26 first.

3. MP firmware must be at or upgraded to revision E.02.29 prior to upgrading to iLO MP revision E.03.13

In fact, a upgrade from a very old version directly to the newest will fail!

[Steven E. Protter]

OF course you catch me after I've gone home.

I'll post in the morning Israel time.

Stand by.

SEP

[Steven E. Protter]

Bingo!

MP Host Name: simba
MP> sysrev


SR

Firmware revision report

MP FW : E.02.07
BMC FW : 01.22
EFI FW : 01.22
System FW : 01.41


MP Host Name: simba


Guess what. HP Support didn't tell us this.

Any idea wher I can get the E.02.10 MP firmware?

SEP

[Torsten.]

Hi,

all versions are here:

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareIndex.jsp?lang=en&cc=us&prodNameId=88839&prodTypeId=15351&prodSeriesId=447335&swLang=13&taskId=135&swEnvOID=54

The very old versions are not on an iso image, so you have to use the FTP method.

[Steven E. Protter]

Torsten, you are going to have a big day on points today.

I am not familiar with the ftp method of updating firmware. I can't find a doc. I'm betting you have one.

SEP

[Steven E. Protter]

Found the procedure.

Cancel last request.

Points today.

SEP

[Torsten.]

Many possibilities - many problems.

You can make this update in several ways:

1. FTP
2. with the ISO file burned to a CD
3. OS patch
4. copy the files from the OS to the EFI partition
5. burn only the files to a CD-ROM run run them from EFI prompt

(3 to 5 are very similar)

Anyway.


Be aware, in the 2.10 bundle is an "update.only" bundle. This is the first you need to install. After this install the version 2.10.

I can't understand why all the needed bundles are NOT provided on the same CD image ;-(


To prevent problems, I would do a complete power cycle (unplug power) after every update and clear the system logs.

After you did the dozens of updates today, I will ask you if I see problems someday, because I know a very experienced person than.

[Steven E. Protter]

I think this will work.

WE have very experienced people here. My boss is at a meeting and wants to see the fun. I expect to get to this in 4-6 hours.

SEP

[Steven E. Protter]

Torten pointed the way.

Final solution:

Manual upgrade to E.02.10
Manual upgrade to E.02.26
Manual upgrade to E.02.29
Manula upgrade to E.03.15

Then a efi based upgrade took care of everything else and we were able to isntall 11i v2.

Wow.

Could that have been any harder?

Don't answer that.

SEP