A500, GSP error logs after installing Quality Pack HP-UX 11.00

[Marcel Preda]

Hi, this is my first post here.
A feew days ago we have patched 3 HP-UX 11.00 servers (two a500 server and one L1000), we have installed Quality Pack - March 2004.

Every thing was going fine.
But after one day one machine was crashed:
no port responding, no ping ... (machine is in other location, more that 1000-thousand- kms ).

Finnally someone had done a hard restart of the machnie, the boot was fine, now the machine is on line.

But... we connect on GSP and read the error/alert logs.

I have no idea waht could happend.
Looks like could be a I/O (disk) problem.
Also I suspect that could be a power supply problem, because the alerts are divers , in a very short period of time ( 15 errors in about 3 minutes).

Maybe someone more experienced can give me some idea about what might have happened.
(could be a firmware patch problem?! )

Thanks,
Marcel


Here comes the GSP logs:


Log Entry # 0 :
SYSTEM NAME: a500ap2-lc
DATE: 04/23/2004 TIME: 23:29:55
ALERT LEVEL: 2 = Non-Urgent operator attention required

SOURCE: 8 = I/O
SOURCE DETAIL: 2 = system bus adapter SOURCE ID: 6
PROBLEM DETAIL: 3 = function failure

CALLER ACTIVITY: 6 = machine check STATUS: 3
CALLER SUBACTIVITY: 34 = implementation dependent
REPORTING ENTITY TYPE: 0 = system firmware REPORTING ENTITY ID: 01

0x7000102382066343 00000000 00070200 type 14 = Problem Detail
0x5800182382066343 00006803 17171D37 type 11 = Timestamp 04/23/2004 23:29:55
Type CR for next entry, Q CR to quit.



Log Entry # 1 :
SYSTEM NAME: a500ap2-lc
DATE: 04/23/2004 TIME: 23:29:55
ALERT LEVEL: 2 = Non-Urgent operator attention required

SOURCE: 8 = I/O
SOURCE DETAIL: 2 = system bus adapter SOURCE ID: 4
PROBLEM DETAIL: 3 = function failure

CALLER ACTIVITY: 6 = machine check STATUS: 3
CALLER SUBACTIVITY: 34 = implementation dependent
REPORTING ENTITY TYPE: 0 = system firmware REPORTING ENTITY ID: 01

0x7000102382046343 00000000 00070200 type 14 = Problem Detail
0x5800182382046343 00006803 17171D37 type 11 = Timestamp 04/23/2004 23:29:55
Type CR for next entry, - CR for previous entry, Q CR to quit.



Log Entry # 2 :
SYSTEM NAME: a500ap2-lc
DATE: 04/23/2004 TIME: 23:29:55
ALERT LEVEL: 2 = Non-Urgent operator attention required

SOURCE: 8 = I/O
SOURCE DETAIL: 2 = system bus adapter SOURCE ID: 2
PROBLEM DETAIL: 3 = function failure

CALLER ACTIVITY: 6 = machine check STATUS: 3
CALLER SUBACTIVITY: 34 = implementation dependent
REPORTING ENTITY TYPE: 0 = system firmware REPORTING ENTITY ID: 01

0x7000102382026343 00000000 00070200 type 14 = Problem Detail
0x5800182382026343 00006803 17171D37 type 11 = Timestamp 04/23/2004 23:29:55
Type CR for next entry, - CR for previous entry, Q CR to quit.



Log Entry # 3 :
SYSTEM NAME: a500ap2-lc
DATE: 04/23/2004 TIME: 23:29:55
ALERT LEVEL: 2 = Non-Urgent operator attention required

SOURCE: 8 = I/O
SOURCE DETAIL: 2 = system bus adapter SOURCE ID: 0
PROBLEM DETAIL: 3 = function failure

CALLER ACTIVITY: 6 = machine check STATUS: 3
CALLER SUBACTIVITY: 34 = implementation dependent
REPORTING ENTITY TYPE: 0 = system firmware REPORTING ENTITY ID: 01

0x7000102382006343 00000000 00070200 type 14 = Problem Detail
0x5800182382006343 00006803 17171D37 type 11 = Timestamp 04/23/2004 23:29:55
Type CR for next entry, - CR for previous entry, Q CR to quit.

Log Entry # 4 :
SYSTEM NAME: a500ap2-lc
DATE: 04/23/2004 TIME: 23:29:55
ALERT LEVEL: 2 = Non-Urgent operator attention required

SOURCE: 8 = I/O
SOURCE DETAIL: 3 = local bus adapter SOURCE ID: FF
PROBLEM DETAIL: 3 = function failure

CALLER ACTIVITY: 6 = machine check STATUS: 2
CALLER SUBACTIVITY: 11 = implementation dependent
REPORTING ENTITY TYPE: 0 = system firmware REPORTING ENTITY ID: 01

0x2000102383FF6112 000000FF FF03FF83 type 4 = Physical Location
0x5800182383FF6112 00006803 17171D37 type 11 = Timestamp 04/23/2004 23:29:55
Type CR for next entry, - CR for previous entry, Q CR to quit.



Log Entry # 5 :
SYSTEM NAME: a500ap2-lc
DATE: 04/23/2004 TIME: 23:29:55
ALERT LEVEL: 6 = Boot possible, pending failure - action required

SOURCE: 8 = I/O
SOURCE DETAIL: 3 = local bus adapter SOURCE ID: FF
PROBLEM DETAIL: 3 = function failure

CALLER ACTIVITY: 2 = config STATUS: 4
CALLER SUBACTIVITY: 11 = implementation dependent
REPORTING ENTITY TYPE: 0 = system firmware REPORTING ENTITY ID: 01

0x2000106383FF2114 000000FF FF03FF83 type 4 = Physical Location
0x5800186383FF2114 00006803 17171D37 type 11 = Timestamp 04/23/2004 23:29:55
Type CR for next entry, - CR for previous entry, Q CR to quit.



Log Entry # 6 :
SYSTEM NAME: a500ap2-lc
DATE: 04/23/2004 TIME: 23:29:55
ALERT LEVEL: 6 = Boot possible, pending failure - action required

SOURCE: 8 = I/O
SOURCE DETAIL: 3 = local bus adapter SOURCE ID: FF
PROBLEM DETAIL: 3 = function failure

CALLER ACTIVITY: 2 = config STATUS: 4
CALLER SUBACTIVITY: 10 = implementation dependent
REPORTING ENTITY TYPE: 0 = system firmware REPORTING ENTITY ID: 01

0x2000106383FF2104 000000FF FF03FF83 type 4 = Physical Location
0x5800186383FF2104 00006803 17171D37 type 11 = Timestamp 04/23/2004 23:29:55
Type CR for next entry, - CR for previous entry, Q CR to quit.




Log Entry # 7 :
SYSTEM NAME: a500ap2-lc
DATE: 04/23/2004 TIME: 23:27:46
ALERT LEVEL: 2 = Non-Urgent operator attention required

SOURCE: 0 = unknown, no source stated
SOURCE DETAIL: 0 = unknown, no source stated SOURCE ID: FF
PROBLEM DETAIL: 0 = no problem detail

CALLER ACTIVITY: 6 = machine check STATUS: 2
CALLER SUBACTIVITY: 46 = implementation dependent
REPORTING ENTITY TYPE: 0 = system firmware REPORTING ENTITY ID: 01

0x0000102000FF6462 00000000 00000000 type 0 = Data Field Unused
0x5800182000FF6462 00006803 17171B2E type 11 = Timestamp 04/23/2004 23:27:46
Type CR for next entry, - CR for previous entry, Q CR to quit.



Log Entry # 8 :
SYSTEM NAME: a500ap2-lc
DATE: 04/23/2004 TIME: 23:27:46
ALERT LEVEL: 2 = Non-Urgent operator attention required

SOURCE: 7 = memory
SOURCE DETAIL: 9 = unexpected SOURCE ID: 0
PROBLEM DETAIL: 3 = unexpected

CALLER ACTIVITY: 6 = machine check STATUS: 3
CALLER SUBACTIVITY: 13 = implementation dependent
REPORTING ENTITY TYPE: 0 = system firmware REPORTING ENTITY ID: 01

0x7000102379006133 C1BFF0FF FED08040 type 14 = Problem Detail
0x5800182379006133 00006803 17171B2E type 11 = Timestamp 04/23/2004 23:27:46
Type CR for next entry, - CR for previous entry, Q CR to quit.



Log Entry # 9 :
SYSTEM NAME: a500ap2-lc
DATE: 04/23/2004 TIME: 23:27:46
ALERT LEVEL: 2 = Non-Urgent operator attention required

SOURCE: 7 = memory
SOURCE DETAIL: 0 = unknown, no source stated SOURCE ID: 0
PROBLEM DETAIL: 0 = no problem detail

CALLER ACTIVITY: 6 = machine check STATUS: 2
CALLER SUBACTIVITY: 08 = implementation dependent
REPORTING ENTITY TYPE: 0 = system firmware REPORTING ENTITY ID: 01

0x7000102070006082 00000000 00392400 type 14 = Problem Detail
0x5800182070006082 00006803 17171B2E type 11 = Timestamp 04/23/2004 23:27:46
Type CR for next entry, - CR for previous entry, Q CR to quit.


Log Entry # 10 :
SYSTEM NAME: a500ap2-lc
DATE: 04/23/2004 TIME: 23:27:46
ALERT LEVEL: 2 = Non-Urgent operator attention required

SOURCE: 7 = memory
SOURCE DETAIL: 0 = unknown, no source stated SOURCE ID: 0
PROBLEM DETAIL: 0 = no problem detail

CALLER ACTIVITY: 6 = machine check STATUS: 2
CALLER SUBACTIVITY: 06 = implementation dependent
REPORTING ENTITY TYPE: 0 = system firmware REPORTING ENTITY ID: 01

0x6000102070006062 00000000 00000010 type 12 = Device Status
0x5800182070006062 00006803 17171B2E type 11 = Timestamp 04/23/2004 23:27:46
Type CR for next entry, - CR for previous entry, Q CR to quit.



Log Entry # 11 :
SYSTEM NAME: a500ap2-lc
DATE: 04/23/2004 TIME: 23:27:46
ALERT LEVEL: 7 = reserved

SOURCE: 0 = unknown, no source stated
SOURCE DETAIL: 0 = unknown, no source stated SOURCE ID: FF
PROBLEM DETAIL: 0 = no problem detail

CALLER ACTIVITY: 6 = machine check STATUS: 2
CALLER SUBACTIVITY: 29 = implementation dependent
REPORTING ENTITY TYPE: 0 = system firmware REPORTING ENTITY ID: 01

0x0000107000FF6292 00000000 3FB03000 type 0 = Data Field Unused
0x5800187000FF6292 00006803 17171B2E type 11 = Timestamp 04/23/2004 23:27:46
Type CR for next entry, - CR for previous entry, Q CR to quit.



Log Entry # 12 :
SYSTEM NAME: a500ap2-lc
DATE: 04/23/2004 TIME: 23:27:46
ALERT LEVEL: 2 = Non-Urgent operator attention required

SOURCE: 1 = processor
SOURCE DETAIL: 1 = processor general SOURCE ID: 1
PROBLEM DETAIL: 0 = no problem detail

CALLER ACTIVITY: 6 = machine check STATUS: 2
CALLER SUBACTIVITY: 32 = implementation dependent
REPORTING ENTITY TYPE: 0 = system firmware REPORTING ENTITY ID: 01

0x1800102011016322 CB818000 00000000 type 3 = Actual Data
0x5800182011016322 00006803 17171B2E type 11 = Timestamp 04/23/2004 23:27:46
Type CR for next entry, - CR for previous entry, Q CR to quit.


Log Entry # 13 :
SYSTEM NAME: a500ap2-lc
DATE: 04/23/2004 TIME: 23:27:46
ALERT LEVEL: 2 = Non-Urgent operator attention required

SOURCE: 0 = unknown, no source stated
SOURCE DETAIL: 0 = unknown, no source stated SOURCE ID: FF
PROBLEM DETAIL: 0 = no problem detail

CALLER ACTIVITY: 6 = machine check STATUS: 2
CALLER SUBACTIVITY: 24 = implementation dependent
REPORTING ENTITY TYPE: 0 = system firmware REPORTING ENTITY ID: 01

0x0000102000FF6242 00000000 00000000 type 0 = Data Field Unused
0x5800182000FF6242 00006803 17171B2E type 11 = Timestamp 04/23/2004 23:27:46
Type CR for next entry, - CR for previous entry, Q CR to quit.



Log Entry # 14 :
SYSTEM NAME: a500ap2-lc
DATE: 04/23/2004 TIME: 23:26:48
ALERT LEVEL: 2 = Non-Urgent operator attention required

SOURCE: 0 = unknown, no source stated
SOURCE DETAIL: 0 = unknown, no source stated SOURCE ID: FF
PROBLEM DETAIL: 0 = no problem detail

CALLER ACTIVITY: 6 = machine check STATUS: 2
CALLER SUBACTIVITY: 10 = implementation dependent
REPORTING ENTITY TYPE: 0 = system firmware REPORTING ENTITY ID: 00

0x0000002000FF6102 00000000 00000000 type 0 = Data Field Unused
0x5800082000FF6102 00006803 17171A30 type 11 = Timestamp 04/23/2004 23:26:48
Type CR for next entry, - CR for previous entry, Q CR to quit.



Log Entry # 15 :
SYSTEM NAME: a500ap2-lc
ALERT LEVEL: 8 = Boot Possible, performance impaired

SOURCE: 6 = platform
SOURCE DETAIL: 6 = service processor SOURCE ID: 0
PROBLEM DETAIL: 1 = selftest result

CALLER ACTIVITY: 2 = operation STATUS: 0
CALLER SUBACTIVITY: 03 = console
REPORTING ENTITY TYPE: 1 = service processor REPORTING ENTITY ID: 00

0x7010028166002030 0B0003FF D0100019 type 14 = Problem Detail
0x4810028166002030 5461736B 7265666C type 9 = ASCII Message reflTask
0x4810028166002030 78632E43 4C4F4765 type 9 = ASCII Message LOGexc.C
0x7010028166002030 00000000 00002275 type 14 = Problem Detail
0x70100A8166002030 000DA628 226A2004 type 14 = Problem Detail
Type CR for next entry, - CR for previous entry, Q CR to quit.

Log Entry # 16 :
SYSTEM NAME: a500ap2-lc
DATE: 01/01/1970 TIME: 00:00:00
ALERT LEVEL: 10 = Boot possible, functionality lost

SOURCE: 3 = PDH
SOURCE DETAIL: 6 = interconnect medium SOURCE ID: 0
PROBLEM DETAIL: 3 = non-responding, may need GSP reset.

CALLER ACTIVITY: 2 = operation STATUS: 0
CALLER SUBACTIVITY: 02 = platform internal interconnect
REPORTING ENTITY TYPE: 1 = service processor REPORTING ENTITY ID: 00

0x581008A336002020 00004600 01000000 type 11 = Timestamp 01/01/1970 00:00:00
Type CR for next entry, - CR for previous entry, Q CR to quit.



Log Entry # 17 :
SYSTEM NAME: a500ap2-lc
ALERT LEVEL: 8 = Boot Possible, performance impaired

SOURCE: 6 = platform
SOURCE DETAIL: 6 = service processor SOURCE ID: 0
PROBLEM DETAIL: 1 = selftest result

CALLER ACTIVITY: 2 = operation STATUS: 0
CALLER SUBACTIVITY: 03 = console
REPORTING ENTITY TYPE: 1 = service processor REPORTING ENTITY ID: 00

0x7010028166002030 0B0003FF D0100019 type 14 = Problem Detail
0x4810028166002030 5461736B 7265666C type 9 = ASCII Message reflTask
0x4810028166002030 78632E43 4C4F4765 type 9 = ASCII Message LOGexc.C
0x7010028166002030 00000000 00002275 type 14 = Problem Detail
0x70100A8166002030 000DA628 226A2004 type 14 = Problem Detail
Type CR for next entry, - CR for previous entry, Q CR to quit.



Log Entry # 18 :
SYSTEM NAME: a500ap2-lc
DATE: 01/01/1970 TIME: 00:00:00
ALERT LEVEL: 10 = Boot possible, functionality lost

SOURCE: 3 = PDH
SOURCE DETAIL: 6 = interconnect medium SOURCE ID: 0
PROBLEM DETAIL: 3 = non-responding, may need GSP reset.

CALLER ACTIVITY: 2 = operation STATUS: 0
CALLER SUBACTIVITY: 02 = platform internal interconnect
REPORTING ENTITY TYPE: 1 = service processor REPORTING ENTITY ID: 00

0x581008A336002020 00004600 01000000 type 11 = Timestamp 01/01/1970 00:00:00
Type CR for next entry, - CR for previous entry, Q CR to quit.

[Oliver Zell_1]

Hi,

please check what you have in io slot 3 (hardware path 0/6/2).
I think the error is located there. Can you use the card in this slot?

Is there also an actual hpmc (/var/tmomstones/ts99 with a valid timestamp)? In this case someone should better decode the hpmc.

[Marcel Preda]

Thanks for the reply.

how did you find `hardware path 0/6/2' ?

Unfortunatally I can not find nothing with SAM on that hardware path ( not Cards, nor Devices).

Unfortunately (again) between us and the machine is along distance, so... in this moment we have no idea wath is in slot 3.

Thanks again,
Marcel Preda

[Tobias Hartlieb]

Marcel,

please attach to your next response the output of 'ioscan -fn' and the file /var/tombstones/ts99 ...

Regards.

Tobias

[Marcel Preda]

here it comes.

[Tobias Hartlieb · Accepted solution]

Hi,

there are two odd things:
1st with the PCI slot 3 at HW path 0/6/2 (funtional failure of the bus adapter...).
However, the CPU #0 also reacted strange: It did not realize that a HPMC occurred.

Contact HP and get this checked in more detail!

Regards.

Tobias

[Marcel Preda]

thanks,
probable will do that.

where(URL) can I find a doc about decoding HPMC/GSP_errors messages?


the strange thing is that from the last reboot are more than 24 hours and no GSP message apears any more.


Thanks again,
Marcel

[Robert-Jan Goossens]

Hi,

http://www.docs.hp.com/cgi-bin/fsearch/framedisplay?top=/hpux/onlinedocs/5990-7226A/5990-7226A_top.html&con=/hpux/onlinedocs/5990-7226A/00/00/37-con.html&toc=/hpux/onlinedocs/5990-7226A/00/00/37-toc.html&searchterms=gsp%7chpmc&queryid=20040427-045821

Regards,
Robert-Jan