HP5470 system Reboot after panic

[ajiang]

have 1 hp 9000/5470 server while reboots in a day with a following error(shutdownlog):
13:08 Mon Aug 20, 2007. Reboot: (by dl5470!root)
19:59 Tue Sep 04 2007. Reboot after panic: , isr.ior = 0'7.0'1e29a278
20:06 Tue Sep 04 2007. Reboot after panic: Data page fault

and my /var/tomsbstones/ts99:
dl5470:/var/tombstones#>cat ts99
HP-UX dl5470 B.11.11 U 9000/800 1138434606

CPU-ID( Model ) = 0x13

------- Processor 0 HPMC Information - PDC Version: 44.12 ------

Timestamp = Tue Sep 4 11:52:35 GMT 2007 (20:07:09:04:11:52:35)

HPMC Chassis Codes

Chassis Code Extension
------------ ---------
0x0000082000ff6242 0x0000000000000000
0x1800082011006322 0xcb81800000000000
0x0000087000ff6292 0x000000007fa00000
0x6000082013006062 0x0000000000080004
0x6000082013006072 0x0000000000180000
0x7000082013006082 0x0000000000192200
0x6000082013026062 0x2001000000080004
0x6000082013026072 0x0000000000100000
0x7000082013026082 0x0000000000192200
0x6000082070006062 0x0000000000100800
0x7000082070006082 0x0000000000992600
0x6000082070016062 0x0000000000080000
0x7000082070016082 0x0000000000192200
0x0000080080006310 0x0000000000000001
0x7000082082006333 0x0000000000b92200
0x000008008000631f 0x0000000000000000
0x0000082000ff6462 0x0000000000000000


General Registers 0 - 31
00-03 0000000000000000 0000000000c05b78 00000000013373c8 0000000001039e40
04-07 0000000001334000 0000000000b04100 0000000000000000 0003848d00a29a1f
08-11 0000000000000001 0000000000000001 0000000000000001 00038489f5966c72
12-15 0000000000000002 00000000c0022fc3 00000000c68a65f8 0000000000007138
16-19 000000007aa1ad28 000000000000055c 0000000000000007 00000000000b71b0
20-23 0000000000097fae 0000000000000000 8000000800000030 0000000000b03d00
24-27 0000000000c05b78 0000000000000001 0003848d00a29a1f 0000000000c0bb78
28-31 0000000000000000 0003848d00a29a49 0000000000a47640 0000000000000000


Control Registers 0 - 31
00-03 0000000043c21e4a 0000000000000000 0000000000000000 0000000000000000
04-07 0000000000000000 0000000000000000 0000000000000000 0000000000000000
08-11 0000000000006d03 0000253300001960 00000000000000c0 000000000000003b
12-15 0000000000000000 0000000000000000 000000000002a000 c400000000000000
16-19 0003848d00b6e3da 0000000000000000 00000000001b8fac 00000000020008b7
20-23 0000000010240002 0000000091e46588 000000ff086cf21a 8000000800000030
24-27 0000000001334000 0000000000000000 0003848cc6409e3f 0000000040005b68
28-31 8000000800000030 0003848cc640e8c1 000000000015f848 0000000000a47640

Space Registers 0 - 7
00-03 09426000 0a35fc00 00000000 00000000
04-07 00000000 0a35fc00 0bc9b800 00000000


IIA Space (back entry) = 0x0000000000000000
IIA Offset (back entry) = 0x00000000001b8fec
Check Type = 0x20000000
CPU State = 0x9e000004
Cache Check = 0x00000000
TLB Check = 0x00000000
Bus Check = 0x0030000d
Assists Check = 0x00000000
Assist State = 0x00000000
Path Info = 0x00000000
System Responder Address = 0xfffffffffed21000
System Requestor Address = 0x000000fffed20000


Floating Point Registers 0 - 31
00-03 0000002000000000 0000000000000000 0000000000000000 0000000000000000
04-07 415c9ae9c0000000 3f55d867c3ece2a5 40c3871bd194237f 000000000000270e
08-11 0000000000726ba7 415c9c3800000000 3f55d867c3ece2a5 3f55d867c3ece2a5
12-15 5555555555555555 5555555555555555 5555555555555555 5555555555555555
16-19 5555555555555555 5555555555555555 5555555555555555 5555555555555555
20-23 5555555555555555 5555555555555555 3f55d867c3ece2a5 0000000000000037
24-27 3ff8ba2e8ba2e8c2 3ff7dac37dac37e2 4054c00000000000 0000000000000037
28-31 4054400000000000 3ff8253c8253c82d 0000000000000037 412bc38a00000000


Check Summary = 0xcb81800000000000
Available Memory = 0x0000000080000000
CPU Diagnose Register 2 = 0x0203000c04802204
CPU Status Register 0 = 0x3240c20000000000
CPU Status Register 1 = 0x8000000000000000
SADD LOG = 0x48e0000000000002
Read Short LOG = 0xc1af00f57fff6808



----------------- DEW 0 HPMC Information - ------

Timestamp = Tue Sep 4 11:52:36 GMT 2007 (20:07:09:04:11:52:36)

Runway Control Log Reg = 0x0090730000000000
Runway Address Data Log Reg Odd = 0xc0aa1010c4a61010
Runway Address Data Log Reg Even = 0xc8a61010cca61010
Runway Address Log Reg = 0x00000000000002f4
Runway Broad Error Log Reg = 0x0000000000000000

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
ERR_ERROR X X

Merced Bus Requestor Address = 0x0000000000000000
Merced Bus Target Address = 0x0000000000000000
Merced Bus Responder Address = 0x0000000000000000
Merced Error Status Reg = 0x0000000000080004
Merced Error Overflow Reg = 0x0000000000180000
Merced AERR Addr1 Log Reg = 0x00006000ff872bc0
Merced AERR Addr2 Log Reg = 0x000080010808ff18
Merced DERR Log Reg = 0x00c0000000000000
Merced Error Syndrome Reg = 0x00000000000000c0

-------------- Memory Error Log Information --------------

Bus 0 Log Information

Timestamp = Tue Sep 4 11:52:36 GMT 2007 (20:07:09:04:11:52:36)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
X ERR_ERROR X X X

Bus Requestor Address = 0x0000000000000000
Bus Target Address = 0x0000000000000000
Bus Responder Address = 0x0000000000000000

Error Status Reg = 0x0000000000100800
Error Overflow Reg = 0x0000000000000000
AERR Address 1 Log Reg = 0x0001251555555540
AERR Address 2 Log Reg = 0xf830b000078f5518
FERR Log Reg = 0x0000000000000000
DERR Log Reg = 0x00000300000aad40
Error Syndrome Reg = 0x0000000000000000



Address/Control Parity Error Registers

Address/Control Parity Error Bit (AE) Not Set



Bus 1 Log Information

Timestamp = Tue Sep 4 11:52:36 GMT 2007 (20:07:09:04:11:52:36)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
ERR_ERROR X X

Bus Requestor Address = 0x0000000000000000
Bus Target Address = 0x0000000000000000
Bus Responder Address = 0x0000000000000000

Error Status Reg = 0x0000000000080000
Error Overflow Reg = 0x0000000000000000
AERR Address 1 Log Reg = 0x08000001000c0000
AERR Address 2 Log Reg = 0x5800800007cfff00
FERR Log Reg = 0x0000000000000000
DERR Log Reg = 0x0000000000000000
Error Syndrome Reg = 0x0000000000000000



Address/Control Parity Error Registers

Address/Control Parity Error Bit (AE) Not Set



------------ I/O Module Error Log Information ------------

Summary of IO subsystem log entries
-----------------------------------
Phys Loc Vendor Device Severity
Description (hex) Id Id CORR UNC FE CW
----------- ----- ------ ------ ----------------
System Bus Adapter SB 0x000000ffffffff82 0x103c 0x1050 X
System Bus Adapter RP 0x000000ffff01ff83 0x103c 0x1051 X
System Bus Adapter RP 0x000000ffff07ff83 0x103c 0x1051 X
System Bus Adapter RP 0x000000ffff05ff83 0x103c 0x1051 X
System Bus Adapter RP 0x000000ffff08ff83 0x103c 0x1051 X
System Bus Adapter RP 0x000000ffff04ff83 0x103c 0x1051 X
System Bus Adapter RP 0x000000ffff06ff83 0x103c 0x1051 X
System Bus Adapter RP 0x000000ffff0aff83 0x103c 0x1051 X
System Bus Adapter RP 0x000000ffff09ff83 0x103c 0x1051 X
System Bus Adapter RP 0x000000ffff0cff83 0x103c 0x1051 X
System Bus Adapter RP 0x000000ffff0bff83 0x103c 0x1051 X


Detail display of IO subsystem log entries
------------------------------------------

System Bus Adapter -- System Bus Interface
------------------------------------------

Timestamp = Tue Sep 4 11:52:48 GMT 2007 (20:07:09:04:11:52:48)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
X X ERR_ERROR X X

IO Requestor Address = 0x0000000000000000
IO Target Address = 0x0000000000000000
IO Responder Address = 0xfffffffffed00000
IO Physical Location = 0x000000ffffffff82
IO Hardware Path = 0x00ffffffffffff00

Module Error Register = 0x00000000048f0034

System Bus Adapter -- Rope Interface
------------------------------------------

Timestamp = Tue Sep 4 11:52:49 GMT 2007 (20:07:09:04:11:52:49)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
ERR_FUNCTION X

IO Requestor Address = 0x0000000000000000
IO Target Address = 0x0000000000000000
IO Responder Address = 0x0000000000000000
IO Physical Location = 0x000000ffffffff82
IO Hardware Path = 0x00ffffffffffff00

Module Error Register = 0x0000000000000000
Rope Physical Location = 0x000000ffff01ff83

System Bus Adapter -- Rope Interface
------------------------------------------

Timestamp = Tue Sep 4 11:52:49 GMT 2007 (20:07:09:04:11:52:49)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
ERR_FUNCTION X

IO Requestor Address = 0x0000000000000000
IO Target Address = 0x0000000000000000
IO Responder Address = 0x0000000000000000
IO Physical Location = 0x000000ffffffff82
IO Hardware Path = 0x00ffffffffffff00

Module Error Register = 0x0000000000000000
Rope Physical Location = 0x000000ffff07ff83

System Bus Adapter -- Rope Interface
------------------------------------------

Timestamp = Tue Sep 4 11:52:49 GMT 2007 (20:07:09:04:11:52:49)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
ERR_FUNCTION X

IO Requestor Address = 0x0000000000000000
IO Target Address = 0x0000000000000000
IO Responder Address = 0x0000000000000000
IO Physical Location = 0x000000ffffffff82
IO Hardware Path = 0x00ffffffffffff00

Module Error Register = 0x0000000000000000
Rope Physical Location = 0x000000ffff05ff83

System Bus Adapter -- Rope Interface
------------------------------------------

Timestamp = Tue Sep 4 11:52:49 GMT 2007 (20:07:09:04:11:52:49)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
ERR_FUNCTION X

IO Requestor Address = 0x0000000000000000
IO Target Address = 0x0000000000000000
IO Responder Address = 0x0000000000000000
IO Physical Location = 0x000000ffffffff82
IO Hardware Path = 0x00ffffffffffff00

Module Error Register = 0x0000000000000000
Rope Physical Location = 0x000000ffff08ff83

System Bus Adapter -- Rope Interface
------------------------------------------

Timestamp = Tue Sep 4 11:52:49 GMT 2007 (20:07:09:04:11:52:49)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
ERR_FUNCTION X

IO Requestor Address = 0x0000000000000000
IO Target Address = 0x0000000000000000
IO Responder Address = 0x0000000000000000
IO Physical Location = 0x000000ffffffff82
IO Hardware Path = 0x00ffffffffffff00

Module Error Register = 0x0000000000000000
Rope Physical Location = 0x000000ffff04ff83

System Bus Adapter -- Rope Interface
------------------------------------------

Timestamp = Tue Sep 4 11:52:49 GMT 2007 (20:07:09:04:11:52:49)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
ERR_FUNCTION X

IO Requestor Address = 0x0000000000000000
IO Target Address = 0x0000000000000000
IO Responder Address = 0x0000000000000000
IO Physical Location = 0x000000ffffffff82
IO Hardware Path = 0x00ffffffffffff00

Module Error Register = 0x0000000000000000
Rope Physical Location = 0x000000ffff06ff83

System Bus Adapter -- Rope Interface
------------------------------------------

Timestamp = Tue Sep 4 11:52:49 GMT 2007 (20:07:09:04:11:52:49)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
ERR_FUNCTION X

IO Requestor Address = 0x0000000000000000
IO Target Address = 0x0000000000000000
IO Responder Address = 0x0000000000000000
IO Physical Location = 0x000000ffffffff82
IO Hardware Path = 0x00ffffffffffff00

Module Error Register = 0x0000000000000000
Rope Physical Location = 0x000000ffff0aff83

System Bus Adapter -- Rope Interface
------------------------------------------

Timestamp = Tue Sep 4 11:52:49 GMT 2007 (20:07:09:04:11:52:49)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
ERR_FUNCTION X

IO Requestor Address = 0x0000000000000000
IO Target Address = 0x0000000000000000
IO Responder Address = 0x0000000000000000
IO Physical Location = 0x000000ffffffff82
IO Hardware Path = 0x00ffffffffffff00

Module Error Register = 0x0000000000000000
Rope Physical Location = 0x000000ffff09ff83

System Bus Adapter -- Rope Interface
------------------------------------------

Timestamp = Tue Sep 4 11:52:49 GMT 2007 (20:07:09:04:11:52:49)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
ERR_FUNCTION X

IO Requestor Address = 0x0000000000000000
IO Target Address = 0x0000000000000000
IO Responder Address = 0x0000000000000000
IO Physical Location = 0x000000ffffffff82
IO Hardware Path = 0x00ffffffffffff00

Module Error Register = 0x0000000000000000
Rope Physical Location = 0x000000ffff0cff83

System Bus Adapter -- Rope Interface
------------------------------------------

Timestamp = Tue Sep 4 11:52:49 GMT 2007 (20:07:09:04:11:52:49)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
ERR_FUNCTION X

IO Requestor Address = 0x0000000000000000
IO Target Address = 0x0000000000000000
IO Responder Address = 0x0000000000000000
IO Physical Location = 0x000000ffffffff82
IO Hardware Path = 0x00ffffffffffff00

Module Error Register = 0x0000000000000000
Rope Physical Location = 0x000000ffff0bff83


------- Processor 2 HPMC Information - PDC Version: 44.12 ------

Timestamp = Tue Sep 4 11:52:35 GMT 2007 (20:07:09:04:11:52:35)

HPMC Chassis Codes

Chassis Code Extension
------------ ---------
0x0000082000ff6242 0x0000000000000000
0x1800082011026322 0xcb81800000000000
0x0000082000ff6462 0x0000000000000000
0x0000080080006300 0x0000000000000001
0x7000082082006333 0x0000000000b92200
0x7000082382006343 0x0000000000070200
0x7000082382016343 0x0000000000070200
0x7000082382026343 0x0000000000070200
0x7000082382036343 0x0000000000070200
0x7000082382046343 0x0000000000070200
0x7000082382056343 0x0000000000070200
0x7000082382086343 0x0000000000070200
0x7000082382096343 0x0000000000070200
0x70000823820a6343 0x0000000000070200
0x70000823820c6343 0x0000000000070200
0x0000080089006200 0x0000000000000000
0x0000080086006200 0x0000000000000000
0x000008008000630f 0x0000000000000000
0x0000080080006360 0x0000000000000000
0x000008008000636f 0x0000000000000000


General Registers 0 - 31
00-03 0000000000000000 00000000023c5340 0000000000165c48 00000000432480c0
04-07 0000000000000000 000000007fdec470 0000000000000000 000000000000000f
08-11 0000000000000007 000000001e29a278 000000000000000f 0000000000166048
12-15 0000000000000000 0000000000000000 0000000000000000 0000000000000000
16-19 0000000009e64b10 000000000071e29a 000000000095eaa2 0000000000000009
20-23 00000000400003ff 400003ffffff7b30 0000000000000009 0000000000000000
24-27 0001e29a00000007 000000000071e29a 000000000000000f 0000000000c0bb78
28-31 c400000000000000 000000007fdecbc0 000000071e29a280 00000000013373c8


Control Registers 0 - 31
00-03 00000000405dd392 0000000000000000 0000000000000000 0000000000000000
04-07 0000000000000000 0000000000000000 0000000000000000 0000000000000000
08-11 000000000000c047 0000000000007bde 00000000000000c0 0000000000000008
12-15 0000000000000000 0000253200000000 000000000002a000 c400000000000000
16-19 0003848d0080d5db 0000000000000000 00000000001688a4 000000000fc012d1
20-23 0000000000000007 000000001e29a278 0000000008000000 8800000800000000
24-27 00000000013373c8 000000000465d1d6 000000000000000f 000000007b030028
28-31 0000000000000009 0003848d007083d0 0000000000165c48 000000071e299db0

Space Registers 0 - 7
00-03 00000000 00000000 05c2c400 00000000
04-07 00000000 05c2c400 0bc9b800 00000000


IIA Space (back entry) = 0x0000000000000000
IIA Offset (back entry) = 0x00000000001688a8
Check Type = 0x20000000
CPU State = 0x9e000004
Cache Check = 0x00000000
TLB Check = 0x00000000
Bus Check = 0x0030000d
Assists Check = 0x00000000
Assist State = 0x00000000
Path Info = 0x00000000
System Responder Address = 0xfffffffffed61000
System Requestor Address = 0x000000fffed60000


Floating Point Registers 0 - 31
00-03 0000002000000000 0000000000000000 0000000000000000 0000000000000000
04-07 0000200047ae1481 3f70fef010fef016 40f0000000000000 406e200000000000
08-11 3ff0000000000005 0000000000010000 4070fef010fef016 000000000000010f
12-15 5555555555555555 5555555555555555 5555555555555555 5555555555555555
16-19 5555555555555555 5555555555555555 5555555555555555 5555555555555555
20-23 5555555555555555 5555555555555555 000000000000ff1f 3ff0000000000000
24-27 0000000000000000 3f8ea46a0fd69939 3f90ecdbf5257500 3f9d68c762ed62b2
28-31 3f9cb8eecc0e43b1 3ff8253c8253c82d 0000000000000037 41e1cecc14c00000


Check Summary = 0xcb81800000000000
Available Memory = 0x0000000080000000
CPU Diagnose Register 2 = 0x0203020c00802204
CPU Status Register 0 = 0x3240c20000000000
CPU Status Register 1 = 0x8080000000000000
SADD LOG = 0x48e0000000000002
Read Short LOG = 0xc1bff0ffff418010



----------------- DEW 2 HPMC Information - ------

Timestamp = Tue Sep 4 11:52:36 GMT 2007 (20:07:09:04:11:52:36)

Runway Control Log Reg = 0x0000720000000000
Runway Address Data Log Reg Odd = 0xfffffffffffc3f00
Runway Address Data Log Reg Even = 0xfffffffffffc3f00
Runway Address Log Reg = 0x0300000000000048
Runway Broad Error Log Reg = 0x030000000000001c

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
ERR_ERROR X X

Merced Bus Requestor Address = 0x0000000000000000
Merced Bus Target Address = 0x0000000000000000
Merced Bus Responder Address = 0x0000000000000000
Merced Error Status Reg = 0x2001000000080004
Merced Error Overflow Reg = 0x0000000000100000
Merced AERR Addr1 Log Reg = 0x0000000000000400
Merced AERR Addr2 Log Reg = 0x0000000000000400
Merced DERR Log Reg = 0x0000000000000000
Merced Error Syndrome Reg = 0x0000000000000000

-------------- Memory Error Log Information --------------

Bus 0 Log Information

Timestamp = Tue Sep 4 11:52:36 GMT 2007 (20:07:09:04:11:52:36)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
X ERR_ERROR X X X

Bus Requestor Address = 0x0000000000000000
Bus Target Address = 0x0000000000000000
Bus Responder Address = 0x0000000000000000

Error Status Reg = 0x0000000000100800
Error Overflow Reg = 0x0000000000000000
AERR Address 1 Log Reg = 0x0001251555555540
AERR Address 2 Log Reg = 0xf830b000078f5518
FERR Log Reg = 0x0000000000000000
DERR Log Reg = 0x00000300000aad40
Error Syndrome Reg = 0x0000000000000000



Address/Control Parity Error Registers

Address/Control Parity Error Bit (AE) Not Set



Bus 1 Log Information

Timestamp = Tue Sep 4 11:52:36 GMT 2007 (20:07:09:04:11:52:36)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
ERR_ERROR X X

Bus Requestor Address = 0x0000000000000000
Bus Target Address = 0x0000000000000000
Bus Responder Address = 0x0000000000000000

Error Status Reg = 0x0000000000080000
Error Overflow Reg = 0x0000000000000000
AERR Address 1 Log Reg = 0x08000001000c0000
AERR Address 2 Log Reg = 0x5800800007cfff00
FERR Log Reg = 0x0000000000000000
DERR Log Reg = 0x0000000000000000
Error Syndrome Reg = 0x0000000000000000



Address/Control Parity Error Registers

Address/Control Parity Error Bit (AE) Not Set



------------ I/O Module Error Log Information ------------

Summary of IO subsystem log entries
-----------------------------------
Phys Loc Vendor Device Severity
Description (hex) Id Id CORR UNC FE CW
----------- ----- ------ ------ ----------------
System Bus Adapter SB 0x000000ffffffff82 0x103c 0x1050 X
System Bus Adapter RP 0x000000ffff01ff83 0x103c 0x1051 X
System Bus Adapter RP 0x000000ffff07ff83 0x103c 0x1051 X
System Bus Adapter RP 0x000000ffff05ff83 0x103c 0x1051 X
System Bus Adapter RP 0x000000ffff08ff83 0x103c 0x1051 X
System Bus Adapter RP 0x000000ffff04ff83 0x103c 0x1051 X
System Bus Adapter RP 0x000000ffff06ff83 0x103c 0x1051 X
System Bus Adapter RP 0x000000ffff0aff83 0x103c 0x1051 X
System Bus Adapter RP 0x000000ffff09ff83 0x103c 0x1051 X
System Bus Adapter RP 0x000000ffff0cff83 0x103c 0x1051 X
System Bus Adapter RP 0x000000ffff0bff83 0x103c 0x1051 X


Detail display of IO subsystem log entries
------------------------------------------

System Bus Adapter -- System Bus Interface
------------------------------------------

Timestamp = Tue Sep 4 11:52:48 GMT 2007 (20:07:09:04:11:52:48)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
X X ERR_ERROR X X

IO Requestor Address = 0x0000000000000000
IO Target Address = 0x0000000000000000
IO Responder Address = 0xfffffffffed00000
IO Physical Location = 0x000000ffffffff82
IO Hardware Path = 0x00ffffffffffff00

Module Error Register = 0x00000000048f0034

System Bus Adapter -- Rope Interface
------------------------------------------

Timestamp = Tue Sep 4 11:52:49 GMT 2007 (20:07:09:04:11:52:49)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
ERR_FUNCTION X

IO Requestor Address = 0x0000000000000000
IO Target Address = 0x0000000000000000
IO Responder Address = 0x0000000000000000
IO Physical Location = 0x000000ffffffff82
IO Hardware Path = 0x00ffffffffffff00

Module Error Register = 0x0000000000000000
Rope Physical Location = 0x000000ffff01ff83

System Bus Adapter -- Rope Interface
------------------------------------------

Timestamp = Tue Sep 4 11:52:49 GMT 2007 (20:07:09:04:11:52:49)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
ERR_FUNCTION X

IO Requestor Address = 0x0000000000000000
IO Target Address = 0x0000000000000000
IO Responder Address = 0x0000000000000000
IO Physical Location = 0x000000ffffffff82
IO Hardware Path = 0x00ffffffffffff00

Module Error Register = 0x0000000000000000
Rope Physical Location = 0x000000ffff07ff83

System Bus Adapter -- Rope Interface
------------------------------------------

Timestamp = Tue Sep 4 11:52:49 GMT 2007 (20:07:09:04:11:52:49)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
ERR_FUNCTION X

IO Requestor Address = 0x0000000000000000
IO Target Address = 0x0000000000000000
IO Responder Address = 0x0000000000000000
IO Physical Location = 0x000000ffffffff82
IO Hardware Path = 0x00ffffffffffff00

Module Error Register = 0x0000000000000000
Rope Physical Location = 0x000000ffff05ff83

System Bus Adapter -- Rope Interface
------------------------------------------

Timestamp = Tue Sep 4 11:52:49 GMT 2007 (20:07:09:04:11:52:49)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
ERR_FUNCTION X

IO Requestor Address = 0x0000000000000000
IO Target Address = 0x0000000000000000
IO Responder Address = 0x0000000000000000
IO Physical Location = 0x000000ffffffff82
IO Hardware Path = 0x00ffffffffffff00

Module Error Register = 0x0000000000000000
Rope Physical Location = 0x000000ffff08ff83

System Bus Adapter -- Rope Interface
------------------------------------------

Timestamp = Tue Sep 4 11:52:49 GMT 2007 (20:07:09:04:11:52:49)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
ERR_FUNCTION X

IO Requestor Address = 0x0000000000000000
IO Target Address = 0x0000000000000000
IO Responder Address = 0x0000000000000000
IO Physical Location = 0x000000ffffffff82
IO Hardware Path = 0x00ffffffffffff00

Module Error Register = 0x0000000000000000
Rope Physical Location = 0x000000ffff04ff83

System Bus Adapter -- Rope Interface
------------------------------------------

Timestamp = Tue Sep 4 11:52:49 GMT 2007 (20:07:09:04:11:52:49)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
ERR_FUNCTION X

IO Requestor Address = 0x0000000000000000
IO Target Address = 0x0000000000000000
IO Responder Address = 0x0000000000000000
IO Physical Location = 0x000000ffffffff82
IO Hardware Path = 0x00ffffffffffff00

Module Error Register = 0x0000000000000000
Rope Physical Location = 0x000000ffff06ff83

System Bus Adapter -- Rope Interface
------------------------------------------

Timestamp = Tue Sep 4 11:52:49 GMT 2007 (20:07:09:04:11:52:49)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
ERR_FUNCTION X

IO Requestor Address = 0x0000000000000000
IO Target Address = 0x0000000000000000
IO Responder Address = 0x0000000000000000
IO Physical Location = 0x000000ffffffff82
IO Hardware Path = 0x00ffffffffffff00

Module Error Register = 0x0000000000000000
Rope Physical Location = 0x000000ffff0aff83

System Bus Adapter -- Rope Interface
------------------------------------------

Timestamp = Tue Sep 4 11:52:49 GMT 2007 (20:07:09:04:11:52:49)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
ERR_FUNCTION X

IO Requestor Address = 0x0000000000000000
IO Target Address = 0x0000000000000000
IO Responder Address = 0x0000000000000000
IO Physical Location = 0x000000ffffffff82
IO Hardware Path = 0x00ffffffffffff00

Module Error Register = 0x0000000000000000
Rope Physical Location = 0x000000ffff09ff83

System Bus Adapter -- Rope Interface
------------------------------------------

Timestamp = Tue Sep 4 11:52:49 GMT 2007 (20:07:09:04:11:52:49)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
ERR_FUNCTION X

IO Requestor Address = 0x0000000000000000
IO Target Address = 0x0000000000000000
IO Responder Address = 0x0000000000000000
IO Physical Location = 0x000000ffffffff82
IO Hardware Path = 0x00ffffffffffff00

Module Error Register = 0x0000000000000000
Rope Physical Location = 0x000000ffff0cff83

System Bus Adapter -- Rope Interface
------------------------------------------

Timestamp = Tue Sep 4 11:52:49 GMT 2007 (20:07:09:04:11:52:49)

OV RQ RS ESTAT A C D corr unc fe cw pf
-- -- -- ----- - - - ---- --- -- -- --
ERR_FUNCTION X

IO Requestor Address = 0x0000000000000000
IO Target Address = 0x0000000000000000
IO Responder Address = 0x0000000000000000
IO Physical Location = 0x000000ffffffff82
IO Hardware Path = 0x00ffffffffffff00

Module Error Register = 0x0000000000000000
Rope Physical Location = 0x000000ffff0bff83


Module Revision
------ --------
System Board A14203
PA 8700 CPU Module 2.3
PA 8700 CPU Module 2.3


p4 analysis result carsh_info:
libp4 (9.18): Opening ./vmunix ./INDEX

Loading symbols from ./vmunix
Kernel TEXT pages not requested in crashconf
Will use an artificial mapping from a.out TEXT pages

crashinfo (3.32) output

=====================
= Table Of Contents =
=====================


* General Information
* Crash Events
* Message Buffer
* Memory Globals
* Buffer Cache Globals
* Swap Information
* Network Interfaces
* IOVA Usage Check
* Crash Event / Processor Information
* Processor Clock Info
* Load Averages
* Thread Information
* Kernel Patches
* Signature

=======================
= General Information =
=======================

Dump time Thu Aug 16 01:53:18 2007 UTC-8
System has been up 1 day, 10 hours, 49 minutes.

System Name : HP-UX
Node Name : dl5470
Model : 9000/800/L3000-7x
HP-UX version : B.11.11 (64-bit Kernel)
Number of CPU's : 2
Disabled CPU's : 0
CPU type : PCXW+ (750 Mhz)
CPU Architecture : PA-RISC 2.0
Load average : 0.01 0.02 0.02

================
= Crash Events =
================


Note: Crash event 0 was a PANIC !

Panic string : Data page fault


Note: In the case of a PANIC, normally crash event 0 is the crash
event you should concentrate on. There may well be other secondary
panics (for example spinlock panics) that have happened as a
consequence of the original panic.

Getting trap information from trap marker at 0x0.0x6e26270

Trap information
================

Type 15: Data TLB Miss Fault/Data Page Fault

Interruption Instruction Register:
IIR = 0x4b8f0000

Interruption Space and Offset Registers:
ISR.IOR = 0x0.0x3b8

Interruption Instruction Address Queue:
PCSQ.PCOQ = 0x0.0x179538 = idle+0x840

Interrupt Instruction at idle+0x840:
ldw 0(ret0),r15

Note: Arena free list lock(s) are held !
Checking locked arena free list heads for possible corruption...


The following free list is locked:
kmem_arena_t 0x0000000040001140 ""
kmem_flist_hdr_t 0x0000000000a82b58 (cpu 0, size 0)

Error while scanning a "" free list:
kmem_flist_hdr_t 0x0000000000a82b58 (cpu 0, index -1, size 0)
kfh_head 0x000055a63b0e2615 (no translation!)


The following free list is locked:
kmem_arena_t 0x0000000040001140 ""
kmem_flist_hdr_t 0x0000000000a82bd8 (cpu 1, size 0)


The following free list is locked:
kmem_arena_t 0x0000000040001038 " "
kmem_flist_hdr_t 0x0000000000a85000 (cpu 0, size 0)


WARNING: Possible arena freelist corruption ! See above for details.
For further details on this specific problem go to:
"http://wtec.cup.hp.com/~hpux/crash/FirstPassWeb/PA/trap_panic/freelist_corruption.htm"


Stack Trace for Crash event 0
=============================

============== EVENT ============================
= Event #0 is PANIC on CPU #1
= p crash_event_t 0xa8f000
= p rpb_t 0xa88100
= Using pc from pim.wide.rp_rp_hi = 0x20c224
============== EVENT ============================
SR4=0x00000000
SP RP Return Name
0x0000000006e26a60 0x0020c224 panic+0x6c
0x0000000006e269c0 0x0026208c report_trap_or_int_and_panic+0x94
0x0000000006e26920 0x00166580 trap+0xef8
0x0000000006e26740 0x00168890 thandler+0xd24
+------------- TRAP ----------------------------
| Trap type 15 in KERNEL mode at 0x179538 (idle+0x840)
| p struct save_state 0.0x6e26270
+------------- TRAP ----------------------------
SR4=0x00000000
SP RP Return Name
0x0000000006e26270 0x00179538 idle+0x840
0x0000000006e26050 0x0017715c swidle+0x28

Stack Trace for Crash event 0 with all args
===========================================

============== EVENT ============================
= Event #0 is PANIC on CPU #1
= p crash_event_t 0xa8f000
= p rpb_t 0xa88100
= Using pc from pim.wide.rp_rp_hi = 0x20c224
============== EVENT ============================
SR4=0x00000000
SP RP Return Name
0x0000000006e26a60 0x0020c224 panic+0x6c
0x0000000006e269c0 0x0026208c report_trap_or_int_and_panic+0x94
arg0: 0x0000000000000001
arg1: 0x000000000000000f
arg2: 0x0000000006e26270
arg3: 0x000000000092dd80
0x0000000006e26920 0x00166580 trap+0xef8
.... --------n/a-------
arg1: 0x0000000006e26270
0x0000000006e26740 0x00168890 thandler+0xd24
+------------- TRAP ----------------------------
| Trap type 15 in KERNEL mode at 0x179538 (idle+0x840)
| p struct save_state 0.0x6e26270
+------------- TRAP ----------------------------
SR4=0x00000000
SP RP Return Name
0x0000000006e26270 0x00179538 idle+0x840
0x0000000006e26050 0x0017715c swidle+0x28

Stack Traces for all other Crash events
=======================================

============== EVENT ============================
= Event #1 is PANIC on CPU #1
= p crash_event_t 0xa8f030
= p rpb_t 0xa88470
= Using pc from pim.wide.rp_rp_hi = 0x20c224
============== EVENT ============================
SR4=0x00000000
SP RP Return Name
0x0000000040000950 0x0020c224 panic+0x6c
0x00000000400008b0 0x0026208c report_trap_or_int_and_panic+0x94
0x0000000040000810 0x00166580 trap+0xef8
0x0000000040000630 0x00168890 thandler+0xd24
+------------- TRAP ----------------------------
| Trap type 15 in KERNEL mode at 0x167efc (thandler+0x390)
| p struct save_state 0.0x40000160
+------------- TRAP ----------------------------
SR4=0x00000000
SP RP Return Name
0x0000000040000160 0x00167efc thandler+0x390
+------------- TRAP ----------------------------
| Unexpected Trap type 0 at 0x80000.0 (???)
| p struct save_state 0.0x3ffffc90
+------------- TRAP ----------------------------


============== EVENT ============================
= Event #2 is TOC on CPU #0
= p crash_event_t 0xa8f060
= p rpb_t 0xa88e50
= Using pc from pim.wide.rp_pcoq_head_hi = 0x261fdc
============== EVENT ============================
SR5=0x0e98a400
SP RP Return Name
0x400003ffffff14c8 0x00261fdc check_panic_loop+0x24
0x400003ffffff14c8 0x001661dc trap+0xb54
0x400003ffffff12e8 0x00168890 thandler+0xd24
+------------- TRAP ----------------------------
| Trap type 31 in KERNEL mode at 0x14f254 (spluser+0x14)
| p struct save_state 0xe98a400.0x400003ffffff0e18
+------------- TRAP ----------------------------
SR5=0x0e98a400
SP RP Return Name
0x400003ffffff0e18 0x0014f254 spluser+0x14
0x400003ffffff0da8 0x0014e8fc syscall+0x3fc
0x400003ffffff0c78 0x00033f64 syscallinit+0x55c



==================
= Message Buffer =
==================

gate64: sysvec_vaddr = 0xc0002000 for 2 pages
NOTICE: autofs_link(): File system was registered at index 3.
NOTICE: cachefs_link(): File system was registered at index 5.
NOTICE: nfs3_link(): File system was registered at index 6.
0 sba
0/0 lba
0/0/0/0 btlan
0/0/1/0 c720
0/0/1/0.7 tgt
0/0/1/0.7.0 sctl
0/0/1/1 c720
0/0/1/1.0 tgt
0/0/1/1.0.0 sdisk
0/0/1/1.7 tgt
0/0/1/1.7.0 sctl
0/0/2/0 c720
0/0/2/0.0 tgt
0/0/2/0.0.0 sdisk
0/0/2/0.7 tgt
0/0/2/0.7.0 sctl
0/0/2/1 c720
0/0/2/1.2 tgt
0/0/2/1.2.0 sdisk
0/0/2/1.7 tgt
0/0/2/1.7.0 sctl
0/0/4/0 func0
0/0/4/1 asio0
0/1 lba
0/2 lba
0/3 lba
0/4 lba
0/5 lba
0/8 lba
0/9 lba
0/10 lba
0/10/0/0 igelan
0/12 lba
32 pbc
33 processor
96 pbc
97 processor
192 memory
btlan: Initializing 10/100BASE-TX card at 0/0/0/0....

System Console is on the Built-In Serial Interface
igelan1: INITIALIZING HP A6825-60101 PCI 1000Base-T Adapter at hardware path 0/10/0/0
Logical volume 64, 0x3 configured as ROOT
Logical volume 64, 0x2 configured as SWAP
Logical volume 64, 0x2 configured as DUMP
Swap device table: (start & size given in 512-byte blocks)
entry 0 - major is 64, minor is 0x2; start = 0, size = 8388608
Dump device table: (start & size given in 1-Kbyte blocks)
entry 0000000000000000 - major is 31, minor is 0x10000; start = 314208, size = 4194304
Starting the STREAMS daemons-phase 1
Create STCP device files
Starting the STREAMS daemons-phase 2
$Revision: vmunix: vw: -proj selectors: CUPI80_BL2000_1108 -c 'Vw for CUPI80_BL2000_1108 build' -- cupi80_bl2000_1108 'CUPI80_BL2000_1108' Wed Nov 8 19:24:56 PST 2000 $
Memory Information:
physical page size = 4096 bytes, logical page size = 4096 bytes
Physical: 2097152 Kbytes, lockable: 1622592 Kbytes, available: 1870200 Kbytes


Trap Type 15 (Data page fault):
Instruction Address (pcsq.pcoq) = 0x0.0x179538
Instruction (iir) = 0x4b8f0000 (load/store)
Target Address (isr.ior) = 0x0.0x00000000000003b8
Base Register (gr28) = 0x00000000000003b8
Savestate Ptr (ssp) = 0x0.0x0000000006e26270
Savestate Return Pointer (ss_rp) = 0x0000000000c03378

linkstamp: Tue Jul 31 10:30:57 EAT 2007
_release_version: @(#) $Revision: vmunix: vw: -proj selectors: CUPI80_BL2000_1108 -c 'Vw for CUPI80_BL2000_1108 build' -- cupi80_bl2000_1108 'CUPI80_BL2000_1108' Wed Nov 8 19:24:56 PST 2000 $
panic: Data page fault

PC-Offset Stack Trace (read across, top of stack is 1st):
0x0020c258 0x0026208c 0x00166580
0x00168890 0x00179538 0x0017715c
End Of Stack

Trap Type 15 (Data page fault):
Instruction Address (pcsq.pcoq) = 0x0.0x167efc
Instruction (iir) = 0x6bd63661 (load/store)
Target Address (isr.ior) = 0x0.0x000000003ffffc90
Base Register (gr30) = 0x0000000040000160
Savestate Ptr (ssp) = 0x0.0x0000000040000160
Savestate Return Pointer (ss_rp) = 0x0000000000445008

linkstamp: Tue Jul 31 10:30:57 EAT 2007
_release_version: @(#) $Revision: vmunix: vw: -proj selectors: CUPI80_BL2000_1108 -c 'Vw for CUPI80_BL2000_1108 build' -- cupi80_bl2000_1108 'CUPI80_BL2000_1108' Wed Nov 8 19:24:56 PST 2000 $
panic: Data page fault


==================
= Memory Globals =
==================

Physical Memory = 524288 pages (2.00 GB)
Free Memory = 359501 pages (1.37 GB)
Average Free Memory = 359472 pages (1.37 GB)
gpgslim = 1024 pages (4.00 MB)
lotsfree = 8192 pages (32.00 MB)
desfree = 1024 pages (4.00 MB)
minfree = 256 pages (1.00 MB)

========================
= Buffer Cache Globals =
========================

dbc_max_pct = 10 %
dbc_min_pct = 2 %
dbc current pct = 10.0 %
bufpages = 52428 pages (204.80 MB)
Number of buf headers = 27948

fixed_size_cache = 0
dbc_parolemem = 0
dbc_stealavg = 0
dbc_ceiling = 52428 pages (204.80 MB)
dbc_nbuf = 5242
dbc_bufpages = 10485 pages (40.96 MB)
dbc_vhandcredit = 0
orignbuf = 0
origbufpages = 0 pages

====================
= Swap Information =
====================

swapinfo -mt emulation
======================

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 LVM vg00/lv2
reserve - 340 -340
memory 1587 303 1284 19%
total 5683 643 5040 11% - 0 -


======================
= Network Interfaces =
======================


Note: Bad super pointer de-ref hp_dlpi_mib_counter_t-->ifInUDiscards
crashinfo will continue but beware of garbage output !

Note: Bad super pointer de-ref hp_dlpi_mib_counter_t-->ifInNUDiscards
crashinfo will continue but beware of garbage output !

Note: Bad super pointer de-ref hp_dlpi_mib_counter_t-->ifInUDiscards
crashinfo will continue but beware of garbage output !

Note: Bad super pointer de-ref hp_dlpi_mib_counter_t-->ifInNUDiscards
crashinfo will continue but beware of garbage output !
Name PPA Driver Interface Mac States IP
Name Description Address Link IP Address
--------------------------------------------------------------------------------
lan0 0 btlan 100BT 0x00306e2c30ba DOWN n/c n/c
lan1 1 igelan Gigabit 0x00306e49f6c5 UP UP 21.20.1.30

n/c : means "Not Configured", ifconfig has not been done on this interface


If you want more information, you can use : "lanshow -f"

====================
= IOVA Usage Check =
====================


98% of IOVA still available/free.

=======================================
= Crash Event / Processor Information =
=======================================

Number of processors = 2


s
t
a spin reg eiem/spl eirr ipsw
evt cpu t type dpth src cr15 cr23 cr22
--- --- - ----- ---- --- ---------------- ---------------- ----------------
0 1 E PANIC 0 rpb fffffff0ffffffff 0000000000000000 0800001f WRQPDI
15 s_s fffffff0ffffffff N/A 0804e81f WCRQPDI
1 1 E PANIC 0 rpb fffffff0ffffffff 0000000800000000 0800001f WRQPDI
15 s_s fffffff0ffffffff N/A 0804001e WCRQPD
0 s_s 0000000000000000 N/A 00000000
2 0 E TOC 0 rpb fffffff0ffffffff 0000000800000020 0844001f WLCRQPDI
31 s_s ffffffffffffffff N/A 0804001f WCRQPDI

Trap Types :

Trap type 0 = Non Existant Interrupt (I_NONEXIST)
Trap type 15 = Data TLB Miss Fault/Data page Fault (I_DPGFT)
Trap type 31 = Panic Trap (T_PANIC)

SPL/EIEM values:

0xfffffffeffffffff = SPLPREEMPTOK - Default user mode SPL level.
0xfffffff0ffffffff = SPLNOPREEMPT - Disable kernel preemption (scheduling interrupt off).
0xffffff00ffffffff = SPL2 - Disable software interrupt (software triggers off).
0xed00080000000000 = SPL5 - Disable IO modules.
0xc500000000000000 = SPL6+CLOCK_RESYNC - Disable hardclock+enable clock-resync.
0xc400000000000000 = SPL6 - Disable hardclock.
0x0000000700000000 = SPL7/PSW_I=0 - Disable the world.


========================
= Processor Clock Info =
========================

hardclock_late = 0
itick_per_tick = 7500000
lbolt = 12540912 (0xbf5bf0)

event mpi rpb delta clk eiem eirr PSW
cpu type timeinval interval timer secs:ticks od 0,4 0,4 I
--- ----- ------------------ ------------------ ---------- --- ---- ---- ---
0 TOC 0x55a63b7d5ab5 0x55a63b509840 0:0 0 1 1 0 0 1
1 PANIC 0x55a63af5daef 0x55a62193ad02 0:56 0 1 1 0 0 1

Note: For the PANIC crash event type it is normal to see a positive delta ticks.
This is due to the processor executing dumpsys() code after creating it's
crash event and rpb.


=================
= Load Averages =
=================


avenrun
=======
0.01 0.02 0.02

mp_avenrun
==========
cpu0 : 0.012234 0.018423 0.019728
cpu1 : 0.014063 0.012355 0.011775

======================
= Thread Information =
======================

10 Threads ran in the last second
41 Threads ran in the last 5 seconds
46 Threads ran in the last 10 seconds
61 Threads ran in the last minute
70 Threads ran in the last hour

statdaemon ran 122 ticks ago


Note: statdaemon didn't run in the last second !
The statdaemon should normally run every second.
If it has not run in the past second it may be an indication that
it is being starved by higher priority or looping process(es).


Most Common Wait Channels
=========================
ticks since run:
Wait Channel count longest shortest
------------ ----- ---------- ----------
per_processor_selects+0x40 18 12526998 470
async_bufhead 16 12527121 12527120
per_processor_selects 14 12526015 103
svc_run(0x4028e040) 8 12521655 12521653
svc_run(0x4028e080) 8 12521655 12521654
lvmkd_q 6 156 156
msgque[7]+0x5c 3 6049962 6049961
streams_blk_sync 2 12536463 12536463
streams_mp_sync 2 3202672 3198081


Most Common Sleep Callers
=========================
ticks since run:
Sleep Caller count longest shortest
------------ ----- ---------- ----------
ksleep() 30 12521663 1284
select() 26 12526040 398
async_daemon() 16 12527121 12527120
svc_run() 16 12521655 12521653
vx_worklist_thread() 15 257 57
poll() 9 12526998 103
msgrcv2() 7 12528576 409
real_nanosleep() 6 274781 63
lvmkd_daemon() 6 156 156
semop() 5 12528252 1491
pm_sigwait() 5 10642 82
fifo_rdwr() 2 12524952 12524899
wait1() 2 12521474 6049964
sosleep() 2 12514791 12514790
sync_buffer_cache() 2 685 368


Idle Globals
============

candidate_idle_spu = -1
migration_cycles = 1500000

Running Threads (TSRUNPROC) and idle Processors
===============================================


TICKS TICKS I TICKS
SINCE SINCE C SINCE NREADY
TID PID PPID RUN IDLE PRI SPU STATE S MIGR FR LO AL COMMAND
------- ----- ----- ---------- ---------- --- --- ----- - --------- -- -- -- -------
2209 2161 1861 57 57 158 0 SYS N 57 12 0 0 psmctd
57 1 SYS N 57 11 0 0

Note:
FR: free to run on any processor (candidate for thread migration).
LO: locked (via processor affinity/mpctl) to this processor).
AL: Alpha semaphores misses (special scheduling when miss a sema).


Threads waiting on cpu (TSRUN) - sorted by cpu/pri/ticks-since-run
==================================================================

TID PID PPID TICKS PRI SPU BND GRP SYSCALL COMMAND
------- ----- ----- ---------- --- --- --- --- -------------- -------
12 12 0 12536462 -32 0 N -1 n/a ttisr
0 0 0 122 127 0 N -1 n/a swapper
4 4 0 122 128 0 N -1 n/a unhashdaemon
45 37 0 57 138 0 N -1 n/a vxfsd
49 37 0 57 138 0 N -1 n/a vxfsd
50 37 0 57 138 0 N -1 n/a vxfsd
51 37 0 157 138 0 N -1 n/a vxfsd
48 37 0 255 138 0 N -1 n/a vxfsd
46 37 0 257 138 0 N -1 n/a vxfsd
47 37 0 257 138 0 N -1 n/a vxfsd
858 831 1 11497 154 0 N -1 select syslogd
43213 13286 13276 63 168 0 N -1 nanosleep db2fmp
2445 2392 1 125 168 0 N -1 nanosleep db2fmcd
3 3 0 122 128 1 N -1 n/a statdaemon
41 37 0 57 138 1 N -1 n/a vxfsd
43 37 0 57 138 1 N -1 n/a vxfsd
38 37 0 157 138 1 N -1 n/a vxfsd
39 37 0 157 138 1 N -1 n/a vxfsd
42 37 0 157 138 1 N -1 n/a vxfsd
44 37 0 157 138 1 N -1 n/a vxfsd
52 37 0 157 138 1 N -1 n/a vxfsd
40 37 0 257 138 1 N -1 n/a vxfsd
1794 1761 1 137 154 1 N -1 poll pwgrd
1552 1519 1 521 154 1 N -1 select sendmail

All Threads - sorted by ticks-since-run
=======================================

TID PID PPID TICKS PRI SPU STAT SYSCALL COMMAND WCHAN
------- ----- ----- ---------- --- --- ----- -------------- -------------- -----
37 37 0 57 138 1 SLEEP n/a vxfsd vx_event_wait(0x4a311e00)
41 37 0 57 138 1 RUN n/a vxfsd
43 37 0 57 138 1 RUN n/a vxfsd
45 37 0 57 138 0 RUN n/a vxfsd
49 37 0 57 138 0 RUN n/a vxfsd
50 37 0 57 138 0 RUN n/a vxfsd
2209 2161 1861 57 158 0 RUN n/a psmctd
43213 13286 13276 63 168 0 RUN nanosleep db2fmp
918 890 889 82 127 1 SLEEP sigtimedwait netfmt pm_sigwait(0x4244c100)
2207 2159 1861 93 168 0 SLEEP read diaglogd diag2_sleep_flag
2552 2230 1 103 154 0 SLEEP poll vxsvc per_processor_selects
0 0 0 122 127 0 RUN n/a swapper
2 2 0 122 128 0 SLEEP n/a vhand vhandsema
3 3 0 122 128 1 RUN n/a statdaemon
4 4 0 122 128 0 RUN n/a unhashdaemon
2445 2392 1 125 168 0 RUN nanosleep db2fmcd
1794 1761 1 137 154 1 RUN poll pwgrd
2897 2776 1 140 168 1 SLEEP nanosleep db2fmd real_nanosleep(0x48a4e4a8)
19 19 0 156 147 0 SLEEP n/a lvmkd lvmkd_q
20 20 0 156 147 0 SLEEP n/a lvmkd lvmkd_q
21 21 0 156 147 1 SLEEP n/a lvmkd lvmkd_q
22 22 0 156 147 0 SLEEP n/a lvmkd lvmkd_q
23 23 0 156 147 0 SLEEP n/a lvmkd lvmkd_q
24 24 0 156 147 0 SLEEP n/a lvmkd lvmkd_q
38 37 0 157 138 1 RUN n/a vxfsd
39 37 0 157 138 1 RUN n/a vxfsd
42 37 0 157 138 1 RUN n/a vxfsd
44 37 0 157 138 1 RUN n/a vxfsd
51 37 0 157 138 0 RUN n/a vxfsd
52 37 0 157 138 1 RUN n/a vxfsd
2340 2230 1 168 154 0 SLEEP poll vxsvc per_processor_selects
2206 2158 1861 192 158 0 SLEEP sigtimedwait cclogd pm_sigwait(0x4259a2c0)
48 37 0 255 138 0 RUN n/a vxfsd
40 37 0 257 138 1 RUN n/a vxfsd
46 37 0 257 138 0 RUN n/a vxfsd
47 37 0 257 138 0 RUN n/a vxfsd
721 694 1 368 154 1 SLEEP tsync syncer sync_proc_headers+0x10
1898 1861 1 398 154 0 SLEEP select diagmond per_processor_selects
43200 13276 13275 409 155 0 SLEEP msgrcv db2sysc msgque[9]+0x5c
2430 2377 1 435 168 1 SLEEP sigtimedwait samd pm_sigwait(0x4244c340)
2958 2837 1 470 154 1 SLEEP select sysstat_em per_processor_selects+0x40
1552 1519 1 521 154 1 RUN select sendmail
720 694 1 685 154 0 SLEEP tsync syncer sync_proc_headers
1969 1932 1 686 154 1 SLEEP select swagentd per_processor_selects+0x40
2894 2773 1 897 154 1 SLEEP select lpmc_em per_processor_selects+0x40
1705 1672 1 966 154 0 SLEEP select dced per_processor_selects
1617 1584 1 1026 154 0 SLEEP select mib2agt per_processor_selects
1 1 0 1034 168 1 SLEEP sigsuspend init *uptr+0
2455 2377 1 1284 154 1 SLEEP ksleep samd ksleep(0x4290b140)
1229 1201 1 1406 154 0 SLEEP select inetd per_processor_selects
2444 2391 1 1406 168 0 SLEEP sigtimedwait p_client pm_sigwait(0x4259a140)
2554 2438 1 1406 154 1 SLEEP select disk_em per_processor_selects+0x40
2589 2473 1 1406 154 0 SLEEP select dm_core_hw per_processor_selects
2640 2524 1 1406 154 1 SLEEP select dm_memory per_processor_selects+0x40
2756 2640 1 1406 154 1 SLEEP select fpl_em per_processor_selects+0x40
2842 2726 1 1406 154 0 SLEEP select ia64_corehw per_processor_selects
43207 13283 13276 1491 156 1 SLEEP semop db2sysc semop(0x4a24c5c4)
43403 13476 13281 1537 156 0 SLEEP semop db2sysc semop(0x4a3a3224)
43205 13281 13276 1541 155 0 SLEEP msgrcv db2sysc msgque[10]+0x5c
2525 2230 1 2091 168 0 SLEEP nanosleep vxsvc real_nanosleep(0x48b8e4a8)
1170 1142 1 4210 154 1 SLEEP select automount per_processor_selects+0x40
2428 2375 1 10642 138 0 SLEEP sigtimedwait krsd pm_sigwait(0x4259a240)
858 831 1 11497 154 0 RUN select syslogd
749 722 0 18912 153 1 SLEEP n/a dmprestored dmp_restore_daemon_sync+0x8
1727 1694 1 45445 64 0 SLEEP select rbootd per_processor_selects
2495 2377 1 106381 154 1 SLEEP select samd per_processor_selects+0x40
4155 2377 1 106381 154 1 SLEEP ksleep samd ksleep(0x48ca61c0)
2868 2739 1 273999 168 1 SLEEP nanosleep db2dasrrm real_nanosleep(0x48d1e4a8)
2869 2739 1 274781 168 0 SLEEP nanosleep db2dasrrm real_nanosleep(0x48e3f4a8)
2208 2160 1861 281998 154 0 SLEEP select memlogd per_processor_selects
2298 2230 1 3031928 154 1 SLEEP ksleep vxsvc ksleep(0x4246a700)
27 27 0 3198081 100 1 SLEEP n/a smpsched streams_mp_sync
26 26 0 3202672 100 0 SLEEP n/a smpsched streams_mp_sync
43210 13286 13276 6049527 156 0 SLEEP semop db2fmp semop(0x4aad7d64)
43212 13286 13276 6049624 154 1 SLEEP ksleep db2fmp ksleep(0x4290b9c0)
43211 13286 13276 6049625 154 1 SLEEP ksleep db2fmp ksleep(0x4290b940)
43204 13280 13276 6049820 155 0 SLEEP msgrcv db2sysc msgque[8]+0x5c
43208 13284 13280 6049826 156 0 SLEEP semop db2sysc semop(0x4aa963c4)
43206 13282 13276 6049932 154 0 SLEEP select db2sysc per_processor_selects
43203 13279 13276 6049961 155 0 SLEEP msgrcv db2sysc msgque[7]+0x5c
43201 13277 13276 6049962 155 0 SLEEP msgrcv db2sysc msgque[7]+0x5c
43202 13278 13276 6049962 155 0 SLEEP msgrcv db2sysc msgque[7]+0x5c
43199 13275 1 6049964 158 0 SLEEP waitpid db2sysc wait1(0x48eb8040)
8413 8291 1 11773878 156 0 SLEEP read getty DEVICE 0x00000000
2218 2170 1201 12481402 154 1 SLEEP select registrar per_processor_selects+0x40
2463 37 0 12490531 138 0 ZOMB n/a vxfsd
2855 2739 1 12514790 154 0 SLEEP accept db2dasrrm STREAM 0x48e307c0
2874 2739 1 12514790 154 0 SLEEP accept db2dasrrm SOCKET 0x48a40b40
2872 2739 1 12514791 154 0 SLEEP recvfrom db2dasrrm hpstreams_read_int(0x48e30468)
2873 2739 1 12514791 154 1 SLEEP accept db2dasrrm SOCKET 0x48d81e80
2454 2400 2370 12518516 154 1 SLEEP select dtlogin per_processor_selects+0x40
2278 2230 1 12519239 154 1 SLEEP ksleep vxsvc ksleep(0x4290b040)
2405 2230 1 12520740 154 0 SLEEP ksleep vxsvc ksleep(0x4246a780)
2551 2230 1 12520740 154 1 SLEEP ksleep vxsvc ksleep(0x48ca6140)
2550 2230 1 12520742 152 0 ZOMB lwp_detached_e vxsvc
2452 2377 1 12521119 154 0 SLEEP ksleep samd ksleep(0x4246a740)
2498 2377 1 12521119 154 1 SLEEP ksleep samd ksleep(0x4290b180)
2499 2377 1 12521119 154 1 SLEEP ksleep samd ksleep(0x4290b300)
2500 2377 1 12521119 154 0 SLEEP ksleep samd ksleep(0x4290b340)
2501 2377 1 12521119 154 1 SLEEP ksleep samd ksleep(0x4290b380)
2502 2377 1 12521119 154 0 SLEEP ksleep samd ksleep(0x4290b3c0)
2503 2377 1 12521119 154 1 SLEEP ksleep samd ksleep(0x4290b400)
2504 2377 1 12521119 154 0 SLEEP ksleep samd ksleep(0x4290b440)
2505 2377 1 12521119 154 1 SLEEP ksleep samd ksleep(0x4290b480)
2506 2377 1 12521119 154 0 SLEEP ksleep samd ksleep(0x4290b4c0)
2507 2377 1 12521119 154 1 SLEEP ksleep samd ksleep(0x4290b500)
2508 2377 1 12521119 154 1 SLEEP ksleep samd ksleep(0x4290b540)
2509 2377 1 12521119 154 1 SLEEP ksleep samd ksleep(0x4290b580)
2510 2377 1 12521119 154 0 SLEEP ksleep samd ksleep(0x4290b5c0)
2511 2377 1 12521119 154 1 SLEEP ksleep samd ksleep(0x4290b600)
2512 2377 1 12521119 154 0 SLEEP ksleep samd ksleep(0x4290b640)
2513 2377 1 12521119 154 1 SLEEP ksleep samd ksleep(0x4290b680)
2514 2377 1 12521119 154 0 SLEEP ksleep samd ksleep(0x4290b6c0)
2515 2377 1 12521119 154 1 SLEEP ksleep samd ksleep(0x4290b700)
2516 2377 1 12521119 154 0 SLEEP ksleep samd ksleep(0x4290b740)
2517 2377 1 12521119 154 0 SLEEP ksleep samd ksleep(0x4290b780)
2304 2253 1 12521457 154 1 SLEEP poll rpc.mountd per_processor_selects+0x40
2429 2376 1 12521464 154 1 SLEEP ioctl sfd sfd_sleeping
2423 2370 1 12521474 158 1 SLEEP waitpid dtrc wait1(0x48b2d040)
2316 2265 1 12521653 154 1 SLEEP nfssvc nfsd svc_run(0x4028e040)
2331 2280 2265 12521653 154 1 SLEEP nfssvc nfsd svc_run(0x4028e040)
2332 2281 2265 12521653 154 0 SLEEP nfssvc nfsd svc_run(0x4028e040)
2333 2282 2265 12521653 154 1 SLEEP nfssvc nfsd svc_run(0x4028e040)
2335 2284 2265 12521653 154 1 SLEEP nfssvc nfsd svc_run(0x4028e040)
2317 2266 2265 12521654 154 1 SLEEP nfssvc nfsd svc_run(0x4028e080)
2318 2267 2266 12521654 154 0 SLEEP nfssvc nfsd svc_run(0x4028e080)
2321 2270 2266 12521654 154 1 SLEEP nfssvc nfsd svc_run(0x4028e080)
2322 2271 2265 12521654 154 0 SLEEP nfssvc nfsd svc_run(0x4028e040)
2323 2272 2266 12521654 154 1 SLEEP nfssvc nfsd svc_run(0x4028e080)
2324 2273 2265 12521654 154 0 SLEEP nfssvc nfsd svc_run(0x4028e040)
2325 2274 2266 12521654 154 1 SLEEP nfssvc nfsd svc_run(0x4028e080)
2327 2276 2266 12521654 154 1 SLEEP nfssvc nfsd svc_run(0x4028e080)
2329 2278 2266 12521654 154 1 SLEEP nfssvc nfsd svc_run(0x4028e080)
1080 1052 1 12521655 154 1 SLEEP poll rpcbind per_processor_selects+0x40
2319 2268 2265 12521655 154 1 SLEEP nfssvc nfsd svc_run(0x4028e040)
2320 2269 2266 12521655 154 0 SLEEP nfssvc nfsd svc_run(0x4028e080)
2315 2264 1 12521656 154 1 SLEEP poll nfsd per_processor_selects+0x40
2299 2230 1 12521663 154 1 SLEEP ksleep vxsvc ksleep(0x4246a680)
2279 2230 1 12521672 154 1 SLEEP poll vxsvc per_processor_selects+0x40
1159 1131 1 12521993 154 0 SLEEP poll rpc.lockd per_processor_selects
1574 1541 1 12524559 154 1 SLEEP select snmpdm per_processor_selects+0x40
2049 2001 1 12524559 154 1 SLEEP select emsagent per_processor_selects+0x40
1907 1870 1 12524899 154 0 SLEEP read envd FIFO 0x423b0340
1845 13 0 12524942 148 1 ZOMB n/a ioconfigd
1871 1834 1 12524952 154 1 SLEEP read cron FIFO 0x42540040
13 13 0 12525385 152 1 SLEEP n/a ioconfigd SD$#var#tmp#ucAAAAAAa26803
1684 1651 1 12525903 154 0 SLEEP select fddi4subagt per_processor_selects
1632 1599 1 12526015 154 0 SLEEP select trapdestagt per_processor_selects
1602 1569 1 12526040 154 1 SLEEP select hp_unixagt per_processor_selects+0x40
1153 1125 1 12526998 154 1 SLEEP poll rpc.statd per_processor_selects+0x40
1145 1117 1 12527120 154 1 SLEEP async_daemon biod async_bufhead
1130 1102 1 12527121 154 0 SLEEP async_daemon biod async_bufhead
1131 1103 1 12527121 154 1 SLEEP async_daemon biod async_bufhead
1132 1104 1 12527121 154 0 SLEEP async_daemon biod async_bufhead
1133 1105 1 12527121 154 1 SLEEP async_daemon biod async_bufhead
1134 1106 1 12527121 154 0 SLEEP async_daemon biod async_bufhead
1135 1107 1 12527121 154 1 SLEEP async_daemon biod async_bufhead
1136 1108 1 12527121 154 0 SLEEP async_daemon biod async_bufhead
1137 1109 1 12527121 154 1 SLEEP async_daemon biod async_bufhead
1138 1110 1 12527121 154 0 SLEEP async_daemon biod async_bufhead
1139 1111 1 12527121 154 1 SLEEP async_daemon biod async_bufhead
1140 1112 1 12527121 154 0 SLEEP async_daemon biod async_bufhead
1141 1113 1 12527121 154 1 SLEEP async_daemon biod async_bufhead
1142 1114 1 12527121 154 0 SLEEP async_daemon biod async_bufhead
1143 1115 1 12527121 154 1 SLEEP async_daemon biod async_bufhead
1144 1116 1 12527121 154 0 SLEEP async_daemon biod async_bufhead
1085 1057 0 12527136 152 0 SLEEP n/a nfskd nfs_srvr_debug+0x10
917 889 1 12528252 127 1 SLEEP semop ntl_reader semop(0x428e9d2c)
907 879 1 12528479 127 1 SLEEP read nktl_daemon netdiag_ques+0x44
887 860 1 12528541 154 0 SLEEP ioctl hotplugd BS$#var#tmp#ucAAAAAAa19538
867 840 1 12528576 155 0 SLEEP msgrcv ptydaemon msgque[0]+0x5c
30 30 0 12536188 148 1 SLEEP n/a lvmdevd lv_dev_daemon(0x403fb1e4)
12 12 0 12536462 -32 0 RUN n/a ttisr
25 25 0 12536463 148 0 SLEEP n/a lvmschedd lv_pbuf_pending_Q
28 28 0 12536463 100 1 SLEEP n/a sblksched streams_blk_sync
29 29 0 12536463 100 0 SLEEP n/a sblksched streams_blk_sync
31 31 0 12536463 148 0 SLEEP n/a lvmattachd lv_attach_daemon(0x403fa0a8)
8 8 0 12536539 100 0 SLEEP n/a supsched streams_up_runq
9 9 0 12536539 100 0 SLEEP n/a strmem __preinit_end+0x698
10 10 0 12536539 100 0 SLEEP n/a strweld weldq_runq
11 11 0 12536539 100 0 SLEEP n/a strfreebd str_freeb_idle

==================
= Kernel Patches =
==================

PHKL_22874 PHKL_22987 PHKL_22994 PHKL_23196 PHKL_23225
PHKL_23250 PHKL_23335 PHKL_23400 PHKL_23401 PHKL_23445
PHKL_23642 PHKL_24026 PHKL_24046 PHKL_24047 PHKL_24056
PHKL_24188 PHKL_24253 PHKL_24278 PHKL_24343 PHKL_24408
PHKL_24554 PHKL_24555 PHKL_24652 PHKL_24730 PHKL_24751
PHKL_25022 PHKL_25123 PHKL_25166 PHKL_25212 PHKL_25238
PHKL_25304 PHKL_25368 PHKL_25375 PHKL_25468 PHKL_25493
PHKL_25552 PHKL_25582 PHKL_25593 PHKL_25602 PHKL_25610
PHKL_25652 PHKL_25728 PHKL_25760 PHKL_25885 PHKL_25886
PHKL_25994 PHKL_26032 PHKL_26074 PHKL_26104 PHKL_26183
PHKL_26196 PHKL_26425 PHKL_26705 PHKL_26719 PHKL_26755
PHKL_26785 PHKL_26833 PHKL_26834 PHKL_27025 PHKL_27155
PHKL_27172 PHKL_27179 PHKL_27219 PHKL_27321 PHKL_27498
PHKL_27502 PHKL_27532 PHKL_27682 PHKL_27688 PHKL_27734
PHKL_27737 PHKL_27757 PHKL_27825 PHKL_27918 PHKL_27971
PHKL_28062 PHKL_28114 PHKL_28117 PHKL_28122 PHKL_28139
PHKL_28185 PHKL_28264 PHKL_28267 PHKL_28323 PHKL_28326
PHKL_28384 PHKL_28459 PHKL_28474 PHKL_28475 PHKL_28489
PHKL_28512 PHKL_28569 PHKL_28695 PHKL_28794 PHKL_28839
PHKL_28842 PHKL_28984 PHKL_28990 PHKL_29045 PHKL_29115
PHKL_29300 PHKL_29335 PHKL_29527 PHKL_29529 PHKL_29567
PHKL_29611 PHKL_29650 PHKL_29651 PHKL_29696 PHKL_29704
PHKL_29818 PHKL_29826 PHKL_29911 PHKL_29921 PHKL_29981
PHKL_30034 PHKL_30044 PHKL_30105 PHKL_30151 PHKL_30216
PHKL_30218 PHKL_30219 PHKL_30257 PHKL_30259 PHKL_30317
PHKL_30366 PHKL_30398 PHKL_30446 PHKL_30449 PHKL_30472
PHKL_30516 PHKL_30523 PHKL_30539 PHKL_30542 PHKL_30557
PHKL_30579 PHKL_30582 PHKL_30590 PHKL_30596 PHKL_30606
PHKL_30621 PHKL_30651 PHKL_30681 PHKL_30756 PHKL_30796
PHKL_30837 PHKL_30904 PHKL_30992 PHKL_31003 PHKL_31036
PHKL_31212 PHKL_31227 PHKL_31239 PHKL_31241 PHKL_31876
PHKL_31905 PHKL_31993 PHKL_31994 PHKL_32002 PHKL_32005
PHKL_32012 PHKL_32035 PHKL_32037 PHKL_32038 PHKL_32039
PHKL_32061 PHKL_32090 PHKL_32119 PHKL_32126 PHKL_32235
PHKL_32375 PHKL_32457 PHKL_32578 PHKL_32603 PHKL_32619
PHKL_32647 PHKL_32668 PHKL_32669 PHKL_32715 PHKL_32927
PHKL_33066 PHKL_33137 PHKL_33200 PHKL_33258 PHKL_33328
PHKL_33336 PHKL_33338 PHKL_33339 PHKL_33356 PHKL_33367
PHKL_33369 PHKL_33371 PHKL_33372 PHKL_33377 PHKL_33387
PHKL_33390 PHKL_33408 PHKL_33436 PHKL_33590 PHKL_33702
PHKL_33712 PHKL_33843 PHKL_33844 PHKL_33858 PHKL_33988
PHKL_34024 PHKL_34039 PHKL_34100 PHKL_34105 PHKL_34106
PHKL_34110 PHKL_34116 PHKL_34153 PHKL_34161 PHKL_34167
PHKL_34173 PHKL_34187 PHKL_34193 PHKL_34254 PHKL_34296
PHKL_34309 PHKL_34310 PHKL_34311 PHKL_34459 PHKL_34466
PHKL_34478 PHKL_34534 PHKL_34548 PHKL_34552 PHKL_34595
PHKL_34665 PHKL_35037 PHNE_22878 PHNE_23502 PHNE_23594
PHNE_24035 PHNE_24211 PHNE_24492 PHNE_24910 PHNE_25083
PHNE_25627 PHNE_25644 PHNE_27170 PHNE_27218 PHNE_28137
PHNE_28444 PHNE_28923 PHNE_29320 PHNE_29825 PHNE_29883
PHNE_30086 PHNE_30103 PHNE_30378 PHNE_30463 PHNE_30661
PHNE_31091 PHNE_31097 PHNE_31929 PHNE_32396 PHNE_32477
PHNE_32616 PHNE_33138 PHNE_33313 PHNE_33481 PHNE_33498
PHNE_33689 PHNE_33704 PHNE_33729 PHNE_33971 PHNE_34131
PHNE_34293 PHNE_34427 PHNE_34662 PHNE_35183


=============
= Signature =
=============

11.11 and panic and "Data page fault" and arena and freelist and corruption and idle+* and swidle+*



who tell me what problem?

[TY 007]

Hello ajiang,

1. Install Latest ARPA + LVM Kernel Patch:
PHNE_35351 s700_800 11.11 cumulative ARPA Transport patch
PHKL_35970 s700_800 11.11 LVM Cumulative Patch

2. Log a HP Hardware Support Ticket - Possible Bad CPU(s)

Thanks

[melvyn burnard]

and in future do NOT paste all the information into your thread, attach a file containing it instead