topic Re: System Hangs in Operating System - HP-UX

System Hangs

SCSI Error — Wed, 11 Jul 2007 19:45:18 GMT

Hi, Kindly help me interpret the logs I got. Do I have a hardware problem? The system hangs at night and manually rebooted at around 7AM

Logs on service process below:

Log Entry # 0 :
SYSTEM NAME: system
DATE: 07/11/2007 TIME: 21:04:11
ALERT LEVEL: 3 = System blocked waiting for operator input

SOURCE: 1 = processor
SOURCE DETAIL: 1 = processor general SOURCE ID: 0
PROBLEM DETAIL: 0 = no problem detail

CALLER ACTIVITY: E = system warning STATUS: 0
CALLER SUBACTIVITY: F0 = implementation dependent
REPORTING ENTITY TYPE: E = HP-UX REPORTING ENTITY ID: 04

0xF8E040301100EF00 00000000 0000EF00 type 31 = legacy PA HEX chassis-code
0x58E048301100EF00 00006B06 0B15040B type 11 = Timestamp 07/11/2007 21:04:11
Type CR for next entry, Q CR to quit.

Log Entry # 1 :
SYSTEM NAME: system
DATE: 07/11/2007 TIME: 20:57:22
ALERT LEVEL: 3 = System blocked waiting for operator input

SOURCE: 1 = processor
SOURCE DETAIL: 1 = processor general SOURCE ID: 0
PROBLEM DETAIL: 0 = no problem detail

CALLER ACTIVITY: E = system warning STATUS: 0
CALLER SUBACTIVITY: 00 = implementation dependent
REPORTING ENTITY TYPE: E = HP-UX REPORTING ENTITY ID: 04

0xF8E040301100E000 00000000 0000E000 type 31 = legacy PA HEX chassis-code
0x58E048301100E000 00006B06 0B143916 type 11 = Timestamp 07/11/2007 20:57:22
Type CR for next entry, - CR for previous entry, Q CR to quit.

Log Entry # 2 :
SYSTEM NAME: system
DATE: 07/11/2007 TIME: 20:57:15
ALERT LEVEL: 2 = Non-Urgent operator attention required

SOURCE: 8 = I/O
SOURCE DETAIL: 2 = system bus adapter SOURCE ID: 1C
PROBLEM DETAIL: 3 = function failure

CALLER ACTIVITY: 6 = machine check STATUS: 3
CALLER SUBACTIVITY: 34 = implementation dependent
REPORTING ENTITY TYPE: 0 = system firmware REPORTING ENTITY ID: 04

0x70004023821C6343 00000000 00070200 type 14 = Problem Detail
0x58004823821C6343 00006B06 0B14390F type 11 = Timestamp 07/11/2007 20:57:15
Type CR for next entry, - CR for previous entry, Q CR to quit.

Log Entry # 3 :
SYSTEM NAME: system
DATE: 07/11/2007 TIME: 20:57:15
ALERT LEVEL: 2 = Non-Urgent operator attention required

SOURCE: 8 = I/O
SOURCE DETAIL: 2 = system bus adapter SOURCE ID: 1A
PROBLEM DETAIL: 3 = function failure

CALLER ACTIVITY: 6 = machine check STATUS: 3
CALLER SUBACTIVITY: 34 = implementation dependent
REPORTING ENTITY TYPE: 0 = system firmware REPORTING ENTITY ID: 04

0x70004023821A6343 00000000 00070200 type 14 = Problem Detail
0x58004823821A6343 00006B06 0B14390F type 11 = Timestamp 07/11/2007 20:57:15
Type CR for next entry, - CR for previous entry, Q CR to quit.

Log Entry # 38 :
SYSTEM NAME: system
DATE: 07/11/2007 TIME: 20:57:02
ALERT LEVEL: 2 = Non-Urgent operator attention required

SOURCE: 1 = processor
SOURCE DETAIL: 3 = bus translator SOURCE ID: 7
PROBLEM DETAIL: 0 = no problem detail

CALLER ACTIVITY: 6 = machine check STATUS: 2
CALLER SUBACTIVITY: 06 = implementation dependent
REPORTING ENTITY TYPE: 0 = system firmware REPORTING ENTITY ID: 04

0x6000402013076062 00000000 00080004 type 12 = Device Status
0x5800482013076062 00006B06 0B143902 type 11 = Timestamp 07/11/2007 20:57:02
Type CR for next entry, - CR for previous entry, Q CR to quit.

Log Entry # 57 :
SYSTEM NAME: system
DATE: 07/11/2007 TIME: 20:57:01
ALERT LEVEL: 2 = Non-Urgent operator attention required

SOURCE: 1 = processor
SOURCE DETAIL: 1 = processor general SOURCE ID: 2
PROBLEM DETAIL: 0 = no problem detail

CALLER ACTIVITY: 6 = machine check STATUS: 2
CALLER SUBACTIVITY: 32 = implementation dependent
REPORTING ENTITY TYPE: 0 = system firmware REPORTING ENTITY ID: 02

0x1800202011026322 CB818000 00000000 type 3 = Actual Data
0x5800282011026322 00006B06 0B143901 type 11 = Timestamp 07/11/2007 20:57:01
Type CR for next entry, - CR for previous entry, Q CR to quit.

Log Entry # 60 :
SYSTEM NAME: system
DATE: 07/11/2007 TIME: 20:57:01
ALERT LEVEL: 2 = Non-Urgent operator attention required

SOURCE: 1 = processor
SOURCE DETAIL: 1 = processor general SOURCE ID: 0
PROBLEM DETAIL: 0 = no problem detail

CALLER ACTIVITY: 6 = machine check STATUS: 2
CALLER SUBACTIVITY: 32 = implementation dependent
REPORTING ENTITY TYPE: 0 = system firmware REPORTING ENTITY ID: 00

0x1800002011006322 CB818000 00000000 type 3 = Actual Data
0x5800082011006322 00006B06 0B143901 type 11 = Timestamp 07/11/2007 20:57:01
Type CR for next entry, - CR for previous entry, Q CR to quit.

Log Entry # 75 :
SYSTEM NAME: system
DATE: 07/11/2007 TIME: 20:57:01
ALERT LEVEL: 2 = Non-Urgent operator attention required

SOURCE: 0 = unknown, no source stated
SOURCE DETAIL: 0 = unknown, no source stated SOURCE ID: FF
PROBLEM DETAIL: 0 = no problem detail

CALLER ACTIVITY: 6 = machine check STATUS: 2
CALLER SUBACTIVITY: 24 = implementation dependent
REPORTING ENTITY TYPE: 0 = system firmware REPORTING ENTITY ID: 04

0x0000402000FF6242 00000000 00000000 type 0 = Data Field Unused
0x5800482000FF6242 00006B06 0B143901 type 11 = Timestamp 07/11/2007 20:57:01

# last | more

root pts/ta Thu Jul 12 07:59 still logged in
reboot system boot Thu Jul 12 07:11 still logged in
user pts/tb Wed Jul 11 12:26 - 12:49 (00:23)
root pts/ta Wed Jul 11 11:42 - 14:18 (02:35)
root ftp Wed Jul 11 09:44 - 09:45 (00:00)
root ftp Wed Jul 11 09:44 - 09:44 (00:00)
root ftp Wed Jul 11 09:43 - 09:44 (00:00)
root ftp Wed Jul 11 09:43 - 09:45 (00:01)
root ftp Wed Jul 11 09:43 - 09:43 (00:00)
root ftp Wed Jul 11 09:42 - 09:43 (00:00)
root ftp Wed Jul 11 09:41 - 09:41 (00:00)
root pts/ta Wed Jul 11 09:40 - 10:28 (00:47)
root ftp Tue Jul 10 16:59 - 17:00 (00:00)
root ftp Tue Jul 10 16:58 - 17:00 (00:02)
root ftp Tue Jul 10 16:57 - 16:57 (00:00)
root ftp Tue Jul 10 16:56 - 16:56 (00:00)
root ftp Tue Jul 10 16:56 - 16:56 (00:00)
root ftp Tue Jul 10 16:56 - 16:57 (00:01)
root ftp Tue Jul 10 16:56 - 16:56 (00:00)
$ model
9000/800/N4000-55
$

Re: System Hangs

skt_skt — Wed, 11 Jul 2007 20:14:37 GMT

below is what is interesting to me

SOURCE: 8 = I/O
SOURCE DETAIL: 2 = system bus adapter SOURCE ID: 1C
PROBLEM DETAIL: 3 = function failure

Did you check the system CPU/MEM/SWAP/NETWORK utilization at the time the system was crashed.(Perfview/pv is good tool).

what is reported in /etc/shutdownlog.Take a close look on OLDsyslog.log

I would say do a "savecrah -vr DIR" and create the crash dump and at the first check the details in INDEX file.Do a Q 4 analysis and that will help to determine if it is h/w or s/w issue.

what is the OS version?is it part of any cluster?

Re: System Hangs

SCSI Error — Wed, 11 Jul 2007 20:42:23 GMT

Hi,

It is standalone and not part of any cluster.

Below are the infos I got as you'v requested.

Thanks

$ cat /etc/shutdownlog | tail -1
07:19 Thu Jul 12 2007. Reboot after panic: , isr.ior = 0'103401e1.c0000000'4e890aa8
$ model
9000/800/N4000-55
$ uname -a
HP-UX server B.11.11 U 9000/800 610309383 unlimited-user license
$
$ ls -lrt
total 1447712
-rw-r--r-- 1 root root 268378112 Jul 12 07:13 image.1.1
-rw-r--r-- 1 root root 268406784 Jul 12 07:14 image.1.2
-rw-r--r-- 1 root root 14135200 Jul 12 07:15 image.1.3.gz
-rw-r--r-- 1 root root 42654243 Jul 12 07:16 image.2.1.gz
-rw-r--r-- 1 root root 50821586 Jul 12 07:17 image.2.2.gz
-rw-r--r-- 1 root root 34471717 Jul 12 07:18 image.2.3.gz
-rw-r--r-- 1 root root 43283294 Jul 12 07:19 image.3.1.gz
-rw-r--r-- 1 root root 1297 Jul 12 07:19 INDEX
-rw-r--r-- 1 root root 18972616 Jul 12 09:55 vmunix
-rw-rw-rw- 1 root sys 43120 Jul 12 09:59 crash_report
-rw-rw-rw- 1 root sys 6815 Jul 12 10:00 whathappened
$ pwd
/var/adm/crash/crash.37
$ cat INDEX
comment savecrash crash dump INDEX file
version 2
hostname tyco
modelname 9000/800/N4000-55
panic , isr.ior = 0'103401e1.c0000000'4e890aa8
dumptime 1184187420 Thu Jul 12 06:57:00 EST 2007
savetime 1184188311 Thu Jul 12 07:11:51 EST 2007
release @(#) $Revision: vmunix: vw: -proj selectors: CUPI80_BL2000_1108 -c 'Vw for CUPI80_BL2000_1108 build' -- cupi80_bl2000_1108 'CUPI80_BL2000_1108' Wed Nov 8 19:24:56 PST 2000 $
memsize 8589934592
chunksize 268435456
module /stand/vmunix vmunix 29219096 3522642218
image image.1.1 0x0000000000000000 0x000000000fff2000 0x0000000000000000 0x0000000000044d97 1669995390
image image.1.2 0x0000000000000000 0x000000000fff9000 0x0000000000044d98 0x0000000000054d8f 3164530861
image image.1.3 0x0000000000000000 0x00000000040ae000 0x0000000000054d90 0x000000000007ffff 932365382
image image.2.1 0x0000000000000000 0x000000000fff5000 0x0000000000100000 0x0000000000144a3f 2415394606
image image.2.2 0x0000000000000000 0x000000000fff4000 0x0000000000144a40 0x00000000001a0277 451758399
image image.2.3 0x0000000000000000 0x000000000cfa3000 0x00000000001a0278 0x00000000001fffff 27148689
image image.3.1 0x0000000000000000 0x000000000f7ec000 0x0000000000280000 0x00000000002fffff 3065450762
$

Re: System Hangs

skt_skt — Wed, 11 Jul 2007 20:51:48 GMT

What is the history of this server interms of panic or crashing?

Could you sen the output of "ls -lrt /var/adm/crash" and the full output of shutdownlog

Re: System Hangs

SCSI Error — Wed, 11 Jul 2007 20:58:38 GMT

$ cat /etc/shutdownlog
14:54 Tue May 31, 2005. Halt: (by SAM)
14:55 Tue May 31, 2005. Halt: (by hostname!root)
16:40 Tue May 31, 2005. Reboot: (by hostname!root)
09:40 Wed Jun 1, 2005. Halt: (by hostname!root)
11:08 Wed Jun 1, 2005. Reboot: (by hostname!root)
16:49 Wed Jun 1, 2005. Reboot:
11:30 Thu Jun 2, 2005. Halt: (by hostname!root)
18:32 Thu Jun 2, 2005. Reboot: (by hostname!root)
15:01 Wed Jun 22, 2005. Reboot: (by hostname!root)
22:25 Wed Jun 22, 2005. Reboot: (by hostname!root)
00:54 Thu Jun 23, 2005. Reboot: (by hostname!root)
04:31 Thu Jun 23, 2005. Reboot:
12:00 Fri Jun 24, 2005. Halt: (by hostname!root)
13:38 Fri Jun 24, 2005. Reboot: (by hostname!root)
14:24 Fri Jun 24, 2005. Reboot: (by hostname!root)
14:36 Fri Jun 24, 2005. Reboot: (by hostname!root)
14:56 Fri Jun 24, 2005. Reboot: (by hostname!root)
15:35 Fri Jun 24, 2005. Reboot: (by hostname!root)
15:47 Fri Jun 24, 2005. Reboot: (by hostname!root)
16:16 Fri Jun 24, 2005. Reboot: (by hostname!root)
16:35 Fri Jun 24, 2005. Halt: (by hostname!root)
17:13 Fri Jun 24, 2005. Halt: (by hostname!root)
19:10 Fri Jun 24, 2005. Reboot: (by hostname!root)
20:31 Fri Jun 24, 2005. Reboot: (by hostname!root)
15:44 Mon Jun 27, 2005. Reboot: (by hostname!root)
16:17 Mon Jun 27, 2005. Reboot: (by hostname!root)
17:11 Mon Jun 27, 2005. Reboot: (by SAM)
17:12 Mon Jun 27, 2005. Reboot: (by hostname!root)
20:32 Wed Jul 6, 2005. Halt: (by hostname!root)
13:19 Sat Jul 9, 2005. Halt: (by hostname!root)
14:00 Sat Jul 9, 2005. Reboot: (by hostname!root)
11:01 Thu Jul 14, 2005. Halt: (by hostname!root)
12:01 Thu Jul 14, 2005. Reboot: (by hostname!root)
22:45 Wed Jul 13, 2005. Reboot:
12:54 Thu Jul 14 2005. Reboot after panic: , isr.ior = 0'18240a2d.40000000'240deec
14:18 Thu Jul 14, 2005. Reboot: (by hostname!root)
11:48 Tue Jul 19, 2005. Reboot: (by hostname!root)
16:13 Tue Jul 19, 2005. Halt: (by hostname!root)
22:41 Fri Sep 16, 2005. Reboot: (by hostname!root)
22:15 Wed Oct 12, 2005. Reboot: (by SAM)
22:15 Wed Oct 12, 2005. Reboot: (by hostname!root)
02:17 Tue Oct 18, 2005. Reboot: (by hostname!root)
04:53 Tue Oct 18, 2005. Reboot: (by hostname!root)
00:39 Sun Oct 23, 2005. Reboot: (by hostname!root)
02:45 Sat Nov 5, 2005. Reboot: (by hostname!root)
18:05 Thu Nov 10, 2005. Reboot: (by SAM)
18:06 Thu Nov 10, 2005. Reboot: (by hostname!root)
21:14 Wed Nov 16, 2005. Reboot: (by hostname!root)
10:04 Sat Nov 19, 2005. Reboot: (by hostname!root)
10:19 Sat Nov 19, 2005. Reboot: (by hostname!root)
21:29 Wed Nov 23, 2005. Reboot: (by hostname!root)
22:01 Sat Dec 3, 2005. Reboot: (by hostname!root)
02:17 Tue Dec 6, 2005. Reboot: (by hostname!root)
07:40 Sun Dec 11, 2005. Reboot: (by hostname!root)
08:24 Tue Dec 13, 2005. Reboot: (by hostname!root)
21:50 Mon Dec 19, 2005. Reboot: (by hostname!root)
22:32 Wed Jan 11, 2006. Reboot: (by hostname!root)
18:05 Tue Jan 17, 2006. Halt: (by hostname!root)
19:56 Tue Jan 17, 2006. Reboot: (by hostname!root)
10:17 Sat Jan 21, 2006. Reboot: (by hostname!root)
11:28 Sat Jan 21, 2006. Reboot: (by hostname!root)
17:00 Sat Jan 21, 2006. Reboot: (by hostname!root)
11:53 Wed Jan 25, 2006. Reboot: (by hostname!root)
10:44 Sat Jan 28, 2006. Reboot: (by SAM)
10:45 Sat Jan 28, 2006. Reboot: (by hostname!root)
10:56 Sat Jan 28, 2006. Reboot: (by SAM)
10:57 Sat Jan 28, 2006. Reboot: (by hostname!root)
12:24 Sat Jan 28, 2006. Reboot: (by SAM)
12:25 Sat Jan 28, 2006. Reboot: (by hostname!root)
12:48 Sat Jan 28, 2006. Reboot: (by SAM)
12:49 Sat Jan 28, 2006. Reboot: (by hostname!root)
18:12 Sun Jan 29, 2006. Reboot: (by hostname!root)
10:27 Wed Feb 22, 2006. Reboot: (by hostname!root)
05:58 Fri Mar 3, 2006. Reboot: (by hostname!root)
22:39 Tue Mar 7, 2006. Reboot:
21:35 Wed Mar 8, 2006. Reboot: (by hostname!root)
05:53 Thu Mar 9, 2006. Reboot: (by hostname!root)
06:46 Thu Mar 9, 2006. Reboot: (by hostname!root)
20:50 Sat Mar 11, 2006. Reboot: (by hostname!root)
04:22 Wed Mar 15, 2006. Reboot: (by hostname!root)
22:58 Thu Mar 23, 2006. Reboot: (by hostname!root)
03:30 Sat Mar 25, 2006. Reboot: (by hostname!root)
07:08 Sat Mar 25, 2006. Reboot: (by hostname!root)
23:35 Sun Mar 26, 2006. Reboot: (by hostname!root)
22:12 Tue Mar 28, 2006. Reboot: (by SAM)
22:13 Tue Mar 28, 2006. Reboot: (by hostname!root)
21:02 Wed Mar 29, 2006. Reboot: (by hostname!root)
01:36 Fri Mar 31, 2006. Reboot: (by hostname!root)
21:59 Fri Mar 31, 2006. Reboot: (by hostname!root)
22:28 Wed Apr 5, 2006. Halt: (by hostname!root)
18:26 Sun Apr 09 2006. Reboot after panic: , isr.ior = 0'10340007.0'5cd72968
20:50 Mon Apr 10 2006. Reboot after panic: , isr.ior = 0'4340093.0'e43b1120
08:21 Tue Apr 11 2006. Reboot after panic: , isr.ior = 0'41275.80000000'd15aa1b8
08:43 Tue Apr 11 2006. Reboot after panic: , isr.ior = 0'10340007.0'5d574968
10:42 Tue Apr 11 2006. Reboot after panic: , isr.ior = 0'814124b.c0000000'1d86a794
12:33 Tue Apr 11, 2006. Reboot: (by hostname!root)
14:27 Tue Apr 11, 2006. Reboot: (by hostname!root)
22:31 Fri May 5, 2006. Reboot: (by hostname!root)
20:12 Thu May 11, 2006. Reboot: (by hostname!root)
04:57 Fri May 12, 2006. Reboot: (by hostname!root)
19:50 Mon May 15, 2006. Reboot: (by hostname!root)
23:19 Mon May 22, 2006. Reboot: (by hostname!root)
21:20 Mon Jun 12, 2006. Reboot: (by hostname!root)
20:28 Sat Jun 24, 2006. Reboot: (by hostname!root)
23:58 Sat Jun 24, 2006. Reboot: (by hostname!root)
23:02 Sat Jul 1, 2006. Reboot: (by hostname!root)
23:40 Mon Jul 31, 2006. Reboot: (by hostname!root)
20:58 Fri Aug 25, 2006. Reboot: (by hostname!root)
20:55 Sun Aug 27, 2006. Reboot: (by hostname!root)
20:14 Wed Aug 30, 2006. Reboot: (by hostname!root)
21:06 Wed Aug 30, 2006. Reboot: (by hostname!root)
12:30 Thu Aug 31, 2006. Reboot: (by hostname!root)
19:38 Sun Sep 24, 2006. Reboot: (by hostname!root)
19:38 Mon Sep 25, 2006. Reboot:
18:16 Tue Sep 26, 2006. Reboot:
12:18 Wed Sep 27, 2006. Reboot: (by hostname!root)
13:32 Wed Sep 27, 2006. Reboot: (by hostname!root)
01:27 Fri Sep 29 2006. Reboot after panic: , isr.ior = 0'0.0'0
05:34 Fri Sep 29 2006. Reboot after panic: , isr.ior = 0'0.0'0
13:44 Fri Sep 29, 2006. Reboot: (by hostname!root)
19:47 Fri Sep 29 2006. Reboot after panic: , isr.ior = 0'10340007.0'5d575968
23:23 Fri Sep 29 2006. Reboot after panic: , isr.ior = 0'103412a6.80000000'517c07c8
00:12 Sat Sep 30 2006. Reboot after panic: , isr.ior = 0'10340007.0'5c9735a0
02:33 Sat Sep 30 2006. Reboot after panic: , isr.ior = 0'0.0'0
10:36 Sat Sep 30, 2006. Halt: (by hostname!root)
11:25 Sat Sep 30, 2006. Reboot: (by hostname!root)
04:45 Sun Oct 01 2006. Reboot after panic: , isr.ior = 0'10340007.0'5bd6e1d8
06:39 Sun Oct 01 2006. Reboot after panic: , isr.ior = 0'0.0'0
09:26 Sun Oct 01 2006. Reboot after panic: , isr.ior = 0'14011e.0'c6318538
16:48 Sun Oct 1, 2006. Halt: (by hostname!root)
17:26 Sun Oct 1, 2006. Halt: (by hostname!root)
17:53 Sun Oct 1, 2006. Reboot: (by hostname!root)
23:21 Sun Oct 01 2006. Reboot after panic: , isr.ior = 0'10340007.0'5f17d0f8
00:43 Mon Oct 02 2006. Reboot after panic: , isr.ior = 0'10340007.0'5d574968
13:23 Mon Oct 2, 2006. Reboot:
06:48 Tue Oct 3, 2006. Reboot: (by SAM)
06:50 Tue Oct 3, 2006. Reboot: (by hostname!root)
14:46 Tue Oct 03 2006. Reboot after panic: , isr.ior = 0'8340963.80000000'd6454880
07:39 Wed Oct 04 2006. Reboot after panic: , isr.ior = 0'1834086a.40000000'f13f3228
23:29 Wed Oct 4, 2006. Halt: (by hostname!root)
00:05 Thu Oct 5, 2006. Halt: (by hostname!root)
00:15 Thu Oct 5, 2006. Halt: (by hostname!root)
08:36 Fri Oct 6, 2006. Halt: (by hostname!root)
00:23 Fri Oct 06 2006. Reboot (no panic)
07:46 Sat Oct 7, 2006. Reboot:
08:23 Sat Oct 7, 2006. Reboot: (by hostname!root)
08:48 Sat Oct 7, 2006. Reboot:
19:14 Fri Oct 6, 2006. Reboot:
19:37 Fri Oct 6, 2006. Reboot:
10:29 Sat Oct 7, 2006. Reboot: (by hostname!root)
20:39 Fri Oct 6, 2006. Reboot:
22:54 Sat Oct 7, 2006. Reboot:
17:07 Tue Oct 10, 2006. Reboot:
23:13 Wed Oct 11, 2006. Reboot:
17:51 Sat Oct 14, 2006. Reboot:
21:28 Fri Oct 20, 2006. Reboot: (by hostname!root)
19:20 Sat Oct 28 2006. Reboot after panic: , isr.ior = 0'834052f.80000000'a4b24f80
19:46 Sat Oct 28 2006. Reboot after panic: , isr.ior = 0'240003.0'304c0780
21:11 Sat Oct 28 2006. Reboot after panic: , isr.ior = 0'10340011.0'b6adb1c0
21:57 Sat Oct 28 2006. Reboot after panic: , isr.ior = 0'240003.0'304c0780
23:44 Sat Oct 28 2006. Reboot after panic: , isr.ior = 0'4240003.0'304e0798
01:08 Sun Oct 29 2006. Reboot after panic: , isr.ior = 0'240158.0'fc3f10d0
01:20 Sun Oct 29 2006. Reboot after panic: , isr.ior = 0'4340158.0'fc3d10d8
01:32 Sun Oct 29 2006. Reboot after panic: , isr.ior = 0'10240158.0'fc3f10d0
01:44 Sun Oct 29 2006. Reboot after panic: , isr.ior = 0'10340006.0'3e8fb1d8
03:06 Sun Oct 29 2006. Reboot after panic: , isr.ior = 0'10340006.0'3e8fb1d8
12:05 Sun Oct 29 2006. Reboot after panic: , isr.ior = 0'10340006.0'3e8fb1d8
13:05 Sun Oct 29 2006. Reboot after panic: , isr.ior = 0'10340006.0'3e8fb1d8
13:20 Sun Oct 29 2006. Reboot after panic: , isr.ior = 0'0.0'0
13:33 Sun Oct 29 2006. Reboot after panic: , isr.ior = 0'0.0'0
10:03 Mon Oct 30 2006. Reboot after panic: , isr.ior = 0'10340006.0'3e8fb1d8
10:55 Mon Oct 30 2006. Reboot after panic: , isr.ior = 0'10240158.0'fc3f10d0
11:30 Mon Oct 30 2006. Reboot after panic: , isr.ior = 0'10240072.40000000'e6c3acbc
16:12 Mon Oct 30, 2006. Halt: (by hostname!root)
19:29 Wed Nov 22, 2006. Halt: (by hostname!root)
23:08 Sat Jan 6, 2007. Reboot: (by hostname!root)
21:27 Sun Feb 4, 2007. Reboot: (by hostname!root)
04:47 Sun Mar 25, 2007. Reboot: (by hostname!root)
17:10 Tue Apr 24, 2007. Halt: (by hostname!root)
07:19 Thu Jul 12 2007. Reboot after panic: , isr.ior = 0'103401e1.c0000000'4e890aa8
$

$ cd /var/adm/crash
$ ls -lrt
total 128
drwxr-xr-x 2 root root 96 Oct 29 2006 crash.26
drwxr-xr-x 2 root root 96 Oct 29 2006 crash.27
drwxr-xr-x 2 root root 96 Oct 29 2006 crash.28
drwxr-xr-x 2 root root 96 Oct 29 2006 crash.29
drwxr-xr-x 2 root root 96 Oct 29 2006 crash.30
drwxr-xr-x 2 root root 8192 Oct 29 2006 crash.31
drwxr-xr-x 2 root root 8192 Oct 29 2006 crash.32
drwxr-xr-x 2 root root 8192 Oct 29 2006 crash.33
drwxr-xr-x 2 root root 8192 Oct 30 2006 crash.34
drwxr-xr-x 2 root root 8192 Oct 30 2006 crash.35
drwxr-xr-x 2 root root 8192 Oct 30 2006 crash.36
-rwxr-xr-x 1 root root 2 Jul 12 07:11 bounds
drwxr-xr-x 2 root root 8192 Jul 12 10:00 crash.37
$

Re: System Hangs

skt_skt — Wed, 11 Jul 2007 21:36:15 GMT

what is the physical MEM on system
send output of "swapinfo -tam" and "crashconf"

Re: System Hangs

SCSI Error — Wed, 11 Jul 2007 21:47:18 GMT

$ /usr/sbin/swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 3141 955 77% 0 - 1 /dev/vg00/lvol2
dev 80000 3135 76865 4% 0 - 1 /dev/vg7410_1/swap
reserve - 30667 -30667
memory 6327 439 5888 7%
total 90423 37382 53041 41% - 0 -
$

$ /sbin/crashconf
CLASS PAGES INCLUDED IN DUMP DESCRIPTION
-------- ---------- ---------------- -------------------------------------
UNUSED 14458 no, by default unused pages
USERPG 1761485 no, by default user process pages
BCACHE 104560 no, by default buffer cache pages
KCODE 2724 no, by default kernel code pages
USTACK 5209 yes, by default user process stacks
FSDATA 297 yes, by default file system metadata
KDDATA 113682 yes, by default kernel dynamic data
KSDATA 94737 yes, by default kernel static data

Total pages on system: 2097152
Total pages included in dump: 213925

DEVICE OFFSET(kB) SIZE (kB) LOGICAL VOL. NAME
------------ ---------- ---------- ------------ -------------------------
31:0x026000 330592 4194304 64:0x000002 /dev/vg00/lvol2
----------
4194304
$
$

Re: System Hangs

skt_skt — Wed, 11 Jul 2007 22:03:46 GMT

can i guess you have 8GB of physical mem?
send the output of "kmtune|grep -i chunk".

you should process the PV analysis/Q4 as i mentioned earlier and update when you have ready.

Re: System Hangs

SCSI Error — Wed, 11 Jul 2007 22:12:40 GMT

$ q4 -p
@(#) q4 $Revision: 11.X B.11.23d Thu May 6 18:05:11 PST 2003$ 0
Reading kernel symbols ...
Reading data types ...
Initialized PA-RISC 2.0 address translator ...
Initializing stack tracer ...
script /usr/contrib/Q4/lib/q4lib/sample.q4rc.pl
executable /usr/contrib/Q4/bin/perl
version 5.006001
SCRIPT_LIBRARY = /usr/contrib/Q4/lib/q4lib_11.11
perl will try to access scripts from directory
/usr/contrib/Q4/lib/q4lib_11.11

loaded spinlock_depth.pl
run Print_ProcState
q4: (warning) No loadable modules were found
q4: (warning) No loadable modules were found
Try to load dumpdev_t to find dump space configured
Total Dump space CONFIGURED: 4194304 KB 4096 MB
Try to load dumpimage_t to find dump space actually used
Total Dump space USED: 1581116 KB 1544 MB
Okay, Dump space appears to be sufficient : about 2551 MB extra
q4> quit
$ pwd
/var/adm/crash/crash.37
$

$ /usr/sbin/kmtune | grep -i chunk
maxswapchunks 10257 - 10257
swchunk 16384 - 16384
voliomem_chunk_size 65536 - (64*1024)
$

$ cat crash_report
Created with Analyze: @(#) 11.i $Revision: 1.5 $ $Date: 01/07/02 14:43:40 $

HP-UX tyco B.11.11 U 9000/800 610309383

The system had been up for 78.55 days (678689343 ticks).
Load averages: 0.14 0.14 0.14.

System went down at: Thu Jul 12 06:57:00 2007
+--------------------------------------------+
| Message Buffer |
+--------------------------------------------+

sysmap_64bit: rmap ovflo, lost [617031,617032)
sysmap_64bit: rmap ovflo, lost [616656,616657)
sysmap_64bit: rmap ovflo, lost [616171,616172)
sysmap_64bit: rmap ovflo, lost [615993,615994)
sysmap_64bit: rmap ovflo, lost [615649,615650)
sysmap_64bit: rmap ovflo, lost [614371,614372)
sysmap_64bit: rmap ovflo, lost [614141,614142)
sysmap_64bit: rmap ovflo, lost [612527,612528)
sysmap_64bit: rmap ovflo, lost [612149,612150)
sysmap_64bit: rmap ovflo, lost [611016,611017)
sysmap_64bit: rmap ovflo, lost [608080,608081)
sysmap_64bit: rmap ovflo, lost [606235,606236)
sysmap_64bit: rmap ovflo, lost [604316,604317)
sysmap_64bit: rmap ovflo, lost [603218,603219)
sysmap_64bit: rmap ovflo, lost [601436,601437)
sysmap_64bit: rmap ovflo, lost [601010,601011)
sysmap_64bit: rmap ovflo, lost [600660,600661)
sysmap_64bit: rmap ovflo, lost [600002,600003)
sysmap_64bit: rmap ovflo, lost [599781,599782)
sysmap_64bit: rmap ovflo, lost [598627,598628)
sysmap_64bit: rmap ovflo, lost [598458,598459)
sysmap_64bit: rmap ovflo, lost [598451,598452)
sysmap_64bit: rmap ovflo, lost [597856,597857)
sysmap_64bit: rmap ovflo, lost [1822549,1822553)
sysmap_64bit: rmap ovflo, lost [597760,597761)
sysmap_64bit: rmap ovflo, lost [597664,597665)
sysmap_64bit: rmap ovflo, lost [597654,597655)
sysmap_64bit: rmap ovflo, lost [597474,597475)
sysmap_64bit: rmap ovflo, lost [597436,597437)
sysmap_64bit: rmap ovflo, lost [597431,597432)
sysmap_64bit: rmap ovflo, lost [597407,597408)
sysmap_64bit: rmap ovflo, lost [597170,597171)
sysmap_64bit: rmap ovflo, lost [597073,597074)
sysmap_64bit: rmap ovflo, lost [596878,596879)
sysmap_64bit: rmap ovflo, lost [596655,596656)
sysmap_64bit: rmap ovflo, lost [596089,596090)
sysmap_64bit: rmap ovflo, lost [595675,595676)
sysmap_64bit: rmap ovflo, lost [595336,595337)
sysmap_64bit: rmap ovflo, lost [595259,595260)
sysmap_64bit: rmap ovflo, lost [593747,593748)
sysmap_64bit: rmap ovflo, lost [593722,593724)
sysmap_64bit: rmap ovflo, lost [593558,593559)
sysmap_64bit: rmap ovflo, lost [593522,593523)
p ovflo, lost [1824117,1824118)
sysmap_64bit: rmap ovflo, lost [629192,629193)
sysmap_64bit: rmap ovflo, lost [629064,629065)
sysmap_64bit: rmap ovflo, lost [627490,627491)
sysmap_64bit: rmap ovflo, lost [627482,627483)
sysmap_64bit: rmap ovflo, lost [627116,627117)
sysmap_64bit: rmap ovflo, lost [625867,625868)
sysmap_64bit: rmap ovflo, lost [625581,625582)
sysmap_64bit: rmap ovflo, lost [625058,625059)
sysmap_64bit: rmap ovflo, lost [625048,625049)
sysmap_64bit: rmap ovflo, lost [624745,624746)
sysmap_64bit: rmap ovflo, lost [623378,623379)
sysmap_64bit: rmap ovflo, lost [623069,623070)
sysmap_64bit: rmap ovflo, lost [623059,623060)
sysmap_64bit: rmap ovflo, lost [622956,622957)
sysmap_64bit: rmap ovflo, lost [622904,622905)
sysmap_64bit: rmap ovflo, lost [622639,622640)
sysmap_64bit: rmap ovflo, lost [622612,622613)
sysmap_64bit: rmap ovflo, lost [622353,622354)
sysmap_64bit: rmap ovflo, lost [622300,622301)
sysmap_64bit: rmap ovflo, lost [622205,622206)
sysmap_64bit: rmap ovflo, lost [622165,622166)
sysmap_64bit: rmap ovflo, lost [622153,622154)
sysmap_64bit: rmap ovflo, lost [622147,622148)
sysmap_64bit: rmap ovflo, lost [622142,622143)
sysmap_64bit: rmap ovflo, lost [622138,622139)
sysmap_64bit: rmap ovflo, lost [622078,622079)
sysmap_64bit: rmap ovflo, lost [622020,622021)
sysmap_64bit: rmap ovflo, lost [620807,620808)
sysmap_64bit: rmap ovflo, lost [620636,620637)
sysmap_64bit: rmap ovflo, lost [620483,620484)
sysmap_64bit: rmap ovflo, lost [620449,620450)
sysmap_64bit: rmap ovflo, lost [620102,620103)
sysmap_64bit: rmap ovflo, lost [619986,619987)
sysmap_64bit: rmap ovflo, lost [619237,619238)
sysmap_64bit: rmap ovflo, lost [619072,619073)
sysmap_64bit: rmap ovflo, lost [618891,618892)
sysmap_64bit: rmap ovflo, lost [618819,618820)
sysmap_64bit: rmap ovflo, lost [618548,618549)
sysmap_64bit: rmap ovflo, lost [617805,617806)
sysmap_64bit: rmap ovflo, lost [617713,617714)
sysmap_64bit: rmap ovflo, lost [617227,617228)
sysmap_64bit: rmap ovflo, lost [617158,617159)
sysmap_64bit: rmap ovflo, lost [617045,
+--------------------------------------------+
| Addresses |
+--------------------------------------------+

BUFHSZ: 0x00defda0 INOHSZ: 0x00cbf320
UIDHMASK: 0x00cbf3c4 _release_version: 0x00c2f400
base_pdir: 0x00e2baf8 bfreelist: 0x00d3b138
boottime: 0x00fd3c20 bswlist: 0x00e31a28
bufhash: 0x00da3fc0 bufpages: 0x00cbf388
cfree: 0x00df3830 crash_event_table: 0x00b46000
crash_processor_table: 0x00b44000 dbc_hdr: 0x00cc0ab0
desfree: 0x00cbf1d8 end: 0x010001a0
etext: 0x00acb000 file: 0x00e66648
filesys_sema: 0x00bf7280 freemem: 0x00fd9d78
htbl: 0x00e79bb0 ifreeh: 0x00deb5f8
ifreet: 0x00deb600 ihead: 0x00fdf260
inode: 0x00d3ac50 kernvas: 0x00dbb898
lotsfree: 0x00cbf1dc max_pdir: 0x00e57d90
minfree: 0x00cbf1d4 mpproc_info: 0x00cbf600
nbuf: 0x00cbf38c nc_hash: 0x00e12268
nc_hash_size: 0x00cbf348 nc_lru: 0x00ddc7e8
ncache: 0x00de4858 nclist: 0x00cbf364
ncstats: 0x00bc3100 nfile: 0x00cbf35c
nfs_vfsops: 0x00bf1068 nfs_vnodeops: 0x00bf4248
nhtbl: 0x00da6460 ninode: 0x00cbf360
niopdir: 0x00cbf5a4 nmpinfo: 0x00cbf608
npdir: 0x00ddc3d0 nproc: 0x00cbf224
nswbuf: 0x00cc1f68 panic_save_state: 0x00b3f100
panic_second_save_state: 0x00b3f470 pdir: 0x00e9bfe8
pdirhash_type: 0x00cbf4c0 pfdat_ptr: 0x00e2e950
phash: 0x00e6c5f8 phashmask: 0x00e9bd40
phys_mem_pages: 0x00e7ce38 proc_list: 0x00e53e58
kthread_list: 0x00da3cc0 regactive: 0x00e79a18
rootvfs: 0x00e65c70 rpfreelist: 0x00cc1258
rtable: 0x00e580d0 scaled_npdir: 0x00da6488
shmem: 0x00e193b0 shminfo: 0x00afa620
shmmni: 0x00cbf180 swapdev: 0x00cbf094
swapdev_vp: 0x00e890b8 swaptab: 0x00d93428
swbuf: 0x00cc1f60 swdevt: 0x00ae0a48
sysinfo: 0x0041c2d0 ufs_vfsops: 0x00c22088
ufs_vnodeops: 0x00c22dd8 uidhash: 0x00e79210
v: 0x00cbf3d8 physio_buf_list: 0x00e57f90
+--------------------------------------------+
| Crash Processors |
+--------------------------------------------+

Processor 0:
indexof 0 mapped 0x1
spaceof 0 addrof 0xb44000
physaddrof 0xb44000 realmode 0x1
cpt_hpa 0xffffffff'fed21000 cpt_ppdp 0x18fd000
cpt_pdc_vector 0x7fb10fa0 cpt_hpmc_savestate 0xb3fae0
cpt_toc_savestate 0xb3fe50 cpt_hpmc_count 0
cpt_toc_count 0x1 cpt_state_saved 0x1

Processor 1:
indexof 0 mapped 0x1
spaceof 0 addrof 0xb44038
physaddrof 0xb44038 realmode 0x1
cpt_hpa 0xffffffff'fed25000 cpt_ppdp 0x19003c8
cpt_pdc_vector 0x7fb10fa0 cpt_hpmc_savestate 0x18f9000
cpt_toc_savestate 0x18f9370 cpt_hpmc_count 0
cpt_toc_count 0x1 cpt_state_saved 0x1

Processor 2:
indexof 0 mapped 0x1
spaceof 0 addrof 0xb44070
physaddrof 0xb44070 realmode 0x1
cpt_hpa 0xffffffff'fed29000 cpt_ppdp 0x1903790
cpt_pdc_vector 0x7fb10fa0 cpt_hpmc_savestate 0x18f96e0
cpt_toc_savestate 0x18f9a50 cpt_hpmc_count 0
cpt_toc_count 0x1 cpt_state_saved 0x1

Processor 3:
indexof 0 mapped 0x1
spaceof 0 addrof 0xb440a8
physaddrof 0xb440a8 realmode 0x1
cpt_hpa 0xffffffff'fed2d000 cpt_ppdp 0x1906b58
cpt_pdc_vector 0x7fb10fa0 cpt_hpmc_savestate 0x18f9dc0
cpt_toc_savestate 0x18fa130 cpt_hpmc_count 0
cpt_toc_count 0x1 cpt_state_saved 0x1

Processor 4:
indexof 0 mapped 0x1
spaceof 0 addrof 0xb440e0
physaddrof 0xb440e0 realmode 0x1
cpt_hpa 0xffffffff'fed61000 cpt_ppdp 0x1909f20
cpt_pdc_vector 0x7fb10fa0 cpt_hpmc_savestate 0x18fa4a0
cpt_toc_savestate 0x18fa810 cpt_hpmc_count 0x1
cpt_toc_count 0 cpt_state_saved 0x1

Processor 5:
indexof 0 mapped 0x1
spaceof 0 addrof 0xb44118
physaddrof 0xb44118 realmode 0x1
cpt_hpa 0xffffffff'fed65000 cpt_ppdp 0x190d2e8
cpt_pdc_vector 0x7fb10fa0 cpt_hpmc_savestate 0x18fab80
cpt_toc_savestate 0x18faef0 cpt_hpmc_count 0
cpt_toc_count 0x1 cpt_state_saved 0x1

Processor 6:
indexof 0 mapped 0x1
spaceof 0 addrof 0xb44150
physaddrof 0xb44150 realmode 0x1
cpt_hpa 0xffffffff'fed69000 cpt_ppdp 0x19106b0
cpt_pdc_vector 0x7fb10fa0 cpt_hpmc_savestate 0x18fb260
cpt_toc_savestate 0x18fb5d0 cpt_hpmc_count 0
cpt_toc_count 0x1 cpt_state_saved 0x1

Processor 7:
indexof 0 mapped 0x1
spaceof 0 addrof 0xb44188
physaddrof 0xb44188 realmode 0x1
cpt_hpa 0xffffffff'fed6d000 cpt_ppdp 0x1913a78
cpt_pdc_vector 0x7fb10fa0 cpt_hpmc_savestate 0x18fb940
cpt_toc_savestate 0x18fbcb0 cpt_hpmc_count 0
cpt_toc_count 0x1 cpt_state_saved 0x1
+--------------------------------------------+
| Crash Events |
+--------------------------------------------+

Crash Event 0 (HPMC, struct crash_event_table_struct at 0xb46000):
indexof 0 mapped 0x1
spaceof 0 addrof 0xb46000
physaddrof 0xb46000 realmode 0x1
cet_reserved 0 cet_hpa 0xffffffff'fed61000
cet_event 0x3 cet_savestate 0x18fa4a0
cet_start_itmr 0xd431d'8fdf7c04 cet_cur_itmr 0

Crash Event 1 (TOC, struct crash_event_table_struct at 0xb46030):
indexof 0 mapped 0x1
spaceof 0 addrof 0xb46030
physaddrof 0xb46030 realmode 0x1
cet_reserved 0 cet_hpa 0xffffffff'fed65000
cet_event 0x2 cet_savestate 0x18faef0
cet_start_itmr 0xd431d'cec96c42 cet_cur_itmr 0x1

Crash Event 2 (TOC, struct crash_event_table_struct at 0xb46060):
indexof 0 mapped 0x1
spaceof 0 addrof 0xb46060
physaddrof 0xb46060 realmode 0x1
cet_reserved 0 cet_hpa 0xffffffff'fed69000
cet_event 0x2 cet_savestate 0x18fb5d0
cet_start_itmr 0xd431d'ce6021f6 cet_cur_itmr 0x1

Crash Event 3 (TOC, struct crash_event_table_struct at 0xb46090):
indexof 0 mapped 0x1
spaceof 0 addrof 0xb46090
physaddrof 0xb46090 realmode 0x1
cet_reserved 0 cet_hpa 0xffffffff'fed6d000
cet_event 0x2 cet_savestate 0x18fbcb0
cet_start_itmr 0xd431d'cd34557a cet_cur_itmr 0x1

Crash Event 4 (TOC, struct crash_event_table_struct at 0xb460c0):
indexof 0 mapped 0x1
spaceof 0 addrof 0xb460c0
physaddrof 0xb460c0 realmode 0x1
cet_reserved 0 cet_hpa 0xffffffff'fed29000
cet_event 0x2 cet_savestate 0x18f9a50
cet_start_itmr 0xd431d'c4cef69c cet_cur_itmr 0x1

Crash Event 5 (TOC, struct crash_event_table_struct at 0xb460f0):
indexof 0 mapped 0x1
spaceof 0 addrof 0xb460f0
physaddrof 0xb460f0 realmode 0x1
cet_reserved 0 cet_hpa 0xffffffff'fed21000
cet_event 0x2 cet_savestate 0xb3fe50
cet_start_itmr 0xd431d'c903479f cet_cur_itmr 0x1

Crash Event 6 (TOC, struct crash_event_table_struct at 0xb46120):
indexof 0 mapped 0x1
spaceof 0 addrof 0xb46120
physaddrof 0xb46120 realmode 0x1
cet_reserved 0 cet_hpa 0xffffffff'fed2d000
cet_event 0x2 cet_savestate 0x18fa130
cet_start_itmr 0xd431d'c43a5f67 cet_cur_itmr 0x1

Crash Event 7 (TOC, struct crash_event_table_struct at 0xb46150):
indexof 0 mapped 0x1
spaceof 0 addrof 0xb46150
physaddrof 0xb46150 realmode 0x1
cet_reserved 0 cet_hpa 0xffffffff'fed25000
cet_event 0x2 cet_savestate 0x18f9370
cet_start_itmr 0xd431d'c7177d89 cet_cur_itmr 0x1

Registers from the first crash event:
indexof 0
mapped 0x1
spaceof 0
addrof 0x18fa4a0
physaddrof 0x18fa4a0
realmode 0
sp 0x800003ff'ff7f3a30
rp_flag 0x1
pim.narrow.rp_gr0 0
pim.narrow.rp_gr1 0
pim.narrow.rp_rp 0
pim.narrow.rp_gr3 0x30
pim.narrow.rp_gr4 0
pim.narrow.rp_gr5 0x30
pim.narrow.rp_gr6 0x80000001
pim.narrow.rp_gr7 0x635f30
pim.narrow.rp_gr8 0x80000001
pim.narrow.rp_gr9 0x641478
pim.narrow.rp_gr10 0
pim.narrow.rp_gr11 0xa
pim.narrow.rp_gr12 0
pim.narrow.rp_gr13 0x1
pim.narrow.rp_gr14 0x80000001
pim.narrow.rp_gr15 0x63d3f8
pim.narrow.rp_gr16 0x40000000
pim.narrow.rp_gr17 0x40ead0
pim.narrow.rp_gr18 0x40000000
pim.narrow.rp_gr19 0x41ccd8
pim.narrow.rp_gr20 0x40000000
pim.narrow.rp_gr21 0x41ccd8
pim.narrow.rp_gr22 0x40000000
pim.narrow.rp_arg3 0x41ccd8
pim.narrow.rp_arg2 0
pim.narrow.rp_arg1 0
pim.narrow.rp_arg0 0
pim.narrow.rp_dp 0
pim.narrow.rp_ret0 0xffffffff
pim.narrow.rp_ret1 0xffffffff
pim.narrow.rp_sp 0
pim.narrow.rp_gr31 0x1
pim.narrow.rp_cr0 0x40000000
pim.narrow.rp_cr1 0x2ff088
pim.narrow.rp_cr2 0
pim.narrow.rp_cr3 0x2e
pim.narrow.rp_cr4 0
pim.narrow.rp_cr5 0x2
pim.narrow.rp_cr6 0
pim.narrow.rp_cr7 0x8a
pim.narrow.rp_cr8 0x80000001
pim.narrow.rp_cr9 0x63f438
pim.narrow.rp_cr10 0
pim.narrow.rp_cr11 0
pim.narrow.rp_cr12 0xc0000006
pim.narrow.rp_cr13 0x7ec98a78
pim.narrow.rp_cr14 0xc0000004
pim.narrow.rp_cr15 0x2a0670f0
pim.narrow.rp_cr16 0
pim.narrow.rp_pcsq_head 0
pim.narrow.rp_pcoq_head 0
pim.narrow.rp_cr19 0x32
pim.narrow.rp_cr20 0x80000001
pim.narrow.rp_cr21 0x63d6e0
pim.narrow.rp_cr22 0x80000001
pim.narrow.rp_cr23 0x44aea8
pim.narrow.rp_cr24 0x40000000
pim.narrow.rp_cr25 0x40ecd8
pim.narrow.rp_cr26 0x800003ff
pim.narrow.rp_cr27 0xff7f3a00
pim.narrow.rp_cr28 0x800003ff
pim.narrow.rp_cr29 0xff7f3a30
pim.narrow.rp_cr30 0
pim.narrow.rp_cr31 0
pim.narrow.rp_sr0 0
pim.narrow.rp_sr1 0x7ce27700
pim.narrow.rp_sr2 0
pim.narrow.rp_sr3 0
pim.narrow.rp_sr4 0
pim.narrow.rp_sr5 0
pim.narrow.rp_sr6 0
pim.narrow.rp_sr7 0
pim.narrow.rp_pcsq_tail 0
pim.narrow.rp_pcoq_tail 0
pim.narrow.rp_hpmc_type 0
pim.narrow.rp_hpmc_cpu_state 0
pim.narrow.rp_hpmc_detected_by 0
pim.narrow.rp_hpmc_cache 0
pim.narrow.rp_hpmc_tlb 0
pim.narrow.rp_hpmc_bus 0
pim.narrow.rp_hpmc_assists 0x69ba
pim.narrow.rp_hpmc_processor 0xa227
pim.narrow.rp_hpmc_assist_id 0x6b48
pim.narrow.rp_hpmc_slave_addr 0x9894
pim.narrow.rp_hpmc_master_addr 0
pim.narrow.rp_hversion_depend[0] 0xc0
pim.narrow.rp_hversion_depend[1] 0
pim.narrow.rp_hversion_depend[2] 0x31
pim.narrow.rp_hversion_depend[3] 0x5da2
pim.narrow.rp_hversion_depend[4] 0xa6a4
pim.narrow.rp_hversion_depend[5] 0xd772
pim.narrow.rp_hversion_depend[6] 0x4a94
pim.narrow.rp_hversion_depend[7] 0
pim.narrow.rp_hversion_depend[8] 0x2a000
pim.narrow.rp_hversion_depend[9] 0xffffffff
pim.narrow.rp_hversion_depend[10] 0xffffffff
pim.narrow.rp_hversion_depend[11] 0xd431d
pim.narrow.rp_hversion_depend[12] 0x2f002a70
pim.narrow.rp_hversion_depend[13] 0
pim.narrow.rp_hversion_depend[14] 0x7ac800
pim.narrow.rp_hversion_depend[15] 0x40000000
pim.narrow.rp_hversion_depend[16] 0xe32b7f
pim.narrow.rp_hversion_depend[17] 0
pim.narrow.rp_hversion_depend[18] 0x2b661000
pim.narrow.rp_hversion_depend[19] 0
pim.narrow.rp_hversion_depend[20] 0x103401e1
pim.narrow.rp_hversion_depend[21] 0xc0000000
pim.narrow.rp_hversion_depend[22] 0x4e890aa8
pim.narrow.rp_hversion_depend[23] 0xff
pim.narrow.rp_hversion_depend[24] 0x804ff1f
pim.narrow.rp_hversion_depend[25] 0x80000000
pim.narrow.rp_hversion_depend[26] 0
pim.narrow.rp_hversion_depend[27] 0
pim.narrow.rp_hversion_depend[28] 0x1909f20
pim.narrow.rp_hversion_depend[29] 0
pim.narrow.rp_hversion_depend[30] 0xc71018fc
pim.narrow.rp_hversion_depend[31] 0x80000001
pim.narrow.rp_hversion_depend[32] 0x44ef68
pim.narrow.rp_hversion_depend[33] 0x80000001
pim.narrow.rp_hversion_depend[34] 0x204ea60
pim.narrow.rp_hversion_depend[35] 0xc0000006
pim.narrow.rp_hversion_depend[36] 0x7ec3d948
pim.narrow.rp_hversion_depend[37] 0
pim.narrow.rp_hversion_depend[38] 0xfd
pim.narrow.rp_hversion_depend[39] 0x80000001
pim.wide.rp_gr0_hi 0
pim.wide.rp_gr0_low 0
pim.wide.rp_gr1_hi 0
pim.wide.rp_gr1_low 0x30
pim.wide.rp_rp_hi 0
pim.wide.rp_rp_low 0x30
pim.wide.rp_gr3_hi 0x80000001
pim.wide.rp_gr3_low 0x635f30
pim.wide.rp_gr4_hi 0x80000001
pim.wide.rp_gr4_low 0x641478
pim.wide.rp_gr5_hi 0
pim.wide.rp_gr5_low 0xa
pim.wide.rp_gr6_hi 0
pim.wide.rp_gr6_low 0x1
pim.wide.rp_gr7_hi 0x80000001
pim.wide.rp_gr7_low 0x63d3f8
pim.wide.rp_gr8_hi 0x40000000
pim.wide.rp_gr8_low 0x40ead0
pim.wide.rp_gr9_hi 0x40000000
pim.wide.rp_gr9_low 0x41ccd8
pim.wide.rp_gr10_hi 0x40000000
pim.wide.rp_gr10_low 0x41ccd8
pim.wide.rp_gr11_hi 0x40000000
pim.wide.rp_gr11_low 0x41ccd8
pim.wide.rp_gr12_hi 0
pim.wide.rp_gr12_low 0
pim.wide.rp_gr13_hi 0
pim.wide.rp_gr13_low 0
pim.wide.rp_gr14_hi 0xffffffff
pim.wide.rp_gr14_low 0xffffffff
pim.wide.rp_gr15_hi 0
pim.wide.rp_gr15_low 0x1
pim.wide.rp_gr16_hi 0x40000000
pim.wide.rp_gr16_low 0x2ff088
pim.wide.rp_gr17_hi 0
pim.wide.rp_gr17_low 0x2e
pim.wide.rp_gr18_hi 0
pim.wide.rp_gr18_low 0x2
pim.wide.rp_gr19_hi 0
pim.wide.rp_gr19_low 0x8a
pim.wide.rp_gr20_hi 0x80000001
pim.wide.rp_gr20_low 0x63f438
pim.wide.rp_gr21_hi 0
pim.wide.rp_gr21_low 0
pim.wide.rp_gr22_hi 0xc0000006
pim.wide.rp_gr22_low 0x7ec98a78
pim.wide.rp_arg3_hi 0xc0000004
pim.wide.rp_arg3_low 0x2a0670f0
pim.wide.rp_arg2_hi 0
pim.wide.rp_arg2_low 0
pim.wide.rp_arg1_hi 0
pim.wide.rp_arg1_low 0x32
pim.wide.rp_arg0_hi 0x80000001
pim.wide.rp_arg0_low 0x63d6e0
pim.wide.rp_dp_hi 0x80000001
pim.wide.rp_dp_low 0x44aea8
pim.wide.rp_ret0_hi 0x40000000
pim.wide.rp_ret0_low 0x40ecd8
pim.wide.rp_ret1_hi 0x800003ff
pim.wide.rp_ret1_low 0xff7f3a00
pim.wide.rp_sp_hi 0x800003ff
pim.wide.rp_sp_low 0xff7f3a30
pim.wide.rp_gr31_hi 0
pim.wide.rp_gr31_low 0
pim.wide.rp_cr0_hi 0
pim.wide.rp_cr0_low 0x7ce27700
pim.wide.rp_cr1_hi 0
pim.wide.rp_cr1_low 0
pim.wide.rp_cr2_hi 0
pim.wide.rp_cr2_low 0
pim.wide.rp_cr3_hi 0
pim.wide.rp_cr3_low 0
pim.wide.rp_cr4_hi 0
pim.wide.rp_cr4_low 0
pim.wide.rp_cr5_hi 0
pim.wide.rp_cr5_low 0
pim.wide.rp_cr6_hi 0
pim.wide.rp_cr6_low 0
pim.wide.rp_cr7_hi 0
pim.wide.rp_cr7_low 0
pim.wide.rp_cr8_hi 0x69ba
pim.wide.rp_cr8_low 0xa227
pim.wide.rp_cr9_hi 0x6b48
pim.wide.rp_cr9_low 0x9894
pim.wide.rp_cr10_hi 0
pim.wide.rp_cr10_low 0xc0
pim.wide.rp_cr11_hi 0
pim.wide.rp_cr11_low 0x31
pim.wide.rp_cr12_hi 0x5da2
pim.wide.rp_cr12_low 0xa6a4
pim.wide.rp_cr13_hi 0xd772
pim.wide.rp_cr13_low 0x4a94
pim.wide.rp_cr14_hi 0
pim.wide.rp_cr14_low 0x2a000
pim.wide.rp_cr15_hi 0xffffffff
pim.wide.rp_cr15_low 0xffffffff
pim.wide.rp_cr16_hi 0xd431d
pim.wide.rp_cr16_low 0x2f002a70
pim.wide.rp_pcsq_head_hi 0
pim.wide.rp_pcsq_head_low 0x7ac800
pim.wide.rp_pcoq_head_hi 0x40000000
pim.wide.rp_pcoq_head_low 0xe32b7f
pim.wide.rp_cr19_hi 0
pim.wide.rp_cr19_low 0x2b661000
pim.wide.rp_cr20_hi 0
pim.wide.rp_cr20_low 0x103401e1
pim.wide.rp_cr21_hi 0xc0000000
pim.wide.rp_cr21_low 0x4e890aa8
pim.wide.rp_cr22_hi 0xff
pim.wide.rp_cr22_low 0x804ff1f
pim.wide.rp_cr23_hi 0x80000000
pim.wide.rp_cr23_low 0
pim.wide.rp_cr24_hi 0
pim.wide.rp_cr24_low 0x1909f20
pim.wide.rp_cr25_hi 0
pim.wide.rp_cr25_low 0xc71018fc
pim.wide.rp_cr26_hi 0x80000001
pim.wide.rp_cr26_low 0x44ef68
pim.wide.rp_cr27_hi 0x80000001
pim.wide.rp_cr27_low 0x204ea60
pim.wide.rp_cr28_hi 0xc0000006
pim.wide.rp_cr28_low 0x7ec3d948
pim.wide.rp_cr29_hi 0
pim.wide.rp_cr29_low 0xfd
pim.wide.rp_cr30_hi 0x80000001
pim.wide.rp_cr30_low 0x44aea8
pim.wide.rp_cr31_hi 0x800003ff
pim.wide.rp_cr31_low 0xff7f3790
pim.wide.rp_sr0_hi 0
pim.wide.rp_sr0_low 0x135a800
pim.wide.rp_sr1_hi 0
pim.wide.rp_sr1_low 0x3ff2400
pim.wide.rp_sr2_hi 0
pim.wide.rp_sr2_low 0
pim.wide.rp_sr3_hi 0
pim.wide.rp_sr3_low 0
pim.wide.rp_sr4_hi 0
pim.wide.rp_sr4_low 0x97d000
pim.wide.rp_sr5_hi 0
pim.wide.rp_sr5_low 0x7ac800
pim.wide.rp_sr6_hi 0
pim.wide.rp_sr6_low 0x3ff2400
pim.wide.rp_sr7_hi 0
pim.wide.rp_sr7_low 0x135a800
pim.wide.rp_pcsq_tail_hi 0
pim.wide.rp_pcsq_tail_low 0x7ac800
pim.wide.rp_pcoq_tail_hi 0x40000000
pim.wide.rp_pcoq_tail_low 0xe32b83
pim.wide.rp_hpmc_type 0x20000000
pim.wide.rp_hpmc_cpu_state 0x9e000004
pim.wide.rp_hpmc_cache 0
pim.wide.rp_hpmc_tlb 0
pim.wide.rp_hpmc_bus 0x105035
pim.wide.rp_hpmc_assists 0
pim.wide.rp_hpmc_assist_id 0
pim.wide.rp_path_info 0
pim.wide.rp_hpmc_slave_addr_hi 0
pim.wide.rp_hpmc_slave_addr 0
pim.wide.rp_hpmc_master_addr_hi 0xffffffff
pim.wide.rp_hpmc_master_addr 0xfed61000
pim.wide.rp_hversion_depend[0] 0xc2f9800
pim.wide.rp_hversion_depend[1] 0
pim.wide.rp_hversion_depend[2] 0
pim.wide.rp_hversion_depend[3] 0
pim.wide.rp_hversion_depend[4] 0
pim.wide.rp_hversion_depend[5] 0
pim.wide.rp_hversion_depend[6] 0
pim.wide.rp_hversion_depend[7] 0
pim.wide.rp_hversion_depend[8] 0xe0
pim.wide.rp_hversion_depend[9] 0xb4
pim.wide.rp_hversion_depend[10] 0
pim.wide.rp_hversion_depend[11] 0x16c16c1e
pim.wide.rp_hversion_depend[12] 0
pim.wide.rp_hversion_depend[13] 0
pim.wide.rp_hversion_depend[14] 0x40995000
pim.wide.rp_hversion_depend[15] 0
pim.wide.rp_hversion_depend[16] 0x40668000
pim.wide.rp_hversion_depend[17] 0
pim.wide.rp_hversion_depend[18] 0x3ff00000
pim.wide.rp_hversion_depend[19] 0x5
pim.wide.rp_hversion_depend[20] 0x40220000
pim.wide.rp_hversion_depend[21] 0x6
pim.wide.rp_hversion_depend[22] 0
pim.wide.rp_hversion_depend[23] 0x9
pim.wide.rp_hversion_depend[24] 0x55555555
pim.wide.rp_hversion_depend[25] 0x55555555
pim.wide.rp_hversion_depend[26] 0x55555555
pim.wide.rp_hversion_depend[27] 0x55555555
pim.wide.rp_hversion_depend[28] 0x55555555
pim.wide.rp_hversion_depend[29] 0x55555555
pim.wide.rp_hversion_depend[30] 0x55555555
pim.wide.rp_hversion_depend[31] 0x55555555
pim.wide.rp_hversion_depend[32] 0x55555555
pim.wide.rp_hversion_depend[33] 0x55555555
pim.wide.rp_hversion_depend[34] 0x55555555
pim.wide.rp_hversion_depend[35] 0x55555555
pim.wide.rp_hversion_depend[36] 0x55555555
pim.wide.rp_hversion_depend[37] 0x55555555
pim.wide.rp_hversion_depend[38] 0x55555555
pim.wide.rp_hversion_depend[39] 0x55555555
rp_itimer_hi 0xd431d
rp_itimer_low 0x8fdf7c04
rp_time 0x4695441c
rp_time2 0xcf850
rp_proc_hpa 0xffffffff'fed61000
rp_crash_type 0x3
rp_pim_return 0
rp_global_psw 0x40000
rp_nothread 0
end_align 0

Stack trace from the first crash event:
stack trace for event 0
crash event was an HPMC
PC (0x7ac800.0x40000000`00e32b7c) was in user space
+--------------------------------------------+
| Per-Processor Info |
+--------------------------------------------+

Processor 0 (struct mpinfo at 0x18fd000):
prochpa 0xffffffff'fed21000 cpustate 0x1
blktime 0 iva 0x2a000
tmpsavestate 0x4a7c81c0 iva_curpri 0x2b2
iva_nothread 0x1 em_fpu 0xb2a700
timeinval 0xd431d'25bdd2b2 globalpsw 0x40000
topics 0xadf000 ibase 0xad4000
threadp 0xffffffff'ffffffff ics 0xad4050
spinlock_depth 0 entry_int_level 0xffffff00'ffffffff
last_idletime 0x2873fa3f last_tsharetime 0x2873fa3c
rqi_cr_it 0x24b1c6b9

Processor 1 (struct mpinfo at 0x19003c8):
prochpa 0xffffffff'fed25000 cpustate 0x1
blktime 0 iva 0x2a000
tmpsavestate 0x4a7c8640 iva_curpri 0x2b2
iva_nothread 0x1 em_fpu 0xb2a820
timeinval 0xd431d'238ffb29 globalpsw 0x40000
topics 0x1923000 ibase 0x1918000
threadp 0xffffffff'ffffffff ics 0x1918000
spinlock_depth 0 entry_int_level 0xffffff00'ffffffff
last_idletime 0x2873fa3f last_tsharetime 0x2873fa3c
rqi_cr_it 0x22946f41

Processor 2 (struct mpinfo at 0x1903790):
prochpa 0xffffffff'fed29000 cpustate 0x1
blktime 0 iva 0x2a000
tmpsavestate 0x4a7c8ac0 iva_curpri 0x2b2
iva_nothread 0x1 em_fpu 0xb2a940
timeinval 0xd431d'21f823c5 globalpsw 0x40000
topics 0x192f000 ibase 0x1924000
threadp 0xffffffff'ffffffff ics 0x1924000
spinlock_depth 0 entry_int_level 0xffffff00'ffffffff
last_idletime 0x2873fa3f last_tsharetime 0x2873fa3b
rqi_cr_it 0x20a88a22

Processor 3 (struct mpinfo at 0x1906b58):
prochpa 0xffffffff'fed2d000 cpustate 0x1
blktime 0 iva 0x2a000
tmpsavestate 0x4a7c8f40 iva_curpri 0x2b2
iva_nothread 0x1 em_fpu 0xb2aa60
timeinval 0xd431d'20c8c6b0 globalpsw 0x40000
topics 0x193b000 ibase 0x1930000
threadp 0xffffffff'ffffffff ics 0x1930000
spinlock_depth 0 entry_int_level 0xffffff00'ffffffff
last_idletime 0x2873fa3f last_tsharetime 0x2873fa32
rqi_cr_it 0x1c85df29

Processor 4 (struct mpinfo at 0x1909f20):
prochpa 0xffffffff'fed61000 cpustate 0x2
blktime 0 iva 0x2a000
tmpsavestate 0x4a7c93c0 iva_curpri 0x2cd
iva_nothread 0x1 em_fpu 0xb2ab80
timeinval 0xd431d'2eea2e39 globalpsw 0
topics 0x1947000 ibase 0x193c000
threadp 0x1'a455f040 ics 0x193c000
spinlock_depth 0 entry_int_level 0xfffffff0'ffffffff
last_idletime 0x2873fa3c last_tsharetime 0x2873fa3c
rqi_cr_it 0x2e42da72

Processor 5 (struct mpinfo at 0x190d2e8):
prochpa 0xffffffff'fed65000 cpustate 0x1
blktime 0 iva 0x2a000
tmpsavestate 0x4a7c9840 iva_curpri 0x2b2
iva_nothread 0x1 em_fpu 0xb2aca0
timeinval 0xd431d'2e7bdfcf globalpsw 0x40000
topics 0x1953000 ibase 0x1948000
threadp 0xffffffff'ffffffff ics 0x1948000
spinlock_depth 0 entry_int_level 0xfffffff0'ffffffff
last_idletime 0x2873fa3f last_tsharetime 0x2873fa3e
rqi_cr_it 0x2e281214

Processor 6 (struct mpinfo at 0x19106b0):
prochpa 0xffffffff'fed69000 cpustate 0x1
blktime 0 iva 0x2a000
tmpsavestate 0x4a7c9cc0 iva_curpri 0x2b2
iva_nothread 0x1 em_fpu 0xb2adc0
timeinval 0xd431d'2d6e89c9 globalpsw 0x40000
topics 0x195f000 ibase 0x1954000
threadp 0xffffffff'ffffffff ics 0x1954000
spinlock_depth 0 entry_int_level 0xfffffff0'ffffffff
last_idletime 0x2873fa3f last_tsharetime 0x2873fa3e
rqi_cr_it 0x2d1ab78e

Processor 7 (struct mpinfo at 0x1913a78):
prochpa 0xffffffff'fed6d000 cpustate 0x1
blktime 0 iva 0x2a000
tmpsavestate 0x4a7ca140 iva_curpri 0x2b2
iva_nothread 0x1 em_fpu 0xb2aee0
timeinval 0xd431d'2bf66a50 globalpsw 0x40000
topics 0x196b000 ibase 0x1960000
threadp 0xffffffff'ffffffff ics 0x1960000
spinlock_depth 0 entry_int_level 0xfffffff0'ffffffff
last_idletime 0x2873fa3f last_tsharetime 0x2873fa3e
rqi_cr_it 0x2ba29a6e
+--------------------------------------------+
| VM Tables |
+--------------------------------------------+

swbuf 0 swbufNSWBUF 0
nswbuf 2786 sizeof(struct buf) 600
inode 1241583616 inodeNINODE 1245646848
ninode 8192 sizeof(struct inode) 496
file 1245646848 fileNFILE 1246368624
nfile 8202 sizeof(struct file) 88
ncache 1246368640 ncacheNCACHE 1248285568
ncsize 13312 sizeof(struct ncache) 144
cfree 1249587712 cfreeNCFREE 1249673632
nclist 2148 sizeof(struct cblock) 40
ncallout 7216 sizeof(callout_t) 96
pfdat_ptr 1249715392 pfdat_ptrNPFDAT_PTR 1249727680
sizeof(pfd_t) 96 proc_list 12549440
last_proc_allocd 2679066688 num_procs_allocd 398
sizeof(proc_t) 1216
+--------------------------------------------+
| VM Values |
+--------------------------------------------+

pdir 1140850688
scaled_npdir 128
sizeof(struct hpde)*scaled_npdir 3072
htbl 1140850688
nhtbl 0
sizeof(struct hpde)*nhtbl 0
+--------------------------------------------+
| VM Stats |
+--------------------------------------------+

phys_mem_pages 2097152 minfree 3328 lotsfree 32768
freemem 26511 desfree 7168 bufpages 161707
+--------------------------------------------+
| Processes |
+--------------------------------------------+
Undefined subroutine &main::PrintProcesses called at /usr/contrib/Q4/lib/q4lib_11.11/analyze.pl line 133, line 1419.
$ cat whathappened
Created with WhatHappened: @(#) $Revision: 1.9 $ $Date: 01/02/19 16:54:35 $ for 11.i

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

The whathappened script is not actively maintained by WTEC anymore.
We have decided to go for one first passing tool, which is "crashinfo".
crashinfo is available in this dump reading package and should be used instead
of whathappend. Besides the default ascii output, be sure to try the html
(-H option). The -e option allows to specify a email address to get the output
sent to.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

System Name: HP-UX
Node Name: hostname
Release: 11.11
Version: U
Model: 9000/800/N4000-55
Machine ID: 610309383
Processors: 8
Architecture: PA-RISC 2.0
CPU is a: PCXW+/PA-8600
Physical Mem: 8.00 GBytes

This is a 64 Bit Kernel
The system had been up for 78.55 days (678689343 ticks).
Load averages: 0.14 0.14 0.14.

System went down at: Thu Jul 12 06:57:00 2007
+--------------------------------------------+
| Message Buffer |
+--------------------------------------------+

sysmap_64bit: rmap ovflo, lost [617031,617032)
sysmap_64bit: rmap ovflo, lost [616656,616657)
sysmap_64bit: rmap ovflo, lost [616171,616172)
sysmap_64bit: rmap ovflo, lost [615993,615994)
sysmap_64bit: rmap ovflo, lost [615649,615650)
sysmap_64bit: rmap ovflo, lost [614371,614372)
sysmap_64bit: rmap ovflo, lost [614141,614142)
sysmap_64bit: rmap ovflo, lost [612527,612528)
sysmap_64bit: rmap ovflo, lost [612149,612150)
sysmap_64bit: rmap ovflo, lost [611016,611017)
sysmap_64bit: rmap ovflo, lost [608080,608081)
sysmap_64bit: rmap ovflo, lost [606235,606236)
sysmap_64bit: rmap ovflo, lost [604316,604317)
sysmap_64bit: rmap ovflo, lost [603218,603219)
sysmap_64bit: rmap ovflo, lost [601436,601437)
sysmap_64bit: rmap ovflo, lost [601010,601011)
sysmap_64bit: rmap ovflo, lost [600660,600661)
sysmap_64bit: rmap ovflo, lost [600002,600003)
sysmap_64bit: rmap ovflo, lost [599781,599782)
sysmap_64bit: rmap ovflo, lost [598627,598628)
sysmap_64bit: rmap ovflo, lost [598458,598459)
sysmap_64bit: rmap ovflo, lost [598451,598452)
sysmap_64bit: rmap ovflo, lost [597856,597857)
sysmap_64bit: rmap ovflo, lost [1822549,1822553)
sysmap_64bit: rmap ovflo, lost [597760,597761)
sysmap_64bit: rmap ovflo, lost [597664,597665)
sysmap_64bit: rmap ovflo, lost [597654,597655)
sysmap_64bit: rmap ovflo, lost [597474,597475)
sysmap_64bit: rmap ovflo, lost [597436,597437)
sysmap_64bit: rmap ovflo, lost [597431,597432)
sysmap_64bit: rmap ovflo, lost [597407,597408)
sysmap_64bit: rmap ovflo, lost [597170,597171)
sysmap_64bit: rmap ovflo, lost [597073,597074)
sysmap_64bit: rmap ovflo, lost [596878,596879)
sysmap_64bit: rmap ovflo, lost [596655,596656)
sysmap_64bit: rmap ovflo, lost [596089,596090)
sysmap_64bit: rmap ovflo, lost [595675,595676)
sysmap_64bit: rmap ovflo, lost [595336,595337)
sysmap_64bit: rmap ovflo, lost [595259,595260)
sysmap_64bit: rmap ovflo, lost [593747,593748)
sysmap_64bit: rmap ovflo, lost [593722,593724)
sysmap_64bit: rmap ovflo, lost [593558,593559)
sysmap_64bit: rmap ovflo, lost [593522,593523)
p ovflo, lost [1824117,1824118)
sysmap_64bit: rmap ovflo, lost [629192,629193)
sysmap_64bit: rmap ovflo, lost [629064,629065)
sysmap_64bit: rmap ovflo, lost [627490,627491)
sysmap_64bit: rmap ovflo, lost [627482,627483)
sysmap_64bit: rmap ovflo, lost [627116,627117)
sysmap_64bit: rmap ovflo, lost [625867,625868)
sysmap_64bit: rmap ovflo, lost [625581,625582)
sysmap_64bit: rmap ovflo, lost [625058,625059)
sysmap_64bit: rmap ovflo, lost [625048,625049)
sysmap_64bit: rmap ovflo, lost [624745,624746)
sysmap_64bit: rmap ovflo, lost [623378,623379)
sysmap_64bit: rmap ovflo, lost [623069,623070)
sysmap_64bit: rmap ovflo, lost [623059,623060)
sysmap_64bit: rmap ovflo, lost [622956,622957)
sysmap_64bit: rmap ovflo, lost [622904,622905)
sysmap_64bit: rmap ovflo, lost [622639,622640)
sysmap_64bit: rmap ovflo, lost [622612,622613)
sysmap_64bit: rmap ovflo, lost [622353,622354)
sysmap_64bit: rmap ovflo, lost [622300,622301)
sysmap_64bit: rmap ovflo, lost [622205,622206)
sysmap_64bit: rmap ovflo, lost [622165,622166)
sysmap_64bit: rmap ovflo, lost [622153,622154)
sysmap_64bit: rmap ovflo, lost [622147,622148)
sysmap_64bit: rmap ovflo, lost [622142,622143)
sysmap_64bit: rmap ovflo, lost [622138,622139)
sysmap_64bit: rmap ovflo, lost [622078,622079)
sysmap_64bit: rmap ovflo, lost [622020,622021)
sysmap_64bit: rmap ovflo, lost [620807,620808)
sysmap_64bit: rmap ovflo, lost [620636,620637)
sysmap_64bit: rmap ovflo, lost [620483,620484)
sysmap_64bit: rmap ovflo, lost [620449,620450)
sysmap_64bit: rmap ovflo, lost [620102,620103)
sysmap_64bit: rmap ovflo, lost [619986,619987)
sysmap_64bit: rmap ovflo, lost [619237,619238)
sysmap_64bit: rmap ovflo, lost [619072,619073)
sysmap_64bit: rmap ovflo, lost [618891,618892)
sysmap_64bit: rmap ovflo, lost [618819,618820)
sysmap_64bit: rmap ovflo, lost [618548,618549)
sysmap_64bit: rmap ovflo, lost [617805,617806)
sysmap_64bit: rmap ovflo, lost [617713,617714)
sysmap_64bit: rmap ovflo, lost [617227,617228)
sysmap_64bit: rmap ovflo, lost [617158,617159)
sysmap_64bit: rmap ovflo, lost [617045,
+--------------------------------------------+
| Processor activity |
+--------------------------------------------+
Processor 4 started it by HPMC'ing. Here is the stack trace:
stack trace for event 0
crash event was an HPMC
PC (0x7ac800.0x40000000`00e32b7c) was in user space
+--------------------------------------------+
| HPMC Registers for HPA 0xfffffffffed61000 |
+--------------------------------------------+
type: 20000000 state: 9E000004
cache: 00000000 tlb: 00000000 bus: 00105035 assists: 00000000
slave: 00000000 mastr: FED61000

This is most likely a hardware problem, for further HPMC troubleshooting,
please see: http://wtec.cup.hp.com/~cpuhw/

Processor 0: idle
-------------
can not get struct frame_marker at 0x0.0xff`ff408cc0 can not get struct
frame_marker at 0x0.0xff`ff408aa0 idle+0x954
-------------

Processor 1: idle
-------------
can not get struct frame_marker at 0x0.0xff`ff40acc0 can not get struct
frame_marker at 0x0.0xff`ff40aaa0 idle+0xb1c
-------------

Processor 2: idle
-------------
can not get struct frame_marker at 0x0.0xff`ff40ccc0 can not get struct
frame_marker at 0x0.0xff`ff40caa0 idle+0xba8
-------------

Processor 3: idle
-------------
can not get struct frame_marker at 0x0.0xff`ff40ecc0 can not get struct
frame_marker at 0x0.0xff`ff40eaa0 idle+0xb0c
-------------

Processor 4: running thread @ 0x1a455f040 (tid 6890051) Undefined subroutine &main::LoadMatchingProcesses called at /usr/contrib/Q4/lib/q4lib_11.11/whathappened.pl line 436, line 1743.
$

Re: System Hangs

skt_skt — Wed, 11 Jul 2007 22:27:32 GMT

Do you rember when did the firmware is updated last time?

could yo send the sysrev output from MP

Re: System Hangs

SCSI Error — Wed, 11 Jul 2007 22:43:26 GMT

sorry cant' remember.

there's no command sysrev at my GSP>

attached is the ts99 from /var/tombstones

Re: System Hangs

Marcel Burggraeve — Fri, 13 Jul 2007 02:54:10 GMT

Enter your GSP with CTRL-b on the console and type help or he.
As far as I know it should show you the version of the GSP at one of the top lines.

Re: System Hangs

SCSI Error — Sun, 15 Jul 2007 18:04:52 GMT

==== GSP Help ============================================(Administrator)===
Hardware Revision 8 Firmware Revision A.01.09 Mar 20 2000,12:45:29

Re: System Hangs

skt_skt — Sun, 15 Jul 2007 18:28:03 GMT

here is whaht i see for my N4000-55.

==== GSP Help ============================================(Administrator)===
Hardware Revision 8 Firmware Revision A.01.12 Oct 23 2002,13:53:42

Sorry this GSP does not support SYSREV.

Re: System Hangs

skt_skt — Sun, 15 Jul 2007 18:31:55 GMT

Did u analyse the reported HPMC error,i am not sure hot to go further

+--------------------------------------------+
Processor 4 started it by HPMC'ing. Here is the stack trace:
stack trace for event 0
crash event was an HPMC
PC (0x7ac800.0x40000000`00e32b7c) was in user space
+--------------------------------------------+
| HPMC Registers for HPA 0xfffffffffed61000 |
+--------------------------------------------+
type: 20000000 state: 9E000004
cache: 00000000 tlb: 00000000 bus: 00105035 assists: 00000000
slave: 00000000 mastr: FED61000

This is most likely a hardware problem, for further HPMC troubleshooting,
please see: http://wtec.cup.hp.com/~cpuhw/

also i rember the error reported on /var/tombstones/ts99.

Re: System Hangs

SCSI Error — Sun, 15 Jul 2007 18:36:42 GMT

Sorry I dont know how to anylyze this one.. Thats y i posted it coz i need help.

Thanks peeps