Reboot after panic: Data page fault

[Pradep]

Hi All,

hpux 11.11 vPar crashed and rebooted after panic. But Q4 doesn't trace to any HPMC or MCSG error. Can someone pls help to find cause ?
Its running Oracle9i DB.
Attached analyze,What,trace0 and trace1.

Please help. thanks in advance.


#ll
total 2136208
-rw------- 1 root sys 212 Nov 8 12:04 .q4_hist
-rw-r--r-- 1 root sys 1484 Nov 7 21:39 INDEX
-rw-r--r-- 1 root sys 50786 Nov 8 11:47 ana.out
-rw-r--r-- 1 root sys 29487104 Nov 7 21:37 image.1.1
-rw-r--r-- 1 root sys 134189056 Nov 7 21:37 image.2.1
-rw-r--r-- 1 root sys 134189056 Nov 7 21:37 image.2.2
-rw-r--r-- 1 root sys 134189056 Nov 7 21:37 image.2.3
-rw-r--r-- 1 root sys 134184960 Nov 7 21:38 image.2.4
-rw-r--r-- 1 root sys 134189056 Nov 7 21:38 image.2.5
-rw-r--r-- 1 root sys 134189056 Nov 7 21:38 image.2.6
-rw-r--r-- 1 root sys 134189056 Nov 7 21:39 image.2.7
-rw-r--r-- 1 root sys 104910848 Nov 7 21:39 image.2.8
-rw-r--r-- 1 root sys 615 Nov 8 12:00 trace0
-rw-r--r-- 1 root sys 402 Nov 8 12:00 trace1
-rw-r--r-- 1 root sys 19878744 Nov 7 21:37 vmunix
-rw-r--r-- 1 root sys 8590 Nov 8 11:48 what.out
# cat INDEX
comment savecrash crash dump INDEX file
version 2
hostname
modelname 9000/800/SD16A
panic Data page fault
dumptime 1226063853 Fri Nov 7 21:17:33 SST 2008
savetime 1226065021 Fri Nov 7 21:37:01 SST 2008
release @(#) $Revision: vmunix: vw: -proj selectors: CUPI80_BL2000_1108 -c 'Vw for CUPI80_BL2000_1108 build' -- cupi80_bl2000_1108 'CUPI80_BL2000_1108' Wed Nov 8 19:24:56 PST 2000 $
memsize 47223668736
chunksize 134217728
module /stand/vmunix vmunix 19878744 655913312
image image.1.1 0x0000000000000000 0x0000000001c1f000 0x0000000000008000 0x000000000000bfff 652474969
image image.2.1 0x0000000000000000 0x0000000007ff9000 0x0000000000784000 0x000000000078bff7 952728091
image image.2.2 0x0000000000000000 0x0000000007ff9000 0x000000000078bff8 0x0000000000793fef 4249759976
image image.2.3 0x0000000000000000 0x0000000007ff9000 0x0000000000793ff0 0x000000000079bfe7 3604247299
image image.2.4 0x0000000000000000 0x0000000007ff8000 0x000000000079bfe8 0x00000000007a3fdf 74907150
image image.2.5 0x0000000000000000 0x0000000007ff9000 0x00000000007a3fe0 0x00000000007abfd7 675347677
image image.2.6 0x0000000000000000 0x0000000007ff9000 0x00000000007abfd8 0x00000000007b3fcf 961976594
image image.2.7 0x0000000000000000 0x0000000007ff9000 0x00000000007b3fd0 0x00000000007bbfc7 3678319457
image image.2.8 0x0000000000000000 0x000000000640d000 0x00000000007bbfc8 0x000000000127ebff 2312941526
#
#
# q4 .
@(#) q4 $Revision: 11.X B.11.23d Thu May 6 18:05:11 PST 2003$ 0
Reading kernel symbols ...
Reading data types ...
Initialized PA-RISC 2.0 address translator ...
Initializing stack tracer ...
script /usr/contrib/Q4/lib/q4lib/sample.q4rc.pl
executable /usr/contrib/Q4/bin/perl
version 5.006001
SCRIPT_LIBRARY = /usr/contrib/Q4/lib/q4lib_11.11
perl will try to access scripts from directory
/usr/contrib/Q4/lib/q4lib_11.11

loaded spinlock_depth.pl
run Print_ProcState
q4: (warning) No loadable modules were found
q4: (warning) No loadable modules were found
Try to load dumpdev_t to find dump space configured
Total Dump space CONFIGURED: 1048576 KB 1024 MB
Try to load dumpimage_t to find dump space actually used
Total Dump space USED: 1048552 KB 1023 MB
Okay, Dump space appears to be sufficient : about 0 MB extra
q4>
q4>
q4> examine panicstr using s
Data page fault
q4>
q4>
q4> runningprocs
014 12 0xc
q4>
q4>
q4> load mpinfo_t from mpproc_info max nmpinfo
loaded 12 mpinfo_ts as an array (stopped by max count)
q4>
q4>
q4> trace pile
q4: (warning) cr_isaddr passed yet page is not present
q4: (warning) cr_isaddr passed yet page is not present
q4: (warning) failing page number at 0xe23198
q4: (warning) failing page number at 0xe23198
can not get struct kthread at 0x1`1f198040
(there may not be a kernel thread here)

q4: (warning) cr_isaddr passed yet page is not present
q4: (warning) cr_isaddr passed yet page is not present
q4: (warning) failing page number at 0xe213c8
q4: (warning) failing page number at 0xe213c8
can not get struct kthread at 0x1`1d3c8040
(there may not be a kernel thread here)

q4: (warning) cr_isaddr passed yet page is not present
q4: (warning) cr_isaddr passed yet page is not present
q4: (warning) failing page number at 0x1195032
q4: (warning) failing page number at 0x1195032
can not get struct kthread at 0x1`25032040
(there may not be a kernel thread here)

q4: (warning) cr_isaddr passed yet page is not present
q4: (warning) cr_isaddr passed yet page is not present
q4: (warning) failing page number at 0xf03428
q4: (warning) failing page number at 0xf03428
can not get struct kthread at 0x0`67428040
(there may not be a kernel thread here)

q4: (warning) cr_isaddr passed yet page is not present
q4: (warning) cr_isaddr passed yet page is not present
q4: (warning) failing page number at 0xf02b4e
q4: (warning) failing page number at 0xf02b4e
can not get struct kthread at 0x0`66b4e040
(there may not be a kernel thread here)

q4: (warning) cr_isaddr passed yet page is not present
q4: (warning) cr_isaddr passed yet page is not present
q4: (warning) failing page number at 0xe20ca0
q4: (warning) failing page number at 0xe20ca0
can not get struct kthread at 0x1`1cca0040
(there may not be a kernel thread here)

q4: (warning) cr_isaddr passed yet page is not present
q4: (warning) cr_isaddr passed yet page is not present
q4: (warning) failing page number at 0xf0333b
q4: (warning) failing page number at 0xf0333b
can not get struct kthread at 0x0`6733b040
(there may not be a kernel thread here)

q4: (warning) cr_isaddr passed yet page is not present
q4: (warning) cr_isaddr passed yet page is not present
q4: (warning) failing page number at 0xe219a2
q4: (warning) failing page number at 0xe219a2
can not get struct kthread at 0x1`1d9a2040
(there may not be a kernel thread here)

processor 8 claims to be idle
stack trace for event 0
crash event was a panic
q4: (warning) cr_isaddr passed yet page is not present
q4: (warning) cr_isaddr passed yet page is not present
q4: (warning) failing page number at 0x806513
q4: (warning) failing page number at 0x806513
can not get struct frame_marker at 0x0.0x8`06513260
q4: (warning) could not load frame from 0x0.0x8`06513270
q4: (warning) could not load frame from 0x0.0x8`06513270
q4: (warning) cr_isaddr passed yet page is not present
q4: (warning) cr_isaddr passed yet page is not present
q4: (warning) failing page number at 0x806513
q4: (warning) failing page number at 0x806513
can not get struct frame_marker at 0x0.0x8`06513040
panic+0x6c
report_trap_or_int_and_panic+0x94
interrupt+0x208
$ihndlr_rtn+0x0
remove_callout+0x138
invoke_callouts_for_self+0x84
sw_service+0x108
mp_ext_interrupt+0x3ec
ivti_patch_to_nop3+0x0
idle+0xa9c

q4: (warning) cr_isaddr passed yet page is not present
q4: (warning) cr_isaddr passed yet page is not present
q4: (warning) failing page number at 0xb94088
q4: (warning) failing page number at 0xb94088
can not get struct kthread at 0x0`6c088040
(there may not be a kernel thread here)

can not trace running processor w/o a crash event

can not trace running processor w/o a crash event
q4>
q4>
q4>

[Dennis Handly]

I suggest you contact the Response Center to analyze your crash.
I didn't see any patches mentioning "remove_callout".

[Sameer_Nirmal]

Do you have patch PHKL_38430 installed? If no,
maybe the patch PHKL_38430 will help.

[Pradep]

Thanks. I will try this patch. Any more ideas about how Q4 results can be interpreted. any docs for it ?

[Venkatesh BL]

Try this: http://www.unixguide.net/hp/hpuxcrashdump.shtml

[Pradep]

As there is no HPMC, does it mean not hardware issue atleast ?

[Venkatesh BL]

Did you try installing PHKL_38430, as suggested by Sameer?

[Pradep]

I will install the patch this weekend.
the panic happened one time only. so we dont know it will happen again or not.