libp4 (9.152): Opening ./vmunix ./INDEX Loading symbols from ./vmunix Kernel TEXT pages not requested in crashconf Will use an artificial mapping from a.out TEXT pages Command line: ./crashinfo.dat -continue crashinfo (4.23) output Note: An error was encountered while building the thread list. The thread list may not be complete ! ===================== = Table Of Contents = ===================== * Detailed dump and crash configuration information * General Information * Crash Events * Message Buffer (in chronological order) * Memory Globals * Buffer Cache Globals * Swap Information * Global Error Counters / kmem_writes * Network Interfaces * Crash Event / Processor Information * Processor Clock Info * Load Averages * Thread Information * Kernel Patches * Signature WARNING: Possible incomplete coredump ! ===================================================== = Detailed dump and crash configuration information = ===================================================== CLASS PAGES INCLUDED IN DUMP DESCRIPTION -------- ---------- ---------------- ------------------------------------- UNUSED 7217720 no, by default unused pages USERPG 65104 no, by default user process pages BCACHE 388625 no, by default buffer cache pages KCODE 2840 no, by default kernel code pages USTACK 1032 yes, by default user process stacks FSDATA 385 yes, by default file system metadata KDDATA 338009 yes, by default kernel dynamic data KSDATA 370797 yes, by default kernel static data Total pages on system: 8384512 (31.98 GB) Pages that needed to be dumped: 710223 (2.71 GB) DEVICE OFFSET(Kb) SIZE (Kb) LOGICAL VOL. NAME ------------ ---------- ---------- ------------ ------------------------- 31:0x006000 265056 2097152 64:0x000002 LVM vg00/lv2 ---------- 2097152 Total avail dump space: 2097152 ( 524288 pages ) Space for dump headers: - 1044 ( 261 pages ) ========== Total useable dump area: 2096108 ( 524027 pages ) Insufficient dump space configured! Only 73.8% of dump saved!! WARNING: This dump is incomplete. As a consequence, the output of crashinfo is unreliable. Depending on how severely the dump is truncated you might not even get a symbolic stack trace. Please contact HP support referencing CRASHINFO_WARN_DUMP1. ======================= = General Information = ======================= Dump time Wed Feb 28 05:06:32 2007 UTC5 System has been up 61 days, 17 hours, 46 minutes. System Name : HP-UX Node Name : benzer1 Model : 9000/800/S16K-A HP-UX version : B.11.11 (64-bit Kernel) Number of CPU's : 8 Disabled CPU's : 0 CPU type : PCXW+ (875 Mhz) CPU Architecture : PA-RISC 2.0 Load average : 0.00 0.01 0.01 ================ = Crash Events = ================ Note: Crash event 0 was a PANIC ! Panic string : Spinlock deadlock! Note: In the case of a PANIC, normally crash event 0 is the crash event you should concentrate on. There may well be other secondary panics (for example spinlock panics) that have happened as a consequence of the original panic. Note: This appears to be a Spinlock deadlock panic. The panicing processor is most likely the innocent party. What has probably happened is that another processor is holding the lock and has failed to release it in a timely manner. Will try and find the lock and the processor that is holding it.. Sorry - can't access lock pointer in dump... Stack Traces for all other Crash events ======================================= ============== EVENT ============================ = Event #0 is PANIC on CPU #1 = p crash_event_t 0xb9d000 = p rpb_t 0xb96160 = Using pc from pim.wide.rp_rp_hi = 0x2194ac ============== EVENT ============================ SR5=0x04578400 SP RP Return Name 0x400003ffffff1228 0x002194ac panic+0x6c 0x400003ffffff1188 0x001c1a2c too_much_time+0x2b4 0x400003ffffff10d8 0x001c1d88 wait_for_lock+0x1e8 0x400003ffffff0f98 0x0003441c sl_retry+0x1c 0x400003ffffff0f18 0x001855c0 _sleep_one+0x70 0x400003ffffff0d98 0x001b0144 vx_worklist_thread+0x34 0x400003ffffff0cf8 0x00424184 kthread_daemon_startup+0x24 0x400003ffffff0c78 0x00424160 kthread_daemon_startup ============== EVENT ============================ = Event #1 is PANIC on CPU #1 = p crash_event_t 0xb9d030 = p rpb_t 0xb964d0 = Using pc from pim.wide.rp_rp_hi = 0x2194ac ============== EVENT ============================ SR5=0x04578400 SP RP Return Name 0x400003ffffff1c48 0x002194ac panic+0x6c 0x400003ffffff1ba8 0x001c1a2c too_much_time+0x2b4 0x400003ffffff1af8 0x001c1d88 wait_for_lock+0x1e8 0x400003ffffff19b8 0x0003441c sl_retry+0x1c 0x400003ffffff1938 0x00183e6c wakeup_one+0x8c 0x400003ffffff17c8 0x001b00d8 vx_worklist_wakeup+0x38 0x400003ffffff1748 0x001d5134 vx_icache_process+0xe4 0x400003ffffff1668 0x000534c8 vx_fsflushi+0x40 0x400003ffffff1598 0x00824488 vx_sync+0x118 0x400003ffffff14e8 0x0008df38 update+0x70 0x400003ffffff1458 0x0021cf58 panic_boot+0x170 0x400003ffffff13a8 0x0021d844 boot+0xec 0x400003ffffff1228 0x00219594 panic+0x154 0x400003ffffff1188 0x001c1a2c too_much_time+0x2b4 0x400003ffffff10d8 0x001c1d88 wait_for_lock+0x1e8 0x400003ffffff0f98 0x0003441c sl_retry+0x1c 0x400003ffffff0f18 0x001855c0 _sleep_one+0x70 0x400003ffffff0d98 0x001b0144 vx_worklist_thread+0x34 0x400003ffffff0cf8 0x00424184 kthread_daemon_startup+0x24 0x400003ffffff0c78 0x00424160 kthread_daemon_startup ============== EVENT ============================ = Event #2 is TOC on CPU #0 = p crash_event_t 0xb9d060 = p rpb_t 0xb96eb0 = Using pc from pim.wide.rp_pcoq_head_hi = 0x21f22c ============== EVENT ============================ SR5=0x0cc03800 SP RP Return Name 0x400003ffffff1228 0x0021f22c panic_save_register_state+0x144 0x400003ffffff1228 0x002194ac panic+0x6c ============== EVENT ============================ = Event #3 is TOC on CPU #3 = p crash_event_t 0xb9d090 = p rpb_t 0x2421130 = Using pc from pim.wide.rp_pcoq_head_hi = 0x21f22c ============== EVENT ============================ SR5=0x0e756000 SP RP Return Name 0x400003ffffff1228 0x0021f22c panic_save_register_state+0x144 0x400003ffffff1228 0x002194ac panic+0x6c 0x400003ffffff1188 0x001c1a2c too_much_time+0x2b4 0x400003ffffff10d8 0x001c1d88 wait_for_lock+0x1e8 0x400003ffffff0f98 0x0003441c sl_retry+0x1c 0x400003ffffff0f18 0x001855c0 _sleep_one+0x70 0x400003ffffff0d98 0x001b0144 vx_worklist_thread+0x34 0x400003ffffff0cf8 0x00424184 kthread_daemon_startup+0x24 0x400003ffffff0c78 0x00424160 kthread_daemon_startup ============== EVENT ============================ = Event #4 is TOC on CPU #2 = p crash_event_t 0xb9d0c0 = p rpb_t 0x2420a50 = Using pc from pim.wide.rp_pcoq_head_hi = 0x7bff8d8c6c (failed) = Using pc from pim.wide.rp_rp_hi = 0x7bff8d8c68 (failed) = Using pc from pim.wide.rp_gr31_hi = 0x8000008 ============== EVENT ============================ Can't unwind ============== EVENT ============================ = Event #5 is TOC on CPU #4 = p crash_event_t 0xb9d0f0 = p rpb_t 0x2421810 = Using pc from pim.wide.rp_pcoq_head_hi = 0x21f22c ============== EVENT ============================ SR5=0x03661c00 SP RP Return Name 0x400003ffffff1228 0x0021f22c panic_save_register_state+0x144 0x400003ffffff1228 0x002194ac panic+0x6c 0x400003ffffff1188 0x001c1a2c too_much_time+0x2b4 0x400003ffffff10d8 0x001c1d88 wait_for_lock+0x1e8 0x400003ffffff0f98 0x0003441c sl_retry+0x1c 0x400003ffffff0f18 0x001855c0 _sleep_one+0x70 0x400003ffffff0d98 0x001b0144 vx_worklist_thread+0x34 0x400003ffffff0cf8 0x00424184 kthread_daemon_startup+0x24 0x400003ffffff0c78 0x00424160 kthread_daemon_startup ============== EVENT ============================ = Event #6 is TOC on CPU #7 = p crash_event_t 0xb9d120 = p rpb_t 0x2422cb0 = Using pc from pim.wide.rp_pcoq_head_hi = 0x21f22c ============== EVENT ============================ SR5=0x04f60000 SP RP Return Name 0x400003ffffff1228 0x0021f22c panic_save_register_state+0x144 0x400003ffffff1228 0x002194ac panic+0x6c 0x400003ffffff1188 0x001c1a2c too_much_time+0x2b4 0x400003ffffff10d8 0x001c1d88 wait_for_lock+0x1e8 0x400003ffffff0f98 0x0003441c sl_retry+0x1c 0x400003ffffff0f18 0x001855c0 _sleep_one+0x70 0x400003ffffff0d98 0x001b0144 vx_worklist_thread+0x34 0x400003ffffff0cf8 0x00424184 kthread_daemon_startup+0x24 0x400003ffffff0c78 0x00424160 kthread_daemon_startup ============== EVENT ============================ = Event #7 is TOC on CPU #5 = p crash_event_t 0xb9d150 = p rpb_t 0x2421ef0 = Using pc from pim.wide.rp_pcoq_head_hi = 0x21f22c ============== EVENT ============================ SR5=0x09542000 SP RP Return Name 0x400003ffffff1228 0x0021f22c panic_save_register_state+0x144 0x400003ffffff1228 0x002194ac panic+0x6c 0x400003ffffff1188 0x001c1a2c too_much_time+0x2b4 0x400003ffffff10d8 0x001c1d88 wait_for_lock+0x1e8 0x400003ffffff0f98 0x0003441c sl_retry+0x1c 0x400003ffffff0f18 0x001855c0 _sleep_one+0x70 0x400003ffffff0d98 0x001b0144 vx_worklist_thread+0x34 0x400003ffffff0cf8 0x00424184 kthread_daemon_startup+0x24 0x400003ffffff0c78 0x00424160 kthread_daemon_startup ============== EVENT ============================ = Event #8 is TOC on CPU #6 = p crash_event_t 0xb9d180 = p rpb_t 0x24225d0 = Using pc from pim.wide.rp_pcoq_head_hi = 0x21f22c ============== EVENT ============================ SR5=0x0f8b8000 SP RP Return Name 0x400003ffffff1228 0x0021f22c panic_save_register_state+0x144 0x400003ffffff1228 0x002194ac panic+0x6c 0x400003ffffff1188 0x001c1a2c too_much_time+0x2b4 0x400003ffffff10d8 0x001c1d88 wait_for_lock+0x1e8 0x400003ffffff0f98 0x0003441c sl_retry+0x1c 0x400003ffffff0f18 0x001855c0 _sleep_one+0x70 0x400003ffffff0d98 0x001b0144 vx_worklist_thread+0x34 0x400003ffffff0cf8 0x00424184 kthread_daemon_startup+0x24 0x400003ffffff0c78 0x00424160 kthread_daemon_startup =========================================== = Message Buffer (in chronological order) = =========================================== zone configuration on your switch. 1/0/14/0/0: Fabric Name Server rejected GPN_FT query. Reason code: unable to perform request; Explanation: FC-4 Type not registered. This may indicate that the HBA port is not part of a zone. Please check zone configuration on your switch. 1/0/14/0/0: Fabric Name Server rejected GPN_FT query. Reason code: unable to perform request; Explanation: FC-4 Type not registered. This may indicate that the HBA port is not part of a zone. Please check zone configuration on your switch. 1/0/14/0/0: Fabric Name Server rejected GPN_FT query. Reason code: unable to perform request; Explanation: FC-4 Type not registered. This may indicate that the HBA port is not part of a zone. Please check zone configuration on your switch. Spinlock timeout failure: Spinlock timeout failure: Spinlock timeout failure: The spinlock code has NOT failed! Instead, some spinlock Spinlock timeout failure: ThSe psipinnllocok ccokde htaism NeOoutT fafialiulreed:! I nstea STd,hp iesn solpimoneclk o scktpi imcnoeoldueot c hfk aais l TurNheOe: T s fpaiSinlplioencldk co! ocIdkne st tehiaamdse, o NusOtoT m fefa ailsiluperiedn:!l o Ick n s tusTeaihdne, g s spcoiomndele o schpkai snc lofodaeci klh eads tNoO Tr efleaasiel ead !sp inIloncskt esaodo,n s omee nspoiunglohc.k using code has failed to release a spinlock soon enough. The spinlock code has NOT failed! Instead, some spinlock The spinlock code has NOT failed! Instead, some spinlock using code has failed to release a spinlock soon enough. using code has failed to release a spinlock soon enough. Address: 0x000000009c4ce9c0X ; using code has failed to release a spinlock soon enough. Address: 0x000000009c4ce9c0X ; using code has failed to release a spinlock soon enough. using code has failed to release a spinlock soon enough. Address: 0x000000009c4ce9c0X ; Address: 0x000000009c4ce9c0X ; owner 0x000000000242a790X ; Address: 0x000000009c4ce9c0X ; owner 0x000000000242a790X ; Address: 0x000000009c4ce9c0X ; Address: 0x000000009c4ce9c0X ; owner 0x000000000242a790X ; owner 0x000000000242a790X ; lock 0x0000000000000000X ; owner 0x000000000242a790X ; lock 0x0000000000000000X ; owner 0x000000000242a790X ; owner 0x000000000242a790X ; lock 0x0000000000000000X ; lock 0x0000000000000000X ; flag 0x1 lock 0x0000000000000000X ; flag 0x1 lock 0x0000000000000000X ; lock 0x0000000000000000X ; flag 0x1 flag 0x1 next_cpu 0x0 flag 0x1 next_cpu 0x0 flag 0x1 flag 0x1 next_cpu 0x0 next_cpu 0x0 Milliseconds spent spinning =60001 next_cpu 0x0 Milliseconds spent spinning =60001 next_cpu 0x0 next_cpu 0x0 Milliseconds spent spinning =60001 Milliseconds spent spinning =60001 Millseconds/sec = 1000 Milliseconds spent spinning =60001 Millseconds/sec = 1000 Milliseconds spent spinning =60001 Milliseconds spent spinning =60001 Millseconds/sec = 1000 Millseconds/sec = 1000 Millseconds/sec = 1000 Millseconds/sec = 1000 Millseconds/sec = 1000 linkstamp: Mon Sep 25 20:04:07 EDT 2006 _release_version: @(#) $Revision: vmunix: vw: -proj selectors: CUPI80_BL2000_1108 -c 'Vw for CUPI80_BL2000_1108 build' -- cupi80_bl2000_1108 'CUPI80_BL2000_1108' Wed Nov 8 19:24:56 PST 2000 $ panic: Spinlock deadlock! PC-Offset Stack Trace (read across, top of stack is 1st): 0x002194e0 0x001c1a2c 0x001c1d88 0x0003441c 0x001855c0 0x001b0144 0x00424184 0x00424160 End Of Stack Spinlock timeout failure: The spinlock code has NOT failed! Instead, some spinlock using code has failed to release a spinlock soon enough. Address: 0x000000009c4ce9c0X ; owner 0x000000000242a790X ; lock 0x0000000000000000X ; flag 0x1 next_cpu 0x0 Milliseconds spent spinning =60001 Millseconds/sec = 1000 linkstamp: Mon Sep 25 20:04:07 EDT 2006 _release_version: @(#) $Revision: vmunix: vw: -proj selectors: CUPI80_BL2000_1108 -c 'Vw for CUPI80_BL2000_1108 build' -- cupi80_bl2000_1108 'CUPI80_BL2000_1108' Wed Nov 8 19:24:56 PST 2000 $ panic: Spinlock deadlock! ================== = Memory Globals = ================== Note: "freemem" was resynced with freemem caches: was 7199134, now 7199126 Physical Memory = 8384512 pages (31.98 GB) Free Memory = 7199126 pages (27.46 GB) Average Free Memory = 7199186 pages (27.46 GB) gpgslim = 31744 pages (124.00 MB) lotsfree = 131072 pages (512.00 MB) desfree = 31744 pages (124.00 MB) minfree = 15616 pages (61.00 MB) ======================== = Buffer Cache Globals = ======================== dbc_max_pct = 30 % dbc_min_pct = 5 % dbc current pct = 4.6 % bufpages = 389028 pages (1.48 GB) Number of buf headers = 220932 fixed_size_cache = 1 dbc_parolemem = 0 dbc_stealavg = 0 dbc_ceiling = 389028 pages (1.48 GB) dbc_nbuf = 194514 dbc_bufpages = 389028 pages (1.48 GB) dbc_vhandcredit = 0 orignbuf = 194514 origbufpages = 389028 pages (1.48 GB) Note: It is very likely that the buffer cache on this system is incorrectly configured. The current percentage of buffer cache is smaller than the dbc_min_pct value. Please check if this configuration is intentionally this way. If you should have any processes waiting in bfreelist (check threads listing) you should strongly consider increasing the minimum amount of buffer cache space. Results of this setting can be system hangs or very bad performance. The most likely reason for this misconfiguration is that the system file still contains either one of the "nbuf" or "bufpages" kernel tunables. buffer free list counts (bfree_cnts) ==================================== index BQ_LRU BQ_AGE BQ_EMPTY BQ_SENDFILE ----- --------- --------- --------- ----------- 0 35999744 141291520 0 0 1 77008896 120778752 0 0 2 67600384 104947712 0 0 3 25604096 152829952 0 0 4 96169984 96780288 0 0 5 94855168 139771904 0 0 6 90198016 115568640 0 0 7 95559680 138416128 0 0 ==================== = Swap Information = ==================== swapinfo -mt emulation ====================== Mb Mb Mb PCT START/ Mb TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME dev 2048 0 2048 0% 0 - 1 LVM vg00/lv2 reserve - 288 -288 memory 25598 1385 24213 5% total 27646 1673 25973 6% - 0 - ======================================= = Global Error Counters / kmem_writes = ======================================= lpmc_count = 6735 Note: we have LPMC's Attempting to print lpmc_log data LPMC's Note: we will probably only be able to print data for the most recent LPMC. lpmc_log ======== hversion hpa cpu pim_size pim_ptr -------- ------------------ --- -------- ------- 0x5eb0 0xfffffffffc27c000 2 584 0x2494898 pim data ======== check_type : 0x80000000 hversion : 0x8 cache_check : 0x40000000 tlb_check : 0x0 bus_check : 0x0 assts_check : 0x0 assts_state : 0x0 path_info : 0x0 resp_addr : 0x0 req_addr : 0x0 hversion hpa cpu pim_size pim_ptr -------- ------------------ --- -------- ------- 0x5eb0 0xfffffffffc27c000 2 584 0x2494ae0 pim data ======== check_type : 0x80000000 hversion : 0x8 cache_check : 0x40000000 tlb_check : 0x0 bus_check : 0x0 assts_check : 0x0 assts_state : 0x0 path_info : 0x0 resp_addr : 0x0 req_addr : 0x0 hversion hpa cpu pim_size pim_ptr -------- ------------------ --- -------- ------- 0x5eb0 0xfffffffffc27c000 2 584 0x2494d28 pim data ======== check_type : 0x80000000 hversion : 0x8 cache_check : 0x40000000 tlb_check : 0x0 bus_check : 0x0 assts_check : 0x0 assts_state : 0x0 path_info : 0x0 resp_addr : 0x0 req_addr : 0x0 hversion hpa cpu pim_size pim_ptr -------- ------------------ --- -------- ------- 0x5eb0 0xfffffffffc27c000 2 584 0x2494f70 pim data ======== check_type : 0x80000000 hversion : 0x8 cache_check : 0x40000000 tlb_check : 0x0 bus_check : 0x0 assts_check : 0x0 assts_state : 0x0 path_info : 0x0 resp_addr : 0x0 req_addr : 0x0 hversion hpa cpu pim_size pim_ptr -------- ------------------ --- -------- ------- 0x5eb0 0xfffffffffc27c000 2 584 0x24951b8 pim data ======== check_type : 0x80000000 hversion : 0x8 cache_check : 0x40000000 tlb_check : 0x0 bus_check : 0x0 assts_check : 0x0 assts_state : 0x0 path_info : 0x0 resp_addr : 0x0 req_addr : 0x0 hversion hpa cpu pim_size pim_ptr -------- ------------------ --- -------- ------- 0x5eb0 0xfffffffffc27c000 2 584 0x2495400 pim data ======== check_type : 0x80000000 hversion : 0x8 cache_check : 0x40000000 tlb_check : 0x0 bus_check : 0x0 assts_check : 0x0 assts_state : 0x0 path_info : 0x0 resp_addr : 0x0 req_addr : 0x0 hversion hpa cpu pim_size pim_ptr -------- ------------------ --- -------- ------- 0x5eb0 0xfffffffffc27c000 2 584 0x2495648 pim data ======== check_type : 0x80000000 hversion : 0x8 cache_check : 0x40000000 tlb_check : 0x0 bus_check : 0x0 assts_check : 0x0 assts_state : 0x0 path_info : 0x0 resp_addr : 0x0 req_addr : 0x0 hversion hpa cpu pim_size pim_ptr -------- ------------------ --- -------- ------- 0x5eb0 0xfffffffffc27c000 2 584 0x2495890 pim data ======== check_type : 0x80000000 hversion : 0x8 cache_check : 0x40000000 tlb_check : 0x0 bus_check : 0x0 assts_check : 0x0 assts_state : 0x0 path_info : 0x0 resp_addr : 0x0 req_addr : 0x0 hversion hpa cpu pim_size pim_ptr -------- ------------------ --- -------- ------- 0x5eb0 0xfffffffffc27c000 2 584 0x2495ad8 pim data ======== check_type : 0x80000000 hversion : 0x8 cache_check : 0x40000000 tlb_check : 0x0 bus_check : 0x0 assts_check : 0x0 assts_state : 0x0 path_info : 0x0 resp_addr : 0x0 req_addr : 0x0 hversion hpa cpu pim_size pim_ptr -------- ------------------ --- -------- ------- 0x5eb0 0xfffffffffc27c000 2 584 0x2495d20 pim data ======== check_type : 0x80000000 hversion : 0x8 cache_check : 0x40000000 tlb_check : 0x0 bus_check : 0x0 assts_check : 0x0 assts_state : 0x0 path_info : 0x0 resp_addr : 0x0 req_addr : 0x0 hversion hpa cpu pim_size pim_ptr -------- ------------------ --- -------- ------- 0x5eb0 0xfffffffffc27c000 2 584 0x2495f68 pim data ======== check_type : 0x80000000 hversion : 0x8 cache_check : 0x40000000 tlb_check : 0x0 bus_check : 0x0 assts_check : 0x0 assts_state : 0x0 path_info : 0x0 resp_addr : 0x0 req_addr : 0x0 hversion hpa cpu pim_size pim_ptr -------- ------------------ --- -------- ------- 0x5eb0 0xfffffffffc27c000 2 584 0x24961b0 pim data ======== check_type : 0x80000000 hversion : 0x8 cache_check : 0x40000000 tlb_check : 0x0 bus_check : 0x0 assts_check : 0x0 assts_state : 0x0 path_info : 0x0 resp_addr : 0x0 req_addr : 0x0 hversion hpa cpu pim_size pim_ptr -------- ------------------ --- -------- ------- 0x5eb0 0xfffffffffc27c000 2 584 0x24963f8 pim data ======== check_type : 0x80000000 hversion : 0x8 cache_check : 0x40000000 tlb_check : 0x0 bus_check : 0x0 assts_check : 0x0 assts_state : 0x0 path_info : 0x0 resp_addr : 0x0 req_addr : 0x0 hversion hpa cpu pim_size pim_ptr -------- ------------------ --- -------- ------- 0x5eb0 0xfffffffffc27c000 2 584 0x2496640 pim data ======== check_type : 0x80000000 hversion : 0x8 cache_check : 0x40000000 tlb_check : 0x0 bus_check : 0x0 assts_check : 0x0 assts_state : 0x0 path_info : 0x0 resp_addr : 0x0 req_addr : 0x0 hversion hpa cpu pim_size pim_ptr -------- ------------------ --- -------- ------- 0x5eb0 0xfffffffffc27c000 2 584 0x2496888 pim data ======== check_type : 0x80000000 hversion : 0x8 cache_check : 0x40000000 tlb_check : 0x0 bus_check : 0x0 assts_check : 0x0 assts_state : 0x0 path_info : 0x0 resp_addr : 0x0 req_addr : 0x0 hversion hpa cpu pim_size pim_ptr -------- ------------------ --- -------- ------- 0x5eb0 0xfffffffffc27c000 2 584 0x2496ad0 pim data ======== check_type : 0x80000000 hversion : 0x8 cache_check : 0x40000000 tlb_check : 0x0 bus_check : 0x0 assts_check : 0x0 assts_state : 0x0 path_info : 0x0 resp_addr : 0x0 req_addr : 0x0 scsi_bioerrors = 225 scsi_bioerrors_logged = 0 scsi_async_write_bioerrors = 0 scsi_async_write_bioerrors_logged = 0 sd_strategy_bioerrors = 225 sd_strategy_async_write_bioerrors = 0 sd_epowerf_bioerrors = 0 sd_epowerf_async_write_bioerrors = 0 Note: There are scsi_bioerrors ! Please contact HP support referencing CRASHINFO_NOTE_SCSI1. ====================== = Network Interfaces = ====================== WARNING : Cannot retrieve list of interfaces hw_ift_list address: 0x9d1c8040 seems to be invalid WARNING : Dump doesn't seem to be complete. Sorry cannot continue ... ADVICE : Be sure that that this dump is complete ======================================= = Crash Event / Processor Information = ======================================= Number of processors = 8 s t a spin reg eiem_hi eirr ipsw evt cpu t type dpth src cr15_hi spl cr23 cr22 --- --- - ----- ---- --- -------- -------- ---------------- ---------------- 0 1 E PANIC 0 rpb c5000000 SPLSYS 0800000000020000 0800000f WQPDI 1 1 E PANIC 0 rpb c5000000 SPLSYS 0800000000000000 0800000b WQDI 2 0 E TOC 0 rpb c4000000 SPLSYS 8800000100010000 08040008 WCQ 3 3 E TOC 0 rpb c5000000 SPLSYS 8800000100000000 08040008 WCQ 4 2 E TOC 1 rpb c4000000 SPLSYS 0800000900000000 08000008 WQ mpi fffffff0 SPLSCHED 5 4 E TOC 0 rpb c5000000 SPLSYS 8800000100000000 08040008 WCQ 6 7 E TOC 0 rpb c5000000 SPLSYS 8800000100000000 08040008 WCQ 7 5 E TOC 0 rpb c5000000 SPLSYS 8800000900000000 08040008 WCQ 8 6 E TOC 0 rpb c5000000 SPLSYS 8800000100000000 08040008 WCQ Note: PSW_I bit on processor 0 is off ! (Interrupts are disabled) Note: PSW_I bit on processor 3 is off ! (Interrupts are disabled) Note: PSW_I bit on processor 2 is off ! (Interrupts are disabled) Note: PSW_I bit on processor 4 is off ! (Interrupts are disabled) Note: PSW_I bit on processor 7 is off ! (Interrupts are disabled) Note: PSW_I bit on processor 5 is off ! (Interrupts are disabled) Note: PSW_I bit on processor 6 is off ! (Interrupts are disabled) SPL/EIEM values: 0xffffffffffffffff = SPLUSER Default user mode SPL level. 0xfffffff0ffffffff = SPLSCHED Disable scheduling interrupts. 0xffffff00ffffffff = SPLSOFT Disable software interrupts (sw triggers off). 0xed00080000000000 = SPLIO Disable IO modules. 0xc500000000000000 = SPLSYS Enable clock-resync. 0xc400000000000000 = SPLSYS Disable system events (eg. hardclock). 0x0000000700000000 = SPL7 Disable the world (eg. PSW_I=0). ======================== = Processor Clock Info = ======================== hardclock_late = 13 itick_per_tick = 8750000 lbolt = 533459337 (0x1fcbf189) event mpi rpb delta clk eiem eirr PSW cpu type timeinval interval timer secs:ticks od 0,4 0,4 I --- ----- ------------------ ------------------ ---------- --- ---- ---- --- 0 TOC 0x1095aa37cbf55d 0x1095c2cb6f5d5b -120:-63 13 1 0 1 1 0 1 PANIC 0x1095b9997e73f8 0x1095b99833d40c 0:2 13 1 0 0 1 1 2 TOC 0x1095ad57e36958 0x1095c5ebff69f3 -120:-64 25 1 0 0 1 0 3 TOC 0x1095ad53b5a9d7 0x1095c5e73d5fff -120:-63 13 1 0 1 1 0 4 TOC 0x1095a9af131e96 0x1095c242a01d7f -120:-63 13 1 0 1 1 0 5 TOC 0x1095acd6fc4d20 0x1095c56a896e07 -120:-63 13 1 0 1 1 0 6 TOC 0x1095accfaa3392 0x1095c563372b4b -120:-63 13 1 0 1 1 0 7 TOC 0x1095accb7c59a2 0x1095c55f09a59f -120:-63 13 1 0 1 1 0 Note: For the PANIC crash event type it is normal to see a positive delta ticks. This is due to the processor executing dumpsys() code after creating it's crash event and rpb. WARNING: Processor 0 appears to have had clock interrupts held off for approx 120 seconds. Current SPL = 0xc400000000000000 (SPLSYS). WARNING: Processor 2 appears to have had clock interrupts held off for approx 120 seconds. Current SPL = 0xc400000000000000 (SPLSYS). WARNING: Processor 3 appears to have had clock interrupts held off for approx 120 seconds. Current SPL = 0xc500000000000000 (SPLSYS). WARNING: Processor 4 appears to have had clock interrupts held off for approx 120 seconds. Current SPL = 0xc500000000000000 (SPLSYS). WARNING: Processor 5 appears to have had clock interrupts held off for approx 120 seconds. Current SPL = 0xc500000000000000 (SPLSYS). WARNING: Processor 6 appears to have had clock interrupts held off for approx 120 seconds. Current SPL = 0xc500000000000000 (SPLSYS). WARNING: Processor 7 appears to have had clock interrupts held off for approx 120 seconds. Current SPL = 0xc500000000000000 (SPLSYS). ================= = Load Averages = ================= avenrun ======= 0.00 0.01 0.01 mp_avenrun ========== cpu0 : 0.002587 0.004028 0.004389 cpu1 : 0.009276 0.007605 0.009074 cpu2 : 0.001807 0.003684 0.005344 cpu3 : 0.002528 0.005232 0.006577 cpu4 : 0.002762 0.006840 0.010594 cpu5 : 0.005633 0.014350 0.013877 cpu6 : 0.006269 0.007029 0.008151 cpu7 : 0.005933 0.007693 0.007259 ====================== = Thread Information = ====================== 1 Threads ran in the last second 1 Threads ran in the last 5 seconds 1 Threads ran in the last 10 seconds 1 Threads ran in the last minute 1 Threads ran in the last hour statdaemon ran 0 ticks ago Most Common Wait Channels ========================= ticks since run: Wait Channel count longest shortest ------------ ----- ---------- ---------- Most Common Sleep Callers ========================= ticks since run: Sleep Caller count longest shortest ------------ ----- ---------- ---------- Idle Globals ============ candidate_idle_spu = -1 migration_cycles = 1500000 Running Threads (TSRUNPROC) and idle Processors =============================================== TICKS TICKS I TICKS SINCE SINCE C SINCE NREADY TID PID PPID RUN IDLE PRI SPU STATE S MIGR FR LO AL COMMAND ------- ----- ----- --------- ----- --- --- ----- - ----- -- -- -- ------- 0 0 0 533459337 0 -512 0 SYS N 200 4 0 0 0 0 0 533459337 0 -512 1 SYS N 0 1 0 0 0 0 0 533459337 0 -512 2 SYS N 3 0 0 0 0 0 0 533459337 0 -512 3 SYS N 200 3 0 0 0 0 0 533459337 0 -512 4 SYS N 0 2 0 0 0 0 0 533459337 78 -512 5 SYS N 100 7 0 0 0 0 0 533459337 0 -512 6 SYS N 0 4 0 0 0 0 0 533459337 0 -512 7 SYS N 0 3 0 0 Note: FR: free to run on any processor (candidate for thread migration). LO: locked (via processor affinity/mpctl) to this processor). AL: Alpha semaphores misses (special scheduling when miss a sema). Threads waiting on cpu (TSRUN) - sorted by cpu/pri/ticks-since-run ================================================================== All Threads - sorted by ticks-since-run ======================================= TID PID PPID TICKS PRI SPU STAT SYSCALL COMMAND WCHAN ------- ----- ----- ----- --- --- ----- -------------- -------------- ----- 0 0 0 34 127 7 SLEEP n/a swapper ticks_since_boot ================== = Kernel Patches = ================== PHKL_22874 PHKL_22994 PHKL_23196 PHKL_23225 PHKL_23250 PHKL_23335 PHKL_23400 PHKL_23401 PHKL_23445 PHKL_23642 PHKL_23810 PHKL_24026 PHKL_24046 PHKL_24047 PHKL_24056 PHKL_24188 PHKL_24253 PHKL_24278 PHKL_24343 PHKL_24408 PHKL_24548 PHKL_24549 PHKL_24552 PHKL_24554 PHKL_24555 PHKL_24556 PHKL_24565 PHKL_24582 PHKL_24585 PHKL_24626 PHKL_24652 PHKL_24730 PHKL_24751 PHKL_25022 PHKL_25166 PHKL_25212 PHKL_25238 PHKL_25248 PHKL_25304 PHKL_25375 PHKL_25468 PHKL_25493 PHKL_25552 PHKL_25582 PHKL_25593 PHKL_25602 PHKL_25610 PHKL_25728 PHKL_25760 PHKL_25885 PHKL_25886 PHKL_25993 PHKL_25994 PHKL_26032 PHKL_26074 PHKL_26104 PHKL_26193 PHKL_26196 PHKL_26425 PHKL_26502 PHKL_26705 PHKL_26719 PHKL_26755 PHKL_26785 PHKL_26833 PHKL_27025 PHKL_27155 PHKL_27172 PHKL_27179 PHKL_27219 PHKL_27321 PHKL_27498 PHKL_27502 PHKL_27532 PHKL_27682 PHKL_27688 PHKL_27734 PHKL_27737 PHKL_27750 PHKL_27757 PHKL_27825 PHKL_27971 PHKL_28062 PHKL_28114 PHKL_28117 PHKL_28122 PHKL_28139 PHKL_28185 PHKL_28197 PHKL_28237 PHKL_28264 PHKL_28267 PHKL_28323 PHKL_28326 PHKL_28384 PHKL_28459 PHKL_28474 PHKL_28475 PHKL_28489 PHKL_28512 PHKL_28569 PHKL_28695 PHKL_28794 PHKL_28839 PHKL_28842 PHKL_28984 PHKL_28990 PHKL_29045 PHKL_29076 PHKL_29115 PHKL_29335 PHKL_29527 PHKL_29529 PHKL_29567 PHKL_29611 PHKL_29650 PHKL_29651 PHKL_29696 PHKL_29704 PHKL_29818 PHKL_29826 PHKL_29911 PHKL_29921 PHKL_29981 PHKL_30034 PHKL_30038 PHKL_30105 PHKL_30151 PHKL_30216 PHKL_30218 PHKL_30219 PHKL_30257 PHKL_30259 PHKL_30306 PHKL_30317 PHKL_30366 PHKL_30398 PHKL_30446 PHKL_30449 PHKL_30516 PHKL_30523 PHKL_30524 PHKL_30539 PHKL_30542 PHKL_30557 PHKL_30579 PHKL_30582 PHKL_30590 PHKL_30596 PHKL_30606 PHKL_30607 PHKL_30621 PHKL_30681 PHKL_30756 PHKL_30796 PHKL_30837 PHKL_30904 PHKL_30992 PHKL_31003 PHKL_31036 PHKL_31212 PHKL_31227 PHKL_31239 PHKL_31241 PHKL_31876 PHKL_31905 PHKL_31993 PHKL_31994 PHKL_32002 PHKL_32005 PHKL_32012 PHKL_32035 PHKL_32037 PHKL_32038 PHKL_32039 PHKL_32061 PHKL_32089 PHKL_32090 PHKL_32119 PHKL_32126 PHKL_32204 PHKL_32235 PHKL_32375 PHKL_32457 PHKL_32550 PHKL_32578 PHKL_32603 PHKL_32619 PHKL_32647 PHKL_32668 PHKL_32669 PHKL_32715 PHKL_32927 PHKL_33047 PHKL_33066 PHKL_33137 PHKL_33200 PHKL_33258 PHKL_33328 PHKL_33336 PHKL_33338 PHKL_33339 PHKL_33356 PHKL_33367 PHKL_33371 PHKL_33372 PHKL_33377 PHKL_33387 PHKL_33390 PHKL_33408 PHKL_33436 PHKL_33590 PHKL_33702 PHKL_33843 PHKL_33844 PHKL_33988 PHKL_34024 PHKL_34039 PHKL_34093 PHKL_34100 PHKL_34105 PHKL_34106 PHKL_34110 PHKL_34116 PHKL_34153 PHKL_34161 PHKL_34167 PHKL_34173 PHKL_34187 PHKL_34193 PHKL_34254 PHKL_34296 PHKL_34309 PHKL_34310 PHKL_34311 PHKL_34459 PHKL_34478 PHKL_34518 PHKL_34534 PHKL_34548 PHKL_34552 PHKL_34595 PHKL_34665 PHNE_22878 PHNE_22963 PHNE_23502 PHNE_23594 PHNE_24035 PHNE_24211 PHNE_24492 PHNE_24506 PHNE_24910 PHNE_25083 PHNE_25627 PHNE_25644 PHNE_27170 PHNE_27218 PHNE_28137 PHNE_28444 PHNE_28895 PHNE_28923 PHNE_29320 PHNE_29825 PHNE_29883 PHNE_30086 PHNE_30103 PHNE_30378 PHNE_30463 PHNE_30661 PHNE_30695 PHNE_31091 PHNE_31097 PHNE_31153 PHNE_31929 PHNE_32396 PHNE_32477 PHNE_32616 PHNE_32811 PHNE_33138 PHNE_33159 PHNE_33313 PHNE_33315 PHNE_33481 PHNE_33498 PHNE_33628 PHNE_33704 PHNE_33729 PHNE_33971 PHNE_34135 PHSS_31326 Note: An unofficial patch appears to be installed on this system. Unofficial patch listing ======================== volhp.c $Date: 2006/01/12 10:56:02 $ $Revision: 1.83.66.15.2.18 $ PATCH_11.11 (UNOF_3_5p8kpp3) ============= = Signature = ============= 11.11 and panic and "Spinlock deadlock!" and lpmc