Reboot after panic on 9000/785/C360 (HP-UX 10.20)

[Vlad Kluew]

I posted my question to Workstations->hp-ux based, but may be it was not a right place for it, so sorry, but here I'm again.

We have a number of 9000/785/C360s running HP-UX 10.20. All machines have identical configuration. Each machine has FDDI LAN card in addition to built-in LAN interface. The built-in LAN interface is not up for all the time.
It has been noticed that C360s could perform a spontaneous reboot at the time when there is excessive network traffic: a some kind of flood of ICMP and/or SNMP datagrams.

Here is what the crash dump analysis showed:
-----------------------------------------------

System Name: HP-UX
Node Name: xxxxxxx01
Release: 10.20
Version: A
Model: 9000/785/C360
Machine ID: 2014383000
Processors: 1
Architecture: PA-RISC 2.0
CPU is a: PCXU/PA-8000
Physical Mem: 256.00 MBytes
The system had been up for 9.55 days (82528525 ticks).
Load averages: 3.66 2.14 2.08.
System went down at: Mon Sep 22 06:14:51 2003
+--------------------------------------+
| Message Buffer |
+--------------------------------------+
vuseg=173d000
inet_clts:ok inet_cots:ok nfs_init3 added vfs type nfs3 at slot 5
NOTICE: cachefs_link(): File system was registered at index 6.
NOTICE: autofs_link(): File system was registered at index 7.
8 ccio
8/0 GSCtoPCI
BCX ATTACH entered vendor id 0x105d dev id 0x2339
8/0/1/0 img
8/0/3/0 ifi
8/0/19/0 c720
8/0/19/0.6 tgt
8/0/19/0.6.0 sdisk
8/0/19/0.7 tgt
8/0/19/0.7.0 sctl
8/0/20/0 btlan3
8/0/63 asio0
8/16 bus_adapter
8/16/4 asio0
8/16/5 c720
8/16/5.7 tgt
8/16/5.7.0 sctl
8/16/0 CentIf
8/16/1 audio
8/16/7 ps2
8/16/10 fdc
8/16/10.1 pflop
10 ccio
10/0 GSCtoPCI
32 processor
49 memory
System Console is on the Built-In Serial Interface
btlan3: Initializing 10/100BASE-TX card at 8/0/20/0....
Unable to allocate all equivalent entries (0)
Networking memory for fragment reassembly is restricted to 20393984 bytes
Logical volume 64, 0x3 configured as ROOT
Logical volume 64, 0x9 configured as SWAP
Logical volume 64, 0x9 configured as DUMP
Swap device table: (start & size given in 512-byte blocks)
entry 0 - major is 64, minor is 0x9; start = 0, size = 1572864
Dump device table: (start & size given in 1-Kbyte blocks)
entry 0 - major is 31, minor is 0x6000; start = 629599, size = 262145
Starting the STREAMS daemons.
B2352B HP-UX (B.10.20) #1: Sun Jun 9 08:03:38 PDT 1996
Memory Information:
physical page size = 4096 bytes, logical page size = 4096 bytes
Physical: 262144 Kbytes, lockable: 177876 Kbytes, available: 190236 Kbytes
interrupt type 15, pcsq.pcoq = 0.3e5ab0, isr.ior = 0.51
savestate ptr = 0x42ca90, savestate return ptr = 0x3e5a0c
B2352B HP-UX (B.10.20) #1: Sun Jun 9 08:03:38 PDT 1996
panic: (display==0xb800, flags==0x0) Data page fault
PC-Offset Stack Trace (read across, most recent is 1st):
0x002467bc 0x0021ee00 0x0021f8d0 0x00229300 0x003e5ab0 0x003ddecc
0x003dcc68 0x003e7018 0x003e5ed0 0x003ddd40 0x003d9b74 0x003e7730
0x001e3988 0x00050c4c 0x00128600 0x001115e8 0x000d9f64 0x000d8d0c
0x0004e808
End Of Stack
NOT sync'ing disks (on the ICS) (0 buffers to flush):
0 buffers not flushed
90 buffers still dirty
+--------------------------------------+
| Processor activity |
+--------------------------------------+
Processor 0 started it by panic'ing. Here is the stack trace:
stack trace for event 0
crash event was a panic
The Save State registers for this level are:

r0 /r1 /r2 0x0 0x1 0x246790
r3 /r4 /r5 0x2 0xf 0x31000
r6 /r7 /r8 0x42ca90 0x51 0x4a9810
r9 /r10/r11 0xe0 0x48c8e8 0x40
r12/r13/r14 0x40 0x3d 0x1aca000
r15/r16/r17 0x1aca000 0x1 0x1ac87c0
r18/r19/r20 0x60c040 0x3fe 0x0
r21/r22/r23 0x726e2070 0x450668 0xa
r24/r25/r26 0x0 0xa 0x3148c
r27/r28/r29 0x48e0e8 0x7 0x2
r30/r31/r32 0x42d048 0x3e
sr0 /sr1 /sr2 0x0 0x7437400 0x0
sr3 /sr4 /sr5 0x0 0x0 0x2793000
sr6 /sr7 /sr8 0x1e00800 0x0
LEVEL FUNC ARG0 ARG1 ARG2 ARG3
lev 0) panic+0x10 n/a n/a n/a n/a
lev 1) report_trap_or_int_and_panic+0xe8 0x2 0xf 0x42ca90 n/a
lev 2) interrupt+0x458 n/a 0x42ca90 n/a n/a
lev 3) $ihndlr_rtn+0x0 n/a n/a n/a n/a
lev 4) IPh8_2_1_bdm_rcv_buf_alloc+0x218 0x1ace700 0x1000 n/a n/a
lev 5) IPh8_2_1_motofsi_postreads+0x5c 0x1aca000 n/a n/a n/a
lev 6) IPh8_2_1_motofsi_rcv_done_cb+0x1f8 0x1aca000 0x1ac87c0 n/a n/a
lev 7) IPh8_2_1_ipcommon_cache_sync+0x130 0x1ac9200 0x1a0c580 0x1b55003 0x3d
lev 8) IPh8_2_1_bdm_rcv_buf_sync+0x1c0 0x1ace700 0x1ac87c0 n/a 0x10001
lev 9) IPh8_2_1_motofsi_rcv_done+0x7e8 0x1aca000 n/a n/a n/a
lev 10) IPh8_2_1_motofsi_isr_pci+0x84 0x1aca000 n/a n/a n/a
lev 11) IPh8_2_1_ipcommon_isr_func+0x38 0x1742a00 0x1ac9200 0x0 0x0
lev 12) dino_isr+0x230 n/a 0x42c030 n/a n/a
lev 13) inttr_emulate_save_fpu+0xf0 n/a n/a n/a n/a
lev 14) soo_select+0x28 0x121d320 0x1 n/a n/a
lev 15) selscan+0x128 0x7ffe6a70 0x7ffe6a7c 0x7f n/a
lev 16) select+0x6bc n/a n/a n/a n/a
lev 17) syscall+0x75c n/a n/a n/a n/a
lev 18) $syscallrtn+0x0 n/a n/a n/a n/a
Processor 0: servicing interrupt
-------------
can not find unwind or stub descriptor for
pc==0x0`0053ffcc
-------------
Attention, immediate reporting for WSIO disks switched on!!
We have logged 1 request to deallocate memory pages.
Please check the diagnostic logs, it is most likely that a memory board
needs to be exchanged.

--------------------------------------
I checked WSIO and memory and found certain problems there. But I still have a few questions:
1) What is a functional description of IPh8_2_1_xxx group of functions? I realize what IP could mean, but to have more details would be great.
2) Could the "Data page fault" be a not having appropriate patch installed problem? If it could what patches should I check?
3) Could you please share any idea what happened on this particular crash?

Any help would be highly appreciated. I hope somebody would already have gone through this exersice.

Thanks,
Vlad

[Michael Steele_2 · Accepted solution]

1) ?

2) maintenance and support > individual patches > hp-ux > In step one specify 700 > 10.20 > 'data page fault'

There are 68 patches in 10.20 for 'data page fault'.

http://www1.itrc.hp.com/service/patch/search.do

3) Since you're panic'ing when the boot disk is referenced:

"...B2352B HP-UX (B.10.20) #1: Sun Jun 9 08:03:38 PDT 1996
panic: (display==0xb800, flags==0x0) Data page fault...."

...verify your bootlif and O/S.

Do you have a recovery CD? If so then try rebuilding the bootlif. See attachement.

Got ignite?

[Vlad Kluew]

Yes, I got ignite. As I said, all C360s have identical configuration â that means HW and COTS SW. Unfortunately, I [kind of] â inheritedâ ignite. I mean I got it in the form â ready to goâ and everybody was happy until I was testing some software that created excessive ICMP flood and some machines started falling down. Without it, that reboot after panic has never been seeing before. Theoretically, it should not be a situation that some critical patches are missing. But, what if some are? Resent fell down [SNMP flood] of one of C360s, while a number of others in the SAME HW and SW configuration on the SAME network survived, made me thinking that this is a situation when some critical patch or patches need to be applied, because trace event 0 in all cases pointed on

â lev 4) IPh8_2_1_bdm_rcv_buf_alloc+0x218 0x1ace700 0x1000 n/a n/a
lev 5) IPh8_2_1_motofsi_postreads+0x5c 0x1aca000 n/a n/a n/a
lev 6) IPh8_2_1_motofsi_rcv_done_cb+0x1f8 0x1aca000 0x1ac87c0 n/a n/a
lev 7) IPh8_2_1_ipcommon_cache_sync+0x130 0x1ac9200 0x1a0c580 0x1b55003 0x3d
lev 8) IPh8_2_1_bdm_rcv_buf_sync+0x1c0 0x1ace700 0x1ac87c0 n/a 0x10001
lev 9) IPh8_2_1_motofsi_rcv_done+0x7e8 0x1aca000 n/a n/a n/a
lev 10) IPh8_2_1_motofsi_isr_pci+0x84 0x1aca000 n/a n/a n/a
lev 11) IPh8_2_1_ipcommon_isr_func+0x38 0x1742a00 0x1ac9200 0x0 0x0â

and that makes sense. Remember: ICMP/SNMP bombarding? Classical DofS failure. But, cumulative ARPA patch has been applied by my predecessors event twice! Thatâ s why Iâ m looking for information/documentation/anything that would let me understand better what is this group of functions like IPh8_2_1_bdm_rcv_buf_alloc and Co. doing that makes some of identically configured C360s so unhappy? Thatâ s my real question. On all failed machines one part of crash dump is common and related to those mentioned IPh8_2_1_* functions. One machine, which log I posted in my first message, reported problems with memory board and outstanding I/Os on SCSI interface for boot disk (this machine has only one HDD). No tombstones. Thatâ s why Iâ m still thinking about missing patches. Thank you, Michael, that you pointed me on â Data page faultâ patches, but Iâ ve found 45 only so far. And could you, please, tell a little bit more what you start

[Michael Steele_2]

I don't see the connection with a icmp denial of service attack, so keep at it.

Run this command and report the number of total errors in swverify.log:

# swverify -v -x check_permissions=true \*

Also check your system this way:

# swlist -l fileset -a state | grep -i -v -e \# -e configured

#############################################
############################################

Here are the 68 patches, make sure that you reference HP 9000/700 and not HP 9000/8000. 800 is for servers and you want 700 workstattions:

Also see WORD doc. attached.


s700_800 10.X HP Disk Array Utilities w/AutoRAID Manager PHCO_23261
notes PHCO_23261
notes
s700_800 10.20 libc cumulative patch PHCO_23684
notes PHCO_26158
notes
s700_800 10.20 lpspool subsystem cumulative patch PHCO_27133
notes PHCO_27133
notes
s700 10.20 Illegal vnode op, bad dir on CD_ROM cause panic PHKL_14288
notes PHKL_14288
notes
s700 10.20 Data page fault from lpmc on PA 2.0 systems PHKL_14375
notes PHKL_14375
notes
s700 10.20 VxFS Quota/Attr. Fix For Panic During Log Replay PHKL_15952
notes PHKL_15952
notes
s700 10.20 SIG_IGN/SIGCLD,LVM,JFS,PCI/SCSI cumulative patch PHKL_16750
notes PHKL_16750
notes
s700 10.20 proc_open synchronization fixes PHKL_17059
notes PHKL_17059
notes
s700 10.20 Fix data corruption,diaglogd prob,drv log issue PHKL_17499
notes PHKL_17499
notes
s700 10.20 Correct VxFS fsadm resize problems PHKL_18913
notes PHKL_18913
notes
s700 10.20 Correct panics when using quotas on HFS PHKL_19235
notes PHKL_19235
notes
s700 10.20 VxFS: kdm cumulative patch PHKL_20096
notes PHKL_20096
notes
s700 10.20 Cumulative USB Driver patch PHKL_20477
notes PHKL_20477
notes
s700 10.20 Data page fault in vx_map_delayflush() PHKL_20727
notes PHKL_20727
notes
s700 10.20 Multiprocessor graphics/interrupts/PM fixes PHKL_21933
notes PHKL_21933
notes
s700 10.20 Data page fault in remove_callout() fix PHKL_22328
notes PHKL_22328
notes
s700 10.20 stape cumulative patch PHKL_22854
notes PHKL_22854
notes
s700 10.20 mpctl(2) negative SPU check PHKL_23284
notes PHKL_23284
notes
s700 10.20 JFS cumulative patch PHKL_23837
notes PHKL_23837
notes
s700 10.20 HFS Panic moving directories & Hang in lockorder PHKL_23961
notes PHKL_23961
notes
s700 10.20 Advanced VxFS B.10.20 cumulative patch PHKL_24065
notes
s700 10.20 VxFS fsadm, bmap, performance cumulative patch PHKL_18670
notes PHKL_24388
notes
s700 10.20 FXE,Visualize,PCI graphics;ITE;HIL;PS/2;syslog PHKL_23460
notes PHKL_25246
notes
s700 10.20 LVM cumulative patch, array controller hot-swap PHKL_22528
notes PHKL_25438
notes
s700 10.20 Audio driver repair and support patch PHKL_26004
notes PHKL_26004
notes
s700 10.20 SCSI cumulative patch PHKL_22223
notes PHKL_26397
notes
s700 10.20 Cumulative mmap,mprotect,protid;SIGBUS in STDBY PHKL_21925
notes PHKL_26894
notes
s700 10.20 Cumulative mmap,mprotect,protid;SIGBUS in STDBY PHKL_17193
notes PHKL_26894
notes
s700 10.20 Cumulative mmap,mprotect,protid;SIGBUS in STDBY PHKL_21779
notes PHKL_26894
notes
s700_800 10.20 RFC-NETBIOS B.03.03.02 cumulative patch PHNE_15089
notes PHNE_15089
notes
s700_800 10.20 Raptor System Eagle Exportable 4.0 Y2K patch PHNE_15951
notes PHNE_15951
notes
s700 10.20 OTS/9000 C.07.0[01] cumulative patch PHNE_11880
notes PHNE_17863
notes
s700_800 10.20 2.23 ACC X.25 Protocol Software (EISA) Patch PHNE_19224
notes
s700_800 10.01-[12]0 2.2X ACC X.25 Protocol FW (EISA) Patch PHNE_15371
notes PHNE_19234
notes
s700_800 10.20 HSC FDDI B.10.20.[01-11] cumulative patch PHNE_20010
notes PHNE_20010
notes
s700_800 10.20 R6.10.20 SNAplus2 Common patch PHNE_20212
notes PHNE_21214
notes
s700_800 10.20 HSC 100BT lan cumulative patch PHNE_22216
notes PHNE_22216
notes
s700 10.20 CORE PCI 100Base-T cumulative patch PHNE_22405
notes PHNE_22405
notes
s700 10.20 cumulative ARPA Transport patch PHNE_22506
notes PHNE_22506
notes
s700_800 10.20 2.40.00-2.40.02 ACC Base Software Patch PHNE_18716
notes PHNE_22519
notes
s700_800 10.20 R6.10.20 SNAplus2 Link cumulative patch PHNE_20738
notes PHNE_22732
notes
s700_800 10.20 R5.1 SNAplus2 Link cumulative patch PHNE_22811
notes PHNE_22811
notes
s700_800 10.20 2.40.00-2.40.02 X.25/ACC Protocol Patch PHNE_23034
notes
s700_800 10.20 PCI FDDI cumulative patch PHNE_23509
notes PHNE_23509
notes
s700_800 10.20 1000Base-SX/T B.10.20.[01-11] cumul. patch PHNE_23713
notes PHNE_23713
notes
s700_800 10.20 EISA FDDI B.10.20.[01-11] cumulative patch PHNE_23766
notes PHNE_23766
notes
s700_800 10.20 PCI Mux driver/utilities cumulative patch PHNE_23881
notes
s700_800 10.20 EISA Mux driver/utilities cumulative patch PHNE_23882
notes PHNE_23882
notes
s700_800 10.20 ATM-HSC and HP-PB 3.0/6.0 cumulative patch PHNE_21423
notes PHNE_25500
notes
s700_800 10.20 ATM PCI,HSC and HP-PB cumulative patch PHNE_25501
notes PHNE_25501
notes
s700 10.20 LAN products cumulative Patch PHNE_22351
notes PHNE_28535
notes
s700_800 10.20 EISA 100BT cumulative patch PHNE_28635
notes PHNE_28635
notes
s700_800 10.20 NFS/NIS General Release/Performance Patch PHNE_22288
notes PHNE_28886
notes
s700_800 10.20 NFS/NIS General Release/Performance Patch PHNE_22877
notes PHNE_28886
notes
s700_800 10.X HP C++ (A.10.36) to fix numerous defects PHSS_14732 PHSS_14732
s700 10.20 Cumulative Online Diagnostics Tools Patch PHSS_17887
notes PHSS_17887
notes
s700_800 10.X OpenMail B.05.01 June 99 Periodic Patch PHSS_18585
notes PHSS_18585
notes
s700_800 10.X OV NNM6.0x Consolidated Patch PHSS_18891
notes PHSS_18891
notes
s700_800 10.X OV NNM5.0x Consolidated Patch 09-21-99 PHSS_19345
notes PHSS_19345
notes
s700_800 10.X ld(1) and som tools cumulative patch PHSS_21110 PHSS_21110
s700_800 10.20 OV ITO6.0X Consolidated Server Patch A.06.08 PHSS_24263
notes
s700_800 10.20 OV ITO5.3X Consolidated Server Patch A.05.39 PHSS_24499
notes
s700_800 10.20 OV NNM6.1 Consolidated Patch 4 PHSS_24797
notes PHSS_24797
notes
s700_800 10.20 Xserver cumulative patch PHSS_22603
notes PHSS_26640
notes
s700_800 10.20 OV NNM6.2 Consolidated Patch 3 PHSS_27332
notes PHSS_27332
notes
s700_800 10.20 OpenMail B.06.00 June 03 Periodic Patch PHSS_21439
notes PHSS_29063
notes
s700_800 10.20 OpenMail B.07.00 June 03 Periodic Patch PHSS_29065
notes
s700_800 10.20 CDE Runtime Patch PHSS_23796
notes PHSS_29201
notes