kernel error at the end of init 2

[mogiro]

Hi,
I've got a Poltergeist in a rx6600 with hpux11.31.
Let me start from the begining. I've increased the physical memory on this machine from 8Gb to 12Gb by installing 4GB more.
After do it, I've restarted the machine and I've got the following boot sequence (see txt attached). Even I've unmounted the memory but the problem persists.
The most unbelievable is that if I run into single user mode, go to init 1 and from init 1 start/kill all the services of runlevel 2 and 3.. it works!! Even I'm able to start SAP on this machine. I'm turning fool...
So, under my point of view, the problem is something that is executed once the init2 command ends.
No errors on rc.log or syslog.
So, the problem is that the system is not able to boot normally by itself but it works if it is booted manually...
Any idea?

[Torsten.]

Hi again,

your log shows

* - An error has occurred !
* - Refer to the file /etc/rc.log for more information.

What is the error?



You are running firmware 3.01, I would suggest to update to 4.03 first.
Not sure if this is related, but anyway.

see

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareIndex.jsp?lang=en&cc=us&prodNameId=3257037&prodTypeId=15351&prodSeriesId=3257036&swLang=13&taskId=135&swEnvOID=54

[Fred Ruffet]

if you are able to start single, try to start level 2 services by hand, one after the other. If one crashes, if possible disable it or come back here to give this new element.

[Anka · Accepted solution]

http://www11.itrc.hp.com/service/cki/docDisplay.do?docLocale=en&docId=emr_na-c01374040-1

let me know if you can not open the link....

[mogiro]

Hi all,
I've attached the rc.log, but there is nothing interesting...
And yes, I've been able to manually start all the services of rc2.d and rc3.d, even I've started SAP on the machine. But if you enter on single user, do init 1 and then init2, it shows the bad kernel problem just when the init2 command is finished.
I've installed patches PHKL_38083 and PHCO_38048 but problem persists. The swainv shows the machine up to date now.
Thanks!!

[Anka]

check the document, there is the solution :)

http://www11.itrc.hp.com/service/cki/docDisplay.do?docLocale=en&docId=emr_na-c01374040-1

[mogiro]

I have a case opened with HP and I have been told by them about this USB issue, the problem is that I have also these patches installed, so I forced the reinstall of them. I've rebooted machine and it still stops on EFI, if I force the boot from fs0: with hpux, the problem persists.
Maybe it could be a usb HW problem?
Because the stack trace is not exactly the same. In the other hand, I've checked the machine physically to avoid that none plugges a usb device on it, and the SYS led is flashing amber. How can I know why?
Here you've got the new boot process:


HP-UX Start-up in progress
__________________________

Configure system crash dumps .................................. OK
Removing old vxvm files ....................................... OK
VxVM INFO V-5-2-3360 VxVM device node check ................... OK
VxVM INFO V-5-2-3362 VxVM general startup ..................... OK
VxVM INFO V-5-2-3366 VxVM reconfiguration recovery ............ OK
Mount file systems ............................................ OK
Setting hostname .............................................. OK
Start containment subsystem configuration ..................... OK
Set privilege group ........................................... N/A
Display date .................................................. N/A
Save system crash dump if needed .............................. N/A
Start evp ..................................................... OK
Enable auxiliary swap space ................................... OK
Start syncer daemon ........................................... OK
Initializing livedump ......................................... N/A
Start Utmp Daemon : manages User Accounting Database .......... OK
Starting DevFS ................................................ OK
Configure Loopback interfaces (lo0) ........................... OK
Starting Event Management (EVM) (phase 1) .....................
Stored message buffer up to system crash:

MFS is defined: base= 0xe000000101a53000 size= 19468 KB
Found adjacent data tr. Growing size. 0x25ad000 -> 0x65ad000.
Loaded ACPI revision 2.0 tables.
MMIO on this platform supports Write Coalescing.



Memory Class Setup
-------------------------------------------------------------------------
Class Physmem Lockmem Swapmem
-------------------------------------------------------------------------
System : 7762 MB 7762 MB 7762 MB
Kernel : 7762 MB 7762 MB 7762 MB
User : 7119 MB 6311 MB 6336 MB
-------------------------------------------------------------------------

Starting ktracer 0 1
Installing Socket Protocol families AF_INET and AF_INET6
64000/0xfa00 esvroot
Kernel EVM initialized
sec_init(): kernel RPC authentication/security initialization.
secgss_init(): kernel RPCSEC_GSS security initialization.
rpc_init(): kernel RPC initialization.
rpcmod_install(): kernel RPC STREAMS module "rpcmod" installation. ...(driver_install)
NOTICE: nfs_client_pv3_install(): nfs3 File system was registered at index 11.
NOTICE: nfs_client_pv4_install(): nfs4 File system was registered at index 12.
NOTICE: cachefsc_install: cachefs File system was registered at index 14.
btlan_load() Loaded Successfully
0 sba
0/0 lba
0/0/1/0 rmp3f01
0/0/1/1 rmp3f01
0/0/1/2 asio0
0/0/2/0 UsbOhci
0/0/2/1 UsbOhci
NOTICE: USB device attached. Identification String:
Devices/Device/USB/Standard/hp/Multibay/0_a1
<2.1.3.10.1008.294.161>
0/0/2/1.0x0 UsbMiniBus
0/0/2/1.0 UsbMiniBus
NOTICE: USB device attached. Identification String:
Devices/MassStorage-SCSI/USB/BulkOnly/hp/Multibay/0_a1
<2.310.3.150.1008.294.161>
0/0/2/1.0x0.0x0 UsbBulkOnlyMS
0/0/2/1.0.0 UsbBulkOnlyMS
NOTICE: USB device attached. Identification String:
Devices/ScsiControllerAdaptor/USB/BulkOnly/hp/Multibay
<2.1000.3.150.1008.294>
0/0/2/1.0x0.0x10 UsbScsiAdaptor
0/0/2/1.0.16 UsbScsiAdaptor
0/0/2/2 UsbEhci
0/1 lba
0/1/1 pci_slot
0/1/2 pci_slot
0/2 lba
0/2/1 pci_slot
0/3 lba
0/3/1 pci_slot
0/4 lba
0/4/1 pci_slot
sasd: Claimed HP PCI/PCI-X SAS MPT adapter at hardware path 0/4/1/0
0/4/1/0 sasd
0/4/2 pci_slot
0/4/2/0 iether
0/4/2/1 iether
0/5 lba
0/5/1 pci_slot
0/5/2 pci_slot
0/6 lba
0/6/1 pci_slot
0/7 lba
0/7/1 pci_slot
fcd: Claimed HP AB378-60101 4Gb Fibre Channel port at hardware path 0/7/1/0 (FC Port 1 on HBA)
0/7/1/0 fcd
120 processor
122 processor
250 pdh
250/0 ipmi
250/1 asio0
250/2 acpi_node
0/7/1/0.0x5006016310601528 estp
0/7/1/0.0x5006016810601528 estp
0/7/1/0.0x5006016810601528.0x0 eslpt
0/7/1/0.0x5006016310601528.0x0 eslpt
0/7/1/0.0x5006016310601528.0x4000000000000000 eslpt
0/7/1/0.0x5006016310601528.0x4001000000000000 eslpt
0/7/1/0.0x5006016310601528.0x4003000000000000 eslpt
0/7/1/0.0x5006016310601528.0x4002000000000000 eslpt
0/7/1/0.2 fcd_fcp
0/7/1/0.2.11.255.0 fcd_vbus
0/7/1/0.2.11.255.0.0 tgt
0/7/1/0.2.11.255.0.0.0 sctl
0/7/1/0.2.11.0.0 fcd_vbus
0/7/1/0.2.11.0.0.0 tgt
0/7/1/0.2.11.0.0.0.0 sdisk
0/7/1/0.2.11.0.0.0.1 sdisk
0/7/1/0.2.11.0.0.0.2 sdisk
0/7/1/0.2.11.0.0.0.3 sdisk
0/7/1/0.0x5006016810601528.0x4000000000000000 eslpt
0/7/1/0.0x5006016810601528.0x4001000000000000 eslpt
0/7/1/0.0x5006016810601528.0x4002000000000000 eslpt
0/7/1/0.0x5006016810601528.0x4003000000000000 eslpt
0/7/1/0.2.15.255.0 fcd_vbus
0/7/1/0.2.15.255.0.0 tgt
0/7/1/0.2.15.255.0.0.0 sctl
0/7/1/0.2.15.0.0 fcd_vbus
0/7/1/0.2.15.0.0.0 tgt
0/7/1/0.2.15.0.0.0.0 sdisk
0/7/1/0.2.15.0.0.0.1 sdisk
0/7/1/0.2.15.0.0.0.2 sdisk
0/7/1/0.2.15.0.0.0.3 sdisk
0/4/1/0.0x5000c500082fe2f9 estp
0/4/1/0.0x5000c500082ffc29 estp
0/4/1/0.0x5000c500082fe2f9.0x0 eslpt
0/4/1/0.0x5000c500082ffc29.0x0 eslpt
0/4/1/0.0.0 sasd_vbus
0/4/1/0.0.0.0 tgt
0/4/1/0.0.0.0.0 sdisk
0/4/1/0.0.0.1 tgt
0/4/1/0.0.0.1.0 sdisk
64000/0xfa00/0x0 esdisk
64000/0xfa00/0x1 esdisk
64000/0xfa00/0x1c esctl
64000/0xfa00/0x1d esdisk
64000/0xfa00/0x1e esdisk
64000/0xfa00/0x1f esdisk
64000/0xfa00/0x20 esdisk
Boot device's HP-UX HW path is: 0.4.1.0.5000c500082fe2f9.0

System Console is on the Built-In Serial Interface
iether0: INITIALIZING HP AB352-60003 PCI/PCI-X 1000Base-T Dual-port Core at hardware path 0/4/2/0
iether1: INITIALIZING HP AB352-60003 PCI/PCI-X 1000Base-T Dual-port Core at hardware path 0/4/2/1
AF_INET socket/streams output daemon running, pid 34
afinet_prelink: module installed
Starting the STREAMS daemons-phase 1
LVM: Root VG activated
Swap device table: (start & size given in 512-byte blocks)
entry 0 - major is 64, minor is 0x2; start = 0, size = 16777216
Dump device table: (start & size given in 1-Kbyte blocks)
entry 0000000000000000 - major is 3, minor is 0x1; start = 2349920, size = 8388608
Create STCP device files
Starting the STREAMS daemons-phase 2
$Revision: vmunix: B.11.31_LR FLAVOR=perf
Memory Information:
physical page size = 4096 bytes, logical page size = 4096 bytes
Physical: 8357884 Kbytes, lockable: 6046292 Kbytes, available: 6875300 Kbytes

bad_kern_reference: 0xccde031.0x7, fault = 0x1c

Message buffer contents after system crash:

panic: Fault when executing in kernel mode
Stack Trace:
IP Function Name
0xe000000001e8c1a0 bad_kern_reference+0xa0
0xe00000000127bf30 $cold_pfault+0x560
0xe000000000857be0 vm_hndlr+0xd80
0xe000000001b1f780 bubbledown+0x0
0xe000000000b0cfd0 Messaging_MessageToProtocol+0xd0
0xe000000000af6860 Exceptions_CreateException+0x1e0
0xe000000000b1ca70 DeviceDriver_GetUnixEntryPointNumber+0xd0
0xe000000000b211b0 HpuxIoSupport_GioCallbackHandler+0x430
0xe0000000006e2800 io_node_callback+0x220
0xe0000000014774f0 io_get_devs+0x70
0xe000000000dcff60 do_io_get_devs+0x140
0xe000000001096c60 $cold_dev_config_ioctl+0x820
0xe000000000655400 spec_ioctl+0x100
0xe0000000006e3370 vno_ioctl+0x370
0xe000000000723bd0 ioctl+0x3d0
0xe0000000008025d0 syscall+0xd30
End of Stack Trace

linkstamp: Wed Aug 27 13:12:50 METDST 2008
_release_version: @(#) $Revision: vmunix: B.11.31_LR FLAVOR=perf
Calling function e000000001618600 for Shutdown State 1 type 0x2

sync'ing disks (0 buffers to flush):
0 fcache pages still dirty
0 buffers not flushed
0 buffers still dirty
i 0 pfn 0x1 pages 0x3dcc7
i 1 pfn 0x3e1a4 pages 0x384
i 2 pfn 0x3fc00 pages 0x1a2
i 3 pfn 0x3fda4 pages 0xa
i 4 pfn 0x3fdb0 pages 0x4
i 5 pfn 0x3fdd2 pages 0x6
i 6 pfn 0x100000 pages 0x100000
i 7 pfn 0x10040000 pages 0xbf1fe
i 8 pfn 0x100ff200 pages 0xe00
*** Not enough CPUS for a compressed dump ***

*** A system crash has occurred. (See the above messages for details.)
*** The system is now preparing to dump physical memory to disk, for use
*** in debugging the crash.

*** The dump will be a SELECTIVE dump with
compression OFF and concurrency ON: 991 of 8162 megabytes.
*** To change this dump type, press any key within 10 seconds.
*** Proceeding with selective dump, with compression off and concurrency on.

[Tim Nelson]

Update the FW as Torsten suggested. Tough to troubleshoot with old FW.

Also: How do I know why the light is on ?

Log into the MP and veiw the logs. The light is on because the system has panic'd
or maybe there just is a HW issue relating to the panic.

[Anka]

In your last log you mention: "I've rebooted machine and it still stops on EFI"

Where exactly does it stop, what exactly do you mean with it?

And you also mention only the patches but not the usb driver, is the driver also updated to the latest one?

[mogiro]

Hi all,
This is the boot sequence till it arrives to the EFI shell. Then I start on single user mode and so.
***********************************************************
* ROM Version : 03.01
* ROM Date : 09/26/2007
* BMC Version : 05.21
***********************************************************
0 0 0x0015B2 0x0000000018673084 boot time event
1 0 0x0000A4 0x0000000000000000 start memory configuration
2 0 0x001CBB 0x0000000000000000 System set to insecure mode
0 0 0x0015B2 0x0000000029421602 boot time event
1 0 0x000014 0x0000000000000000 CPU0 starting cell relocation
1 0 0x000009 0x0000000000000000 CPU0 launch EFI
EFI version 2.00 [14.62]
EFI64 Running on Intel(R) Itanium Processor Family
UEFI 2.0 IPF server rx6600 6.20 [Fri Feb 23 17:21:28 2007] firmware by HP
Loading 'SysROM:Floating-Point Software Assistance Handler'...
-- 'SysROM:Floating-Point Software Assistance Handler' returned Success
Loading 'SysROM:Broadcom Gigabit Ethernet Driver'...

Copyright (c) 2000-2006 Broadcom Corporation
Broadcom NetXtreme Gigabit Ethernet EFI driver v9.4.4

-- 'SysROM:Broadcom Gigabit Ethernet Driver' returned Success
Loading 'SysROM:SCSI Bus Driver'...
-- 'SysROM:SCSI Bus Driver' returned Success
Loading 'SysROM:SCSI Disk Driver'...
-- 'SysROM:SCSI Disk Driver' returned Success
Loading 'SysROM:SCSI Tape Driver'...
-- 'SysROM:SCSI Tape Driver' returned Success
Loading device drivers

Searching for devices
HP PCI-X 1Port 4Gb Fibre Channel Adapter (driver 1.09, firmware 4.00.090)
Loading.: EFI Shell [Built-in]
EFI Shell version 2.00 [14.62]
Device mapping table
fs0 : Acpi(HWP0002,PNP0A03,400)/Pci(1|0)/Sas(Addr5000C500082FE2F9,Lun0)/HD(Part1,Sig5ACE41E0-F5A4-11DC-8000-D6217B60E588)
fs1 : Acpi(HWP0002,PNP0A03,400)/Pci(1|0)/Sas(Addr5000C500082FE2F9,Lun0)/HD(Part3,Sig5ACE4208-F5A4-11DC-8000-D6217B60E588)
fs2 : Acpi(HWP0002,PNP0A03,400)/Pci(1|0)/Sas(Addr5000C500082FFC29,Lun0)/HD(Part1,Sig650D64C8-D99E-11DC-8000-D6217B60E588)
fs3 : Acpi(HWP0002,PNP0A03,400)/Pci(1|0)/Sas(Addr5000C500082FFC29,Lun0)/HD(Part3,Sig650D6518-D99E-11DC-8000-D6217B60E588)
blk0 : Acpi(HWP0002,PNP0A03,0)/Pci(2|1)/Usb(0, 0)
blk1 : Acpi(HWP0002,PNP0A03,400)/Pci(1|0)/Sas(Addr5000C500082FE2F9,Lun0)
blk2 : Acpi(HWP0002,PNP0A03,400)/Pci(1|0)/Sas(Addr5000C500082FE2F9,Lun0)/HD(Part1,Sig5ACE41E0-F5A4-11DC-8000-D6217B60E588)
blk3 : Acpi(HWP0002,PNP0A03,400)/Pci(1|0)/Sas(Addr5000C500082FE2F9,Lun0)/HD(Part2,Sig5ACE41F4-F5A4-11DC-8000-D6217B60E588)
blk4 : Acpi(HWP0002,PNP0A03,400)/Pci(1|0)/Sas(Addr5000C500082FE2F9,Lun0)/HD(Part3,Sig5ACE4208-F5A4-11DC-8000-D6217B60E588)
blk5 : Acpi(HWP0002,PNP0A03,400)/Pci(1|0)/Sas(Addr5000C500082FFC29,Lun0)
blk6 : Acpi(HWP0002,PNP0A03,400)/Pci(1|0)/Sas(Addr5000C500082FFC29,Lun0)/HD(Part1,Sig650D64C8-D99E-11DC-8000-D6217B60E588)
blk7 : Acpi(HWP0002,PNP0A03,400)/Pci(1|0)/Sas(Addr5000C500082FFC29,Lun0)/HD(Part2,Sig650D6504-D99E-11DC-8000-D6217B60E588)
blk8 : Acpi(HWP0002,PNP0A03,400)/Pci(1|0)/Sas(Addr5000C500082FFC29,Lun0)/HD(Part3,Sig650D6518-D99E-11DC-8000-D6217B60E588)
startup.nsh> echo -off

setting hpux path(\EFI\HPUX)...
type 'fs[x]:' where x is your bootdisk (0, 1, 2...)
type 'hpux' to start hpux bootloader