Serwers on Redhat 5.3 on HP hardware rebooted iautomaticaly

[Grzegorz Jajuga]

On the serwer I have oracle RAC 11.1.0.7 and 3 nodes.
All servers in cluster were rebooted iautomaticaly in messages I have following error.

ep 11 02:21:00 cib-db01 multipathd: dm-12: devmap already registered
Sep 11 02:44:05 cib-db01 kernel: sd 1:0:3:15: SCSI error: return code = 0x000d0000
Sep 11 02:44:05 cib-db01 kernel: end_request: I/O error, dev sddw, sector 2048
Sep 11 02:44:05 cib-db01 kernel: device-mapper: multipath: Failing path 71:224.
Sep 11 02:44:05 cib-db01 kernel: sd 1:0:2:4: SCSI error: return code = 0x000d0000
Sep 11 02:44:05 cib-db01 kernel: end_request: I/O error, dev sdcv, sector 1983
Sep 11 02:44:05 cib-db01 kernel: device-mapper: multipath: Failing path 70:48.
Sep 11 02:44:05 cib-db01 kernel: sd 1:0:2:10: SCSI error: return code = 0x06000000
Sep 11 02:44:05 cib-db01 kernel: end_request: I/O error, dev sddb, sector 1983
Sep 11 02:44:05 cib-db01 kernel: device-mapper: multipath: Failing path 70:144.
Sep 11 02:44:05 cib-db01 kernel: sd 1:0:2:2: SCSI error: return code = 0x000d0000
Sep 11 02:44:05 cib-db01 kernel: end_request: I/O error, dev sdct, sector 6193439
Sep 11 02:44:05 cib-db01 kernel: device-mapper: multipath: Failing path 70:16.
Sep 11 04:48:16 cib-db01 syslogd 1.4.1: restart.
Sep 11 04:48:16 cib-db01 kernel: klogd 1.4.1, log source = /proc/kmsg started.
Sep 11 04:48:16 cib-db01 kernel: Linux version 2.6.18-128.el5 (mockbuild@hs20-bc1-7.build.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) #1 SMP Wed Dec 17 11:41:38 EST 2008
Sep 11 04:48:16 cib-db01 kernel: Command line: ro root=LABEL=/ rhgb quiet elevator=noop
Sep 11 04:48:16 cib-db01 kernel: BIOS-provided physical RAM map:
Sep 11 04:48:16 cib-db01 kernel: BIOS-e820: 0000000000000000 - 000000000009f400 (usable)
Sep 11 04:48:16 cib-db01 kernel: BIOS-e820: 000000000009f400 - 00000000000a0000 (reserved)
Sep 11 04:48:16 cib-db01 kernel: BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
Sep 11 04:48:16 cib-db01 kernel: BIOS-e820: 0000000000100000 - 00000000cfe4e000 (usable)
Sep 11 04:48:16 cib-db01 kernel: BIOS-e820: 00000000cfe4e000 - 00000000cfe56000 (ACPI data)
Sep 11 04:48:16 cib-db01 kernel: BIOS-e820: 00000000cfe56000 - 00000000cfe57000 (usable)
Sep 11 04:48:16 cib-db01 kernel: BIOS-e820: 00000000cfe57000 - 00000000e0000000 (reserved)
Sep 11 04:48:16 cib-db01 kernel: BIOS-e820: 00000000fec00000 - 00000000fed00000 (reserved)
Sep 11 04:48:16 cib-db01 kernel: BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
Sep 11 04:48:16 cib-db01 kernel: BIOS-e820: 00000000ffc00000 - 0000000100000000 (reserved)
Sep 11 04:48:16 cib-db01 kernel: BIOS-e820: 0000000100000000 - 000000082ffff000 (usable)
Sep 11 04:48:16 cib-db01 kernel: DMI 2.4 present.
Sep 11 04:48:17 cib-db01 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
Sep 11 04:48:17 cib-db01 kernel: SRAT: PXM 0 -> APIC 1 -> Node 0
Sep 11 04:48:17 cib-db01 kernel: SRAT: PXM 0 -> APIC 2 -> Node 0
Sep 11 04:48:17 cib-db01 kernel: SRAT: PXM 0 -> APIC 3 -> Node 0
Sep 11 04:48:17 cib-db01 kernel: SRAT: PXM 1 -> APIC 4 -> Node 1
Sep 11 04:48:17 cib-db01 kernel: SRAT: PXM 1 -> APIC 5 -> Node 1
Sep 11 04:48:17 cib-db01 kernel: SRAT: PXM 1 -> APIC 6 -> Node 1
Sep 11 04:48:17 cib-db01 kernel: SRAT: PXM 1 -> APIC 7 -> Node 1
Sep 11 04:48:17 cib-db01 kernel: SRAT: Node 0 PXM 0 0-a0000
Sep 11 04:48:17 cib-db01 kernel: SRAT: Node 0 PXM 0 0-d0000000
Sep 11 04:48:17 cib-db01 kernel: SRAT: Node 0 PXM 0 0-430000000
Sep 11 04:48:17 cib-db01 kernel: SRAT: Node 1 PXM 1 430000000-830000000
Sep 11 04:48:17 cib-db01 kernel: Bootmem setup node 0 0000000000000000-0000000430000000
Sep 11 04:48:17 cib-db01 kernel: Bootmem setup node 1 0000000430000000-000000082ffff000
Sep 11 04:48:17 cib-db01 kernel: Memory for crash kernel (0x0 to 0x0) notwithin permissible range
Sep 11 04:48:17 cib-db01 kernel: disabling kdump
Sep 11 04:48:17 cib-db01 kernel: ACPI: PM-Timer IO Port: 0x920
Sep 11 04:48:17 cib-db01 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Sep 11 04:48:17 cib-db01 kernel: Processor #0 0:4 APIC version 16
Sep 11 04:48:17 cib-db01 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled)
Sep 11 04:48:17 cib-db01 kernel: Processor #4 0:4 APIC version 16
Sep 11 04:48:17 cib-db01 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
Sep 11 04:48:17 cib-db01 kernel: Processor #1 0:4 APIC version 16
Sep 11 04:48:17 cib-db01 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] enabled)
Sep 11 04:48:17 cib-db01 kernel: Processor #5 0:4 APIC version 16
Sep 11 04:48:17 cib-db01 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
Sep 11 04:48:17 cib-db01 kernel: Processor #2 0:4 APIC version 16
Sep 11 04:48:17 cib-db01 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled)
Sep 11 04:48:17 cib-db01 kernel: Processor #6 0:4 APIC version 16
Sep 11 04:48:17 cib-db01 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
Sep 11 04:48:17 cib-db01 kernel: Processor #3 0:4 APIC version 16
Sep 11 04:48:17 cib-db01 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled)
Sep 11 04:48:17 cib-db01 kernel: Processor #7 0:4 APIC version 16
Sep 11 04:48:17 cib-db01 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
Sep 11 04:48:17 cib-db01 kernel: ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
Sep 11 04:48:17 cib-db01 kernel: IOAPIC[0]: apic_id 8, version 17, address 0xfec00000, GSI 0-15
Sep 11 04:48:17 cib-db01 kernel: ACPI: IOAPIC (id[0x09] address[0xfec01000] gsi_base[16])
Sep 11 04:48:17 cib-db01 kernel: IOAPIC[1]: apic_id 9, version 17, address 0xfec01000, GSI 16-31
Sep 11 04:48:18 cib-db01 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
Sep 11 04:48:18 cib-db01 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
Sep 11 04:48:18 cib-db01 kernel: Setting APIC routing to physical flat
Sep 11 04:48:18 cib-db01 kernel: ACPI: HPET id: 0x1166a201 base: 0xfed00000
Sep 11 04:48:18 cib-db01 kernel: Using ACPI (MADT) for SMP configuration information
Sep 11 04:48:18 cib-db01 kernel: Nosave address range: 000000000009f000 - 00000000000a0000
Sep 11 04:48:18 cib-db01 kernel: Nosave address range: 00000000000a0000 - 00000000000f0000
Sep 11 04:48:18 cib-db01 fcauthd: Reading Config File: /etc/fcauthd.conf
Sep 11 04:48:18 cib-db01 kernel: Nosave address range: 00000000000f0000 - 0000000000100000
Sep 11 04:48:18 cib-db01 kernel: Nosave address range: 00000000cfe4e000 - 00000000cfe56000
Sep 11 04:48:18 cib-db01 kernel: Nosave address range: 00000000cfe57000 - 00000000e0000000
Sep 11 04:48:18 cib-db01 kernel: Nosave address range: 00000000e0000000 - 00000000fec00000
Sep 11 04:48:18 cib-db01 kernel: Nosave address range: 00000000fec00000 - 00000000fed00000
Sep 11 04:48:18 cib-db01 kernel: Nosave address range: 00000000fed00000 - 00000000fee00000
Sep 11 04:48:18 cib-db01 kernel: Nosave address range: 00000000fee00000 - 00000000fee10000
Sep 11 04:48:18 cib-db01 fcauthd: Successful SCSI nl registration.
Sep 11 04:48:18 cib-db01 kernel: Nosave address range: 00000000fee10000 - 00000000ffc00000
Sep 11 04:48:18 cib-db01 modprobe: FATAL: Module ocfs2_stackglue not found.
Sep 11 04:48:18 cib-db01 kernel: Nosave address range: 00000000ffc00000 - 0000000100000000
Sep 11 04:48:18 cib-db01 kernel: Allocating PCI resources starting at e2000000 (gap: e0000000:1ec00000)
Sep 11 04:48:18 cib-db01 kernel: SMP: Allowing 8 CPUs, 0 hotplug CPUs
Sep 11 04:48:18 cib-db01 kernel: Built 2 zonelists. Total pages: 8269409
Sep 11 04:48:18 cib-db01 kernel: Kernel command line: ro root=LABEL=/ rhgb quiet elevator=noop
Sep 11 04:48:18 cib-db01 kernel: Initializing CPU#0
Sep 11 04:48:18 cib-db01 kernel: PID hash table entries: 4096 (order: 12, 32768 bytes)
Sep 11 04:48:18 cib-db01 kernel: Console: colour VGA+ 80x25
Sep 11 04:48:18 cib-db01 kernel: Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes)
Sep 11 04:48:18 cib-db01 kernel: Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes)
Sep 11 04:48:18 cib-db01 kernel: Checking aperture...
Sep 11 04:48:18 cib-db01 kernel: CPU 0: aperture @ 8000000 size 32 MB
Sep 11 04:48:18 cib-db01 kernel: Aperture too small (32 MB)
Sep 11 04:48:18 cib-db01 kernel: No AGP bridge found
Sep 11 04:48:18 cib-db01 kernel: Your BIOS doesn't leave a aperture memory hole
Sep 11 04:48:18 cib-db01 kernel: Please enable the IOMMU option in the BIOS setup
Sep 11 04:48:19 cib-db01 kernel: This costs you 64 MB of RAM
Sep 11 04:48:19 cib-db01 kernel: Mapping aperture over 65536 KB of RAM @ 8000000
Sep 11 04:48:19 cib-db01 kernel: Nosave address range: 0000000008000000 - 000000000c000000
Sep 11 04:48:19 cib-db01 kernel: Memory: 32959412k/34340860k available (2494k kernel code, 592896k reserved, 1263k data, 200k init)
Sep 11 04:48:19 cib-db01 kernel: Calibrating delay using timer specific routine.. 5404.13 BogoMIPS (lpj=2702067)
Sep 11 04:48:19 cib-db01 kernel: Security Framework v1.0.0 initialized
Sep 11 04:48:19 cib-db01 kernel: SELinux: Initializing.
Sep 11 04:48:19 cib-db01 kernel: selinux_register_security: Registering secondary module capability
Sep 11 04:48:19 cib-db01 kernel: Capability LSM initialized as secondary
Sep 11 04:48:19 cib-db01 kernel: Mount-cache hash table entries: 256
Sep 11 04:48:19 cib-db01 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
Sep 11 04:48:19 cib-db01 kernel: CPU: L2 Cache: 512K (64 bytes/line)
Sep 11 04:48:19 cib-db01 kernel: CPU 0/0 -> Node 0
Sep 11 04:48:19 cib-db01 kernel: CPU: Physical Processor ID: 0
Sep 11 04:48:19 cib-db01 kernel: CPU: Processor Core ID: 0
Sep 11 04:48:19 cib-db01 kernel: SMP alternatives: switching to UP code
Sep 11 04:48:19 cib-db01 kernel: ACPI: Core revision 20060707
Sep 11 04:48:19 cib-db01 kernel: Using local APIC timer interrupts.
Sep 11 04:48:19 cib-db01 kernel: result 12500449
Sep 11 04:48:19 cib-db01 kernel: Detected 12.500 MHz APIC timer.
Sep 11 04:48:19 cib-db01 kernel: SMP alternatives: switching to SMP code
Sep 11 04:48:19 cib-db01 kernel: Booting processor 1/8 APIC 0x4
Sep 11 04:48:19 cib-db01 kernel: Initializing CPU#1
Sep 11 04:48:19 cib-db01 kernel: Calibrating delay using timer specific routine.. 5400.24 BogoMIPS (lpj=2700120)
Sep 11 04:48:19 cib-db01 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
Sep 11 04:48:19 cib-db01 kernel: CPU: L2 Cache: 512K (64 bytes/line)
Sep 11 04:48:19 cib-db01 kernel: CPU 1/4 -> Node 1
Sep 11 04:48:19 cib-db01 kernel: CPU: Physical Processor ID: 1
Sep 11 04:48:19 cib-db01 kernel: CPU: Processor Core ID: 0
Sep 11 04:48:19 cib-db01 kernel: Quad-Core AMD Opteron(tm) Processor 2384 stepping 02
Sep 11 04:48:19 cib-db01 kernel: SMP alternatives: switching to SMP code
Sep 11 04:48:19 cib-db01 kernel: Booting processor 2/8 APIC 0x1
Sep 11 04:48:19 cib-db01 kernel: Initializing CPU#2
Sep 11 04:48:19 cib-db01 kernel: Calibrating delay using timer specific routine.. 5400.05 BogoMIPS (lpj=2700029)
Sep 11 04:48:19 cib-db01 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
Sep 11 04:48:19 cib-db01 kernel: CPU: L2 Cache: 512K (64 bytes/line)
Sep 11 04:48:19 cib-db01 kernel: CPU 2/1 -> Node 0
Sep 11 04:48:19 cib-db01 kernel: CPU: Physical Processor ID: 0
Sep 11 04:48:19 cib-db01 kernel: CPU: Processor Core ID: 1
Sep 11 04:48:20 cib-db01 kernel: Quad-Core AMD Opteron(tm) Processor 2384 stepping 02
Sep 11 04:48:20 cib-db01 kernel: SMP alternatives: switching to SMP code
Sep 11 04:48:20 cib-db01 kernel: Booting processor 3/8 APIC 0x5
Sep 11 04:48:20 cib-db01 kernel: Initializing CPU#3
Sep 11 04:48:20 cib-db01 kernel: Calibrating delay using timer specific routine.. 5401.03 BogoMIPS (lpj=2700518)
Sep 11 04:48:20 cib-db01 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
Sep 11 04:48:20 cib-db01 kernel: CPU: L2 Cache: 512K (64 bytes/line)
Sep 11 04:48:20 cib-db01 kernel: CPU 3/5 -> Node 1
Sep 11 04:48:20 cib-db01 kernel: CPU: Physical Processor ID: 1
Sep 11 04:48:20 cib-db01 kernel: CPU: Processor Core ID: 1
Sep 11 04:48:20 cib-db01 kernel: Quad-Core AMD Opteron(tm) Processor 2384 stepping 02
Sep 11 04:48:20 cib-db01 kernel: SMP alternatives: switching to SMP code
Sep 11 04:48:20 cib-db01 kernel: Booting processor 4/8 APIC 0x2
Sep 11 04:48:20 cib-db01 kernel: Initializing CPU#4
Sep 11 04:48:20 cib-db01 kernel: Calibrating delay using timer specific routine.. 5401.43 BogoMIPS (lpj=2700717)
Sep 11 04:48:20 cib-db01 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
Sep 11 04:48:20 cib-db01 kernel: CPU: L2 Cache: 512K (64 bytes/line)
Sep 11 04:48:20 cib-db01 kernel: CPU 4/2 -> Node 0
Sep 11 04:48:20 cib-db01 kernel: CPU: Physical Processor ID: 0
Sep 11 04:48:20 cib-db01 kernel: CPU: Processor Core ID: 2
Sep 11 04:48:20 cib-db01 kernel: Quad-Core AMD Opteron(tm) Processor 2384 stepping 02
Sep 11 04:48:20 cib-db01 kernel: SMP alternatives: switching to SMP code
Sep 11 04:48:20 cib-db01 kernel: Booting processor 5/8 APIC 0x6
Sep 11 04:48:20 cib-db01 kernel: Initializing CPU#5
Sep 11 04:48:20 cib-db01 kernel: Calibrating delay using timer specific routine.. 5402.69 BogoMIPS (lpj=2701348)
Sep 11 04:48:20 cib-db01 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
Sep 11 04:48:20 cib-db01 kernel: CPU: L2 Cache: 512K (64 bytes/line)
Sep 11 04:48:20 cib-db01 kernel: CPU 5/6 -> Node 1
Sep 11 04:48:20 cib-db01 kernel: CPU: Physical Processor ID: 1
Sep 11 04:48:20 cib-db01 kernel: CPU: Processor Core ID: 2
Sep 11 04:48:20 cib-db01 kernel: Quad-Core AMD Opteron(tm) Processor 2384 stepping 02
Sep 11 04:48:20 cib-db01 kernel: SMP alternatives: switching to SMP code
Sep 11 04:48:20 cib-db01 kernel: Booting processor 6/8 APIC 0x3
Sep 11 04:48:20 cib-db01 kernel: Initializing CPU#6
Sep 11 04:48:20 cib-db01 kernel: Calibrating delay using timer specific routine.. 5402.44 BogoMIPS (lpj=2701220)
Sep 11 04:48:20 cib-db01 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
Sep 11 04:48:20 cib-db01 kernel: CPU: L2 Cache: 512K (64 bytes/line)
Sep 11 04:48:20 cib-db01 kernel: CPU 6/3 -> Node 0
Sep 11 04:48:20 cib-db01 kernel: CPU: Physical Processor ID: 0
Sep 11 04:48:20 cib-db01 kernel: CPU: Processor Core ID: 3
Sep 11 04:48:20 cib-db01 kernel: Quad-Core AMD Opteron(tm) Processor 2384 stepping 02
Sep 11 04:48:20 cib-db01 kernel: SMP alternatives: switching to SMP code
Sep 11 04:48:21 cib-db01 kernel: Booting processor 7/8 APIC 0x7
Sep 11 04:48:21 cib-db01 kernel: Initializing CPU#7
Sep 11 04:48:21 cib-db01 kernel: Calibrating delay using timer specific routine.. 5402.15 BogoMIPS (lpj=2701076)
Sep 11 04:48:21 cib-db01 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
Sep 11 04:48:21 cib-db01 kernel: CPU: L2 Cache: 512K (64 bytes/line)
Sep 11 04:48:21 cib-db01 kernel: CPU 7/7 -> Node 1
Sep 11 04:48:21 cib-db01 kernel: CPU: Physical Processor ID: 1
Sep 11 04:48:21 cib-db01 kernel: CPU: Processor Core ID: 3
Sep 11 04:48:21 cib-db01 kernel: Quad-Core AMD Opteron(tm) Processor 2384 stepping 02
Sep 11 04:48:21 cib-db01 kernel: Brought up 8 CPUs
Sep 11 04:48:21 cib-db01 kernel: testing NMI watchdog ... OK.
Sep 11 04:48:21 cib-db01 kernel: time.c: Using 14.318180 MHz WALL HPET GTOD HPET/TSC timer.
Sep 11 04:48:21 cib-db01 kernel: time.c: Detected 2700.096 MHz processor.
Sep 11 04:48:21 cib-db01 kernel: migration_cost=85,6029
Sep 11 04:48:21 cib-db01 kernel: checking if image is initramfs... it is
Sep 11 04:48:21 cib-db01 kernel: Freeing initrd memory: 2711k freed
Sep 11 04:48:21 cib-db01 kernel: NET: Registered protocol family 16
Sep 11 04:48:21 cib-db01 kernel: No dock devices found.
Sep 11 04:48:21 cib-db01 kernel: ACPI: bus type pci registered
Sep 11 04:48:21 cib-db01 kernel: PCI: Using MMCONFIG at d0000000
Sep 11 04:48:21 cib-db01 kernel: ACPI: Interpreter enabled
Sep 11 04:48:21 cib-db01 kernel: ACPI: Using IOAPIC for interrupt routing
Sep 11 04:48:21 cib-db01 kernel: ACPI: PCI Root Bridge [PCI0] (0000:00)
Sep 11 04:48:21 cib-db01 kernel: PCI: Enabling HT MSI Mapping on 0000:00:05.0
Sep 11 04:48:21 cib-db01 kernel: ACPI: PCI Root Bridge [PCI1] (0000:40)
Sep 11 04:48:21 cib-db01 kernel: ACPI: PCI Interrupt Link [IUSB] (IRQs *5)
Sep 11 04:48:21 cib-db01 kernel: ACPI: PCI Interrupt Link [IN00] (IRQs 7 10 *11)
Sep 11 04:48:21 cib-db01 kernel: ACPI: PCI Interrupt Link [IN01] (IRQs 7 10 *11)
Sep 11 04:48:21 cib-db01 kernel: ACPI: PCI Interrupt Link [IN02] (IRQs 7 10 *11)
Sep 11 04:48:21 cib-db01 kernel: ACPI: PCI Interrupt Link [IN03] (IRQs 7 10 *11)
Sep 11 04:48:21 cib-db01 kernel: ACPI: PCI Interrupt Link [IN04] (IRQs *7 10 11)
Sep 11 04:48:21 cib-db01 kernel: ACPI: PCI Interrupt Link [IN05] (IRQs 7 *10 11)
Sep 11 04:48:21 cib-db01 kernel: ACPI: PCI Interrupt Link [IN06] (IRQs 7 *10 11)
Sep 11 04:48:21 cib-db01 kernel: ACPI: PCI Interrupt Link [IN07] (IRQs *7 10 11)
Sep 11 04:48:21 cib-db01 kernel: ACPI: PCI Interrupt Link [IN08] (IRQs 7 10 11) *0, disabled.
Sep 11 04:48:21 cib-db01 kernel: ACPI: PCI Interrupt Link [IN09] (IRQs 7 10 11) *0, disabled.
Sep 11 04:48:21 cib-db01 kernel: ACPI: PCI Interrupt Link [IN10] (IRQs 7 10 11) *0, disabled.
Sep 11 04:48:21 cib-db01 kernel: ACPI: PCI Interrupt Link [IN11] (IRQs 7 10 11) *0, disabled.
Sep 11 04:48:21 cib-db01 kernel: ACPI: PCI Interrupt Link [IN12] (IRQs 7 10 11) *0, disabled.
Sep 11 04:48:21 cib-db01 kernel: ACPI: PCI Interrupt Link [IN13] (IRQs 7 10 11) *0, disabled.
Sep 11 04:48:21 cib-db01 kernel: ACPI: PCI Interrupt Link [IN14] (IRQs 7 10 11) *0, disabled.
Sep 11 04:48:21 cib-db01 kernel: ACPI: PCI Interrupt Link [IN15] (IRQs 7 10 11) *0, disabled.
Sep 11 04:48:22 cib-db01 kernel: ACPI: PCI Interrupt Link [IN16] (IRQs 7 10 11) *0, disabled.
Sep 11 04:48:22 cib-db01 kernel: ACPI: PCI Interrupt Link [IN17] (IRQs 7 10 11) *0, disabled.
Sep 11 04:48:22 cib-db01 kernel: ACPI: PCI Interrupt Link [IN18] (IRQs 7 10 11) *0, disabled.
Sep 11 04:48:22 cib-db01 kernel: ACPI: PCI Interrupt Link [IN19] (IRQs 7 10 11) *0, disabled.
Sep 11 04:48:22 cib-db01 kernel: ACPI: PCI Interrupt Link [IN20] (IRQs 7 10 11) *0, disabled.
Sep 11 04:48:22 cib-db01 kernel: ACPI: PCI Interrupt Link [IN21] (IRQs 7 10 11) *0, disabled.
Sep 11 04:48:22 cib-db01 kernel: ACPI: PCI Interrupt Link [IN22] (IRQs 7 10 11) *0, disabled.
Sep 11 04:48:22 cib-db01 kernel: ACPI: PCI Interrupt Link [IN23] (IRQs 7 10 11) *0, disabled.
Sep 11 04:48:22 cib-db01 kernel: ACPI: PCI Interrupt Link [IN24] (IRQs 7 10 11) *0, disabled.
Sep 11 04:48:22 cib-db01 kernel: ACPI: PCI Interrupt Link [IN25] (IRQs 7 10 11) *0, disabled.
Sep 11 04:48:22 cib-db01 kernel: ACPI: PCI Interrupt Link [IN26] (IRQs 7 10 11) *0, disabled.
Sep 11 04:48:22 cib-db01 kernel: ACPI: PCI Interrupt Link [IN27] (IRQs 7 10 11) *0, disabled.
Sep 11 04:48:22 cib-db01 kernel: ACPI: PCI Interrupt Link [IN28] (IRQs 7 10 11) *0, disabled.
Sep 11 04:48:22 cib-db01 kernel: ACPI: PCI Interrupt Link [IN29] (IRQs 7 10 11) *0, disabled.
Sep 11 04:48:22 cib-db01 kernel: ACPI: PCI Interrupt Link [IN30] (IRQs 7 10 11) *0, disabled.
Sep 11 04:48:22 cib-db01 kernel: ACPI: PCI Interrupt Link [IN31] (IRQs 7 10 11) *0, disabled.

[Matti_Kurkela · Accepted solution]

(Welcome to the ITRC Forums!)

At 02:44, your server(s) seem to have started losing disk connectivity. Then the logging just stops: no panic message or anything explicitly indicating the system is going to reboot.

The reboot apparently happened at 04:48. All the messages after that seem to be regular boot-up messages: no significant information there.

Do these systems have a local system disk, or are they booting from a SAN? Losing access to the system disk would obviously explain the lack of logs and would probably have caused the system to reboot.

If this happened to the entire cluster at the same time, it would suggest some problem with the storage system. I'd recommend checking the hardware error log of the storage system (assuming it has one).

Did someone trip over the storage system's power and/or data cables? :-)

MK

[Abushad]

Hi,

According to the log your server lost connectivity with multipathed disks and rebooted.
This may have caused due to a SAN switch fault.

Regards,
Abushad

[Grzegorz Jajuga]

Hi. We didn't have any problem with SAN, optical switch and server.

On one node we only clear CMOS(reset bios).

For share file sysem we use OCFS2.On OCFS2 we install RAC on 3 nodes.

All 3 nodes were restarted at the same time.

[Grzegorz Jajuga]

It is log from 3 node in RAC
Sep 11 01:57:33 cib-db03 kernel: sd 1:0:0:4: SCSI error: return code = 0x000d0000
Sep 11 01:57:33 cib-db03 kernel: end_request: I/O error, dev sdbp, sector 42940882
Sep 11 01:57:33 cib-db03 kernel: device-mapper: multipath: Failing path 68:48.
Sep 11 01:57:33 cib-db03 kernel: sd 1:0:0:1: SCSI error: return code = 0x000d0000
Sep 11 01:57:33 cib-db03 kernel: end_request: I/O error, dev sdbm, sector 6326039
Sep 11 01:57:33 cib-db03 kernel: device-mapper: multipath: Failing path 68:0.
Sep 11 01:57:33 cib-db03 kernel: sd 1:0:0:1: SCSI error: return code = 0x000d0000
Sep 11 01:57:33 cib-db03 kernel: end_request: I/O error, dev sdbm, sector 19167343
Sep 11 01:57:33 cib-db03 kernel: sd 1:0:0:1: SCSI error: return code = 0x000d0000
Sep 11 01:57:33 cib-db03 kernel: end_request: I/O error, dev sdbm, sector 29586367
Sep 11 01:57:33 cib-db03 kernel: sd 1:0:0:1: SCSI error: return code = 0x000d0000
Sep 11 01:57:33 cib-db03 kernel: end_request: I/O error, dev sdbm, sector 31475071
Sep 11 01:57:33 cib-db03 kernel: sd 1:0:0:1: SCSI error: return code = 0x000d0000
Sep 11 01:57:33 cib-db03 multipathd: dm-15: add map (uevent)
Sep 11 01:57:33 cib-db03 kernel: end_request: I/O error, dev sdbm, sector 33827647
Sep 11 01:57:33 cib-db03 kernel: sd 1:0:0:4: SCSI error: return code = 0x000d0000
Sep 11 01:57:33 cib-db03 kernel: end_request: I/O error, dev sdbp, sector 10323167
Sep 11 01:57:33 cib-db03 kernel: sd 1:0:0:8: SCSI error: return code = 0x000d0000
Sep 11 01:57:33 cib-db03 kernel: end_request: I/O error, dev sdbt, sector 1985
Sep 11 01:57:33 cib-db03 kernel: device-mapper: multipath: Failing path 68:112.
Sep 11 01:57:33 cib-db03 kernel: sd 1:0:0:1: SCSI error: return code = 0x000d0000
Sep 11 01:57:33 cib-db03 kernel: end_request: I/O error, dev sdbm, sector 1985
Sep 11 01:57:33 cib-db03 kernel: sd 1:0:0:1: SCSI error: return code = 0x000d0000
Sep 11 01:57:33 cib-db03 kernel: end_request: I/O error, dev sdbm, sector 21982423
Sep 11 01:57:42 cib-db03 multipathd: dm-15: devmap already registered
Sep 11 01:57:42 cib-db03 multipathd: 68:48: mark as failed
Sep 11 01:57:42 cib-db03 multipathd: retail_1: remaining active paths: 7
Sep 11 01:57:42 cib-db03 multipathd: 68:0: mark as failed
Sep 11 01:57:42 cib-db03 multipathd: oracle: remaining active paths: 7
Sep 11 01:57:42 cib-db03 multipathd: dm-9: add map (uevent)
Sep 11 01:57:42 cib-db03 multipathd: dm-9: devmap already registered
Sep 11 01:57:42 cib-db03 multipathd: dm-24: add map (uevent)
Sep 11 01:57:42 cib-db03 multipathd: dm-24: devmap already registered
Sep 11 01:57:42 cib-db03 multipathd: 68:112: mark as failed
Sep 11 01:57:42 cib-db03 multipathd: hades_0: remaining active paths: 7
Sep 11 01:57:42 cib-db03 multipathd: sdbm: tur checker reports path is up
Sep 11 01:57:42 cib-db03 multipathd: 68:0: reinstated
Sep 11 01:57:42 cib-db03 multipathd: oracle: remaining active paths: 8
Sep 11 01:57:42 cib-db03 multipathd: sdbp: tur checker reports path is up
Sep 11 01:57:42 cib-db03 multipathd: 68:48: reinstated
Sep 11 01:57:42 cib-db03 multipathd: retail_1: remaining active paths: 8
Sep 11 01:57:42 cib-db03 multipathd: sdbt: tur checker reports path is up
Sep 11 01:57:42 cib-db03 multipathd: 68:112: reinstated
Sep 11 01:57:42 cib-db03 multipathd: hades_0: remaining active paths: 8
Sep 11 01:57:42 cib-db03 multipathd: dm-9: add map (uevent)
Sep 11 01:57:42 cib-db03 multipathd: dm-9: devmap already registered
Sep 11 01:57:42 cib-db03 multipathd: dm-15: add map (uevent)
Sep 11 01:57:42 cib-db03 multipathd: dm-15: devmap already registered
Sep 11 01:57:42 cib-db03 multipathd: dm-24: add map (uevent)
Sep 11 01:57:42 cib-db03 multipathd: dm-24: devmap already registered
Sep 11 02:13:24 cib-db03 kernel: sd 0:0:3:15: SCSI error: return code = 0x000d0000
Sep 11 02:13:24 cib-db03 kernel: end_request: I/O error, dev sdbk, sector 2048
Sep 11 02:13:24 cib-db03 kernel: device-mapper: multipath: Failing path 67:224.
Sep 11 02:13:24 cib-db03 kernel: sd 0:0:0:1: SCSI error: return code = 0x000d0000
Sep 11 02:13:24 cib-db03 kernel: end_request: I/O error, dev sda, sector 1985
Sep 11 02:13:24 cib-db03 kernel: device-mapper: multipath: Failing path 8:0.
Sep 11 02:13:24 cib-db03 kernel: sd 0:0:0:12: SCSI error: return code = 0x000d0000
Sep 11 02:13:24 cib-db03 kernel: end_request: I/O error, dev sdl, sector 1985
Sep 11 02:13:24 cib-db03 kernel: device-mapper: multipath: Failing path 8:176.
Sep 11 02:13:24 cib-db03 multipathd: dm-6: add map (uevent)
Sep 11 02:13:24 cib-db03 multipathd: dm-6: devmap already registered
Sep 11 02:13:24 cib-db03 multipathd: dm-9: add map (uevent)
Sep 11 02:13:24 cib-db03 multipathd: dm-9: devmap already registered
Sep 11 02:13:24 cib-db03 multipathd: dm-1: add map (uevent)
Sep 11 02:13:24 cib-db03 multipathd: dm-1: devmap already registered
Sep 11 02:13:24 cib-db03 multipathd: 8:176: mark as failed
Sep 11 02:13:24 cib-db03 multipathd: cluster: remaining active paths: 7
Sep 11 02:13:24 cib-db03 multipathd: 67:224: mark as failed
Sep 11 02:13:24 cib-db03 multipathd: corpo_3: remaining active paths: 7
Sep 11 02:13:24 cib-db03 multipathd: 8:0: mark as failed
Sep 11 02:13:24 cib-db03 multipathd: oracle: remaining active paths: 7
Sep 11 02:13:33 cib-db03 multipathd: sda: tur checker reports path is up
Sep 11 02:13:33 cib-db03 multipathd: 8:0: reinstated
Sep 11 02:13:33 cib-db03 multipathd: oracle: remaining active paths: 8
Sep 11 02:13:33 cib-db03 multipathd: sdbk: tur checker reports path is up
Sep 11 02:13:33 cib-db03 multipathd: 67:224: reinstated
Sep 11 02:13:33 cib-db03 multipathd: corpo_3: remaining active paths: 8
Sep 11 02:13:33 cib-db03 multipathd: sdl: tur checker reports path is up
Sep 11 02:13:33 cib-db03 multipathd: 8:176: reinstated
Sep 11 02:13:33 cib-db03 multipathd: cluster: remaining active paths: 8
Sep 11 02:13:33 cib-db03 multipathd: dm-9: add map (uevent)
Sep 11 02:13:33 cib-db03 multipathd: dm-9: devmap already registered
Sep 11 02:13:33 cib-db03 multipathd: dm-6: add map (uevent)
Sep 11 02:13:33 cib-db03 multipathd: dm-6: devmap already registered
Sep 11 02:13:33 cib-db03 multipathd: dm-1: add map (uevent)
Sep 11 02:13:33 cib-db03 multipathd: dm-1: devmap already registered
Sep 11 02:20:51 cib-db03 kernel: sd 0:0:0:11: SCSI error: return code = 0x000d0000
Sep 11 02:20:51 cib-db03 kernel: end_request: I/O error, dev sdk, sector 1985
Sep 11 02:20:51 cib-db03 kernel: device-mapper: multipath: Failing path 8:160.
Sep 11 02:20:51 cib-db03 kernel: sd 0:0:0:9: SCSI error: return code = 0x000d0000
Sep 11 02:20:51 cib-db03 kernel: end_request: I/O error, dev sdi, sector 1983
Sep 11 02:20:51 cib-db03 kernel: device-mapper: multipath: Failing path 8:128.
Sep 11 02:20:51 cib-db03 kernel: sd 0:0:0:6: SCSI error: return code = 0x000d0000
Sep 11 02:20:51 cib-db03 kernel: end_request: I/O error, dev sdf, sector 1983
Sep 11 02:20:51 cib-db03 kernel: device-mapper: multipath: Failing path 8:80.
Sep 11 02:20:51 cib-db03 kernel: sd 0:0:0:2: SCSI error: return code = 0x000d0000
Sep 11 02:20:51 cib-db03 kernel: end_request: I/O error, dev sdb, sector 6194559
Sep 11 02:20:51 cib-db03 kernel: device-mapper: multipath: Failing path 8:16.
Sep 11 02:20:51 cib-db03 kernel: sd 0:0:0:6: SCSI error: return code = 0x000d0000
Sep 11 02:20:51 cib-db03 multipathd: dm-0: add map (uevent)
Sep 11 02:20:51 cib-db03 kernel: end_request: I/O error, dev sdf, sector 12386591
Sep 11 02:20:51 cib-db03 kernel: sd 0:0:0:9: SCSI error: return code = 0x000d0000
Sep 11 02:20:51 cib-db03 kernel: end_request: I/O error, dev sdi, sector 41288960
Sep 11 02:20:51 cib-db03 kernel: sd 0:0:0:6: SCSI error: return code = 0x000d0000
Sep 11 02:20:51 cib-db03 kernel: end_request: I/O error, dev sdf, sector 12387551
Sep 11 02:20:51 cib-db03 kernel: sd 0:0:0:2: SCSI error: return code = 0x000d0000
Sep 11 02:20:51 cib-db03 kernel: end_request: I/O error, dev sdb, sector 1985
Sep 11 02:20:51 cib-db03 kernel: sd 0:0:0:2: SCSI error: return code = 0x000d0000
Sep 11 02:20:51 cib-db03 kernel: end_request: I/O error, dev sdb, sector 6194399
Sep 11 02:21:00 cib-db03 multipathd: dm-0: devmap already registered
Sep 11 02:21:00 cib-db03 multipathd: 8:160: mark as failed
Sep 11 02:21:00 cib-db03 multipathd: crs: remaining active paths: 7
Sep 11 02:21:00 cib-db03 multipathd: 8:128: mark as failed
Sep 11 02:21:00 cib-db03 multipathd: retail_0: remaining active paths: 7
Sep 11 02:21:00 cib-db03 multipathd: 8:80: mark as failed
Sep 11 02:21:00 cib-db03 multipathd: corpo_1: remaining active paths: 7
Sep 11 02:21:00 cib-db03 multipathd: 8:16: mark as failed
Sep 11 02:21:00 cib-db03 multipathd: hades_1: remaining active paths: 7
Sep 11 02:21:00 cib-db03 multipathd: dm-26: add map (uevent)
Sep 11 02:21:00 cib-db03 multipathd: dm-26: devmap already registered
Sep 11 02:21:00 cib-db03 multipathd: dm-20: add map (uevent)
Sep 11 02:21:00 cib-db03 multipathd: dm-20: devmap already registered
Sep 11 02:21:00 cib-db03 multipathd: dm-11: add map (uevent)
Sep 11 02:21:00 cib-db03 multipathd: dm-11: devmap already registered
Sep 11 02:21:00 cib-db03 multipathd: sdb: tur checker reports path is up
Sep 11 02:21:00 cib-db03 multipathd: 8:16: reinstated
Sep 11 02:21:00 cib-db03 multipathd: hades_1: remaining active paths: 8
Sep 11 02:21:00 cib-db03 multipathd: sdf: tur checker reports path is up
Sep 11 02:21:00 cib-db03 multipathd: 8:80: reinstated
Sep 11 02:21:00 cib-db03 multipathd: corpo_1: remaining active paths: 8
Sep 11 02:21:00 cib-db03 multipathd: sdi: tur checker reports path is up
Sep 11 02:21:00 cib-db03 multipathd: 8:128: reinstated
Sep 11 02:21:00 cib-db03 multipathd: retail_0: remaining active paths: 8
Sep 11 02:21:00 cib-db03 multipathd: sdk: tur checker reports path is up
Sep 11 02:21:00 cib-db03 multipathd: 8:160: reinstated
Sep 11 02:21:00 cib-db03 multipathd: crs: remaining active paths: 8
Sep 11 02:21:00 cib-db03 multipathd: dm-11: add map (uevent)
Sep 11 02:21:00 cib-db03 multipathd: dm-11: devmap already registered
Sep 11 02:21:00 cib-db03 multipathd: dm-20: add map (uevent)
Sep 11 02:21:00 cib-db03 multipathd: dm-20: devmap already registered
Sep 11 02:21:00 cib-db03 multipathd: dm-26: add map (uevent)
Sep 11 02:21:00 cib-db03 multipathd: dm-26: devmap already registered
Sep 11 02:21:00 cib-db03 multipathd: dm-0: add map (uevent)
Sep 11 02:21:00 cib-db03 multipathd: dm-0: devmap already registered
Sep 11 02:29:20 cib-db03 kernel: sd 1:0:0:8: SCSI error: return code = 0x000d0000
Sep 11 02:29:20 cib-db03 kernel: end_request: I/O error, dev sdbt, sector 1983
Sep 11 02:29:20 cib-db03 kernel: device-mapper: multipath: Failing path 68:112.
Sep 11 02:29:20 cib-db03 multipathd: dm-24: add map (uevent)
Sep 11 02:29:20 cib-db03 multipathd: dm-24: devmap already registered
Sep 11 02:29:20 cib-db03 multipathd: 68:112: mark as failed
Sep 11 02:29:20 cib-db03 multipathd: hades_0: remaining active paths: 7
Sep 11 02:29:29 cib-db03 multipathd: sdbt: tur checker reports path is up
Sep 11 02:29:29 cib-db03 multipathd: 68:112: reinstated
Sep 11 02:29:29 cib-db03 multipathd: hades_0: remaining active paths: 8
Sep 11 02:29:29 cib-db03 multipathd: dm-24: add map (uevent)
Sep 11 02:29:29 cib-db03 multipathd: dm-24: devmap already registered
Sep 11 02:44:03 cib-db03 kernel: sd 1:0:0:6: SCSI error: return code = 0x000d0000
Sep 11 02:44:03 cib-db03 kernel: end_request: I/O error, dev sdbr, sector 1985
Sep 11 02:44:03 cib-db03 kernel: device-mapper: multipath: Failing path 68:80.
Sep 11 02:44:03 cib-db03 kernel: sd 1:0:0:9: SCSI error: return code = 0x000d0000
Sep 11 02:44:03 cib-db03 kernel: end_request: I/O error, dev sdbu, sector 1985
Sep 11 02:44:03 cib-db03 kernel: device-mapper: multipath: Failing path 68:128.
Sep 11 02:44:03 cib-db03 kernel: sd 1:0:0:10: SCSI error: return code = 0x000d0000
Sep 11 02:44:03 cib-db03 kernel: end_request: I/O error, dev sdbv, sector 1985
Sep 11 02:44:03 cib-db03 kernel: device-mapper: multipath: Failing path 68:144.
Sep 11 02:44:03 cib-db03 kernel: sd 1:0:0:14: SCSI error: return code = 0x000d0000
Sep 11 02:44:03 cib-db03 kernel: end_request: I/O error, dev sdbz, sector 2050
Sep 11 02:44:03 cib-db03 kernel: device-mapper: multipath: Failing path 68:208.
Sep 11 02:48:22 cib-db03 syslogd 1.4.1: restart.
Sep 11 02:48:22 cib-db03 kernel: klogd 1.4.1, log source = /proc/kmsg started.
Sep 11 02:48:22 cib-db03 kernel: Linux version 2.6.18-128.el5 (mockbuild@hs20-bc1-7.build.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) #1 SMP Wed Dec 17 11:41:38 EST 2008
Sep 11 02:48:22 cib-db03 kernel: Command line: ro root=LABEL=/ rhgb quiet elevator=noop
Sep 11 02:48:22 cib-db03 kernel: BIOS-provided physical RAM map:
Sep 11 02:48:22 cib-db03 kernel: BIOS-e820: 0000000000000000 - 000000000009f400 (usable)
Sep 11 02:48:22 cib-db03 kernel: BIOS-e820: 000000000009f400 - 00000000000a0000 (reserved)
Sep 11 02:48:22 cib-db03 kernel: BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
Sep 11 02:48:22 cib-db03 kernel: BIOS-e820: 0000000000100000 - 00000000cfe4e000 (usable)
Sep 11 02:48:22 cib-db03 kernel: BIOS-e820: 00000000cfe4e000 - 00000000cfe56000 (ACPI data)
Sep 11 02:48:22 cib-db03 kernel: BIOS-e820: 00000000cfe56000 - 00000000cfe57000 (usable)
Sep 11 02:48:22 cib-db03 kernel: BIOS-e820: 00000000cfe57000 - 00000000e0000000 (reserved)
Sep 11 02:48:22 cib-db03 kernel: BIOS-e820: 00000000fec00000 - 00000000fed00000 (reserved)
Sep 11 02:48:22 cib-db03 kernel: BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
Sep 11 02:48:22 cib-db03 kernel: BIOS-e820: 00000000ffc00000 - 0000000100000000 (reserved)
Sep 11 02:48:22 cib-db03 kernel: BIOS-e820: 0000000100000000 - 000000082ffff000 (usable)
Sep 11 02:48:22 cib-db03 kernel: DMI 2.4 present.
Sep 11 02:48:22 cib-db03 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
Sep 11 02:48:22 cib-db03 kernel: SRAT: PXM 0 -> APIC 1 -> Node 0
Sep 11 02:48:22 cib-db03 kernel: SRAT: PXM 0 -> APIC 2 -> Node 0
Sep 11 02:48:22 cib-db03 kernel: SRAT: PXM 0 -> APIC 3 -> Node 0
Sep 11 02:48:22 cib-db03 kernel: SRAT: PXM 1 -> APIC 4 -> Node 1
Sep 11 02:48:22 cib-db03 kernel: SRAT: PXM 1 -> APIC 5 -> Node 1
Sep 11 02:48:22 cib-db03 kernel: SRAT: PXM 1 -> APIC 6 -> Node 1
Sep 11 02:48:22 cib-db03 kernel: SRAT: PXM 1 -> APIC 7 -> Node 1
Sep 11 02:48:22 cib-db03 kernel: SRAT: Node 0 PXM 0 0-a0000
Sep 11 02:48:22 cib-db03 kernel: SRAT: Node 0 PXM 0 0-d0000000
Sep 11 02:48:22 cib-db03 kernel: SRAT: Node 0 PXM 0 0-430000000
Sep 11 02:48:22 cib-db03 kernel: SRAT: Node 1 PXM 1 430000000-830000000
Sep 11 02:48:22 cib-db03 kernel: Bootmem setup node 0 0000000000000000-0000000430000000
Sep 11 02:48:22 cib-db03 kernel: Bootmem setup node 1 0000000430000000-000000082ffff000
Sep 11 02:48:22 cib-db03 kernel: Memory for crash kernel (0x0 to 0x0) notwithin permissible range
Sep 11 02:48:23 cib-db03 kernel: disabling kdump
Sep 11 02:48:23 cib-db03 kernel: ACPI: PM-Timer IO Port: 0x920
Sep 11 02:48:23 cib-db03 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Sep 11 02:48:23 cib-db03 kernel: Processor #0 0:4 APIC version 16
Sep 11 02:48:23 cib-db03 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled)
Sep 11 02:48:23 cib-db03 kernel: Processor #4 0:4 APIC version 16
Sep 11 02:48:23 cib-db03 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
Sep 11 02:48:23 cib-db03 kernel: Processor #1 0:4 APIC version 16
Sep 11 02:48:23 cib-db03 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] enabled)
Sep 11 02:48:23 cib-db03 kernel: Processor #5 0:4 APIC version 16
Sep 11 02:48:23 cib-db03 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
Sep 11 02:48:23 cib-db03 kernel: Processor #2 0:4 APIC version 16
Sep 11 02:48:23 cib-db03 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled)
Sep 11 02:48:23 cib-db03 kernel: Processor #6 0:4 APIC version 16
Sep 11 02:48:23 cib-db03 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
Sep 11 02:48:23 cib-db03 kernel: Processor #3 0:4 APIC version 16
Sep 11 02:48:23 cib-db03 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled)
Sep 11 02:48:23 cib-db03 kernel: Processor #7 0:4 APIC version 16
Sep 11 02:48:23 cib-db03 kernel: ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
Sep 11 02:48:23 cib-db03 kernel: ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
Sep 11 02:48:23 cib-db03 kernel: IOAPIC[0]: apic_id 8, version 17, address 0xfec00000, GSI 0-15
Sep 11 02:48:23 cib-db03 kernel: ACPI: IOAPIC (id[0x09] address[0xfec01000] gsi_base[16])
Sep 11 02:48:23 cib-db03 kernel: IOAPIC[1]: apic_id 9, version 17, address 0xfec01000, GSI 16-31
Sep 11 02:48:23 cib-db03 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
Sep 11 02:48:23 cib-db03 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
Sep 11 02:48:23 cib-db03 kernel: Setting APIC routing to physical flat
Sep 11 02:48:23 cib-db03 kernel: ACPI: HPET id: 0x1166a201 base: 0xfed00000
Sep 11 02:48:23 cib-db03 kernel: Using ACPI (MADT) for SMP configuration information
Sep 11 02:48:23 cib-db03 kernel: Nosave address range: 000000000009f000 - 00000000000a0000
Sep 11 02:48:23 cib-db03 kernel: Nosave address range: 00000000000a0000 - 00000000000f0000
Sep 11 02:48:23 cib-db03 kernel: Nosave address range: 00000000000f0000 - 0000000000100000
Sep 11 02:48:23 cib-db03 kernel: Nosave address range: 00000000cfe4e000 - 00000000cfe56000
Sep 11 02:48:24 cib-db03 kernel: Nosave address range: 00000000cfe57000 - 00000000e0000000
Sep 11 02:48:24 cib-db03 kernel: Nosave address range: 00000000e0000000 - 00000000fec00000
Sep 11 02:48:24 cib-db03 kernel: Nosave address range: 00000000fec00000 - 00000000fed00000
Sep 11 02:48:24 cib-db03 kernel: Nosave address range: 00000000fed00000 - 00000000fee00000
Sep 11 02:48:24 cib-db03 kernel: Nosave address range: 00000000fee00000 - 00000000fee10000
Sep 11 02:48:24 cib-db03 kernel: Nosave address range: 00000000fee10000 - 00000000ffc00000
Sep 11 02:48:24 cib-db03 kernel: Nosave address range: 00000000ffc00000 - 0000000100000000