rx 4640 crash dump recover

[Sakui]

Hi guys,

I have a rx 4640 with hp-ux 11v2 installed. The root password was lock and there was no way to telnet into the server with both SSH or port 23. The Server was hard restart by pressing the on/off button. Below are the errors i am getting. How do i recover this server.

I have try to boot into single user mode.But the crash dump is still coming.

The dump file seems too large for the disk



A system crash has occurred. (See the above messages for details.)
*** The system is now preparing to dump physical memory to disk, for use
*** in debugging the crash.

*** Cannot dump with compression because there are too few processors
*** Dumping without compression
*** The dump will be a SELECTIVE dump: 2021 of 32739 megabytes.
*** To change this dump type, press any key within 10 seconds.

*** Select one of the following dump types, by pressing the corresponding key:
N) There will be NO DUMP performed.
S) The dump will be a SELECTIVE dump: 2021 of 32739 megabytes.
P) The dump will be a PARTIAL dump: 12287 of 32739 megabytes.
(A FULL dump is not possible.)
*** Enter your selection now.
p*** Proceeding with partial dump.



Primary Dump Header Location :
Device : /dev/rdsk/c2t0d0 offset: 826228.
*** The dump may be aborted at any time by pressing ESC.
*** Dumping: 10% complete (1260 of 12287 MB) (device /dev/dsk/c2t0d0 )



* Cannot dump with compression because there are too few processors
*** Dumping without compression
*** The dump will be a SELECTIVE dump: 2021 of 32739 megabytes.
*** To change this dump type, press any key within 10 seconds.

*** Select one of the following dump types, by pressing the corresponding key:
N) There will be NO DUMP performed.
S) The dump will be a SELECTIVE dump: 2021 of 32739 megabytes.




A system crash has occurred. (See the above messages for details.)
*** The system is now preparing to dump physical memory to disk, for use
*** in debugging the crash.

*** Cannot dump with compression because there are too few processors
*** Dumping without compression
*** The dump will be a SELECTIVE dump: 2021 of 32739 megabytes.
*** To change this dump type, press any key within 10 seconds.

*** Select one of the following dump types, by pressing the corresponding key:
N) There will be NO DUMP performed.
S) The dump will be a SELECTIVE dump: 2021 of 32739 megabytes.
P) The dump will be a PARTIAL dump: 12287 of 32739 megabytes.
(A FULL dump is not possible.)
*** Enter your selection now.
n*** Dump disabled. System rebooting.




10 SFW 0 0 0x148002C500E008D0 0000000000000000 BOOT_REBOOT
19 Jul 2008 00:08:00
109 BMC 2 0x20488130520208C0 FFFF027000120300 Type-02 127002 1208322
19 Jul 2008 00:07:46
108 HPUX 0 *3 0x7F80033700E008A0 00000000000AD700 HP-UX_CRASHDUMP_STARTED
19 Jul 2008 00:07:29
107 HPUX 0 *5 0xBF80033800E00880 000000000002B000 HP-UX_HEX_FAULT_CODE
19 Jul 2008 00:07:27
106 OSA *7 0x214881303C020870 FF0F016F00200300 Type-02 206f01 2125569
19 Jul 2008 00:07:24
105 SFW 0 0 0x148002C500E00850 0000000000000000 BOOT_REBOOT
19 Jul 2008 00:00:59
104 BMC 2 0x2048812EAD020840 FFFF027000120300 Type-02 127002 1208322
19 Jul 2008 00:00:45
103 HPUX 0 *3 0x7F80033700E00820 00000000000AD700 HP-UX_CRASHDUMP_STARTED
19 Jul 2008 00:00:13
102 HPUX 0 *5 0xBF80033800E00800 000000000002B000 HP-UX_HEX_FAULT_CODE
19 Jul 2008 00:00:1




-------------------------------------------------------------------------------
101 OSA *7 0x2148812E870207F0 FF0F016F00200300 Type-02 206f01 2125569
19 Jul 2008 00:00:07
100 SFW 0 0 0x148002C500E007D0 0000000000000000 BOOT_REBOOT
18 Jul 2008 23:53:46
99 BMC 2 0x2048812CFB0207C0 FFFF027000120300 Type-02 127002 1208322
18 Jul 2008 23:53:31
98 HPUX 0 *3 0x7F80033700E007A0 00000000000AD700 HP-UX_CRASHDUMP_STARTED
18 Jul 2008 23:48:21
97 HPUX 0 *5 0xBF80033800E00780 000000000002B000 HP-UX_HEX_FAULT_CODE
18 Jul 2008 23:48:19
96 OSA *7 0x2148812BBF020770 FF0F016F00200300 Type-02 206f01 2125569
18 Jul 2008 23:48:15
95 SFW 0 0 0x148002C500E00750 0000000000000000 BOOT_REBOOT
18 Jul 2008 23:41:55
94 BMC 2 0x2048812A34020740 FFFF027000120300 Type-02 127002 1208322
18 Jul 2008 23:41:40
93 HPUX 0 *3 0x7F80033700E00720 00000000000AD700 HP-UX_CRASHDUMP_STARTED
18 Jul 2008 23:41:

P) The dump will be a PARTIAL dump: 12287 of 32739 megabytes.
(A FULL dump is not possible.)




vices/Device/USB/Standard/Avocent/KVMAdaptor/1_0
<2.1.3.10.1572.833.256>
NOTICE: USB device attached. Identification String:
Devices/Keyboard/USB/Boot/Avocent/KVMAdaptor/1_0
<2.305.3.100.1572.833.256>
NOTICE: USB device attached. Identification String:
Devices/Mouse/USB/Boot/Avocent/KVMAdaptor/1_0
<2.307.3.100.1572.833.256>
CISS: RAID SA controller on hardware path 0/1/1/0/4/0 is now online.
td: claimed Tachyon XL2 Fibre Channel Mass Storage card at 0/4/2/0
Boot device's HP-UX HW path is: 0/1/1/0/4/0.0.0

System Console is on the Built-In Serial Interface
iether0: INITIALIZING HP AB352-60003 PCI/PCI-X 1000Base-T Dual-port Core at hard
ware path 0/1/2/0
iether1: INITIALIZING HP AB352-60003 PCI/PCI-X 1000Base-T Dual-port Core at hard
ware path 0/1/2/1
Logical volume 64, 0x3 configured as ROOT
Logical volume 64, 0x2 configured as SWAP
Logical volume 64, 0x2 configured as DUMP
Swap device table: (start & size given in 512-byte blocks)
entry 0 - major is 64, minor is 0x2; start = 0, size = 25165824
Starting the STREAMS daemons-phase 1


*** Enter your selection now.
p*** Proceeding with partial dump.



Primary Dump Header Location :
Device : /dev/rdsk/c2t0d0 offset: 826228.
*** The dump may be aborted at any time by pressing ESC.
*** Dumping: 10% complete (1260 of 12287 MB) (device /dev/dsk/c2t0d0 )
*** Dumping: 15% complete (1896 of 12287 MB) (device /dev/dsk/c2t0d0 )
*** Dumping: 20% complete (2532 of 12287 MB) (device /dev/dsk/c2t0d0 )
*** Dumping: 25% complete (3164 of 12287 MB) (device /dev/dsk/c2t0d0 )
*** Dumping: 30% complete (3800 of 12287 MB) (device /dev/dsk/c2t0d0 )

[YAQUB_1]

Hi,

System MP envent log analysis is here:

1. Keyword = HP-UX_HEX_FAULT_CODE
Possible I/O error or system panic

2. Keyword = OS_CRITICAL_SHUTDOWN
OS run-time critical shutdown.

Now U can successfully OS boot up or still same think.

>>> The root password was lock and there was no way to telnet into the server with both SSH or port 23.

==> Sakui, do not doing in future in that station. BTW, U can do it now login MP LAN then execute "MP> cm" then "pc" then "off" then "ctrl+B" (have U any basic knowledge on system H/W level) then U can check Ur system SCSI card or HBA card most properly one of card create that panic. Before doing U just unplug the power cable, SCSI cable, Fiber cable & other cable.

Again "MP> cm" then "pc" then "on" & then "ctrl+B" then "MP> co"
-- Interrupt the boot process and you will get this:

Main Menu: Enter command or "menu > bo pri"
Interact with IPL (Y, N, or Cancel)?> "y"
ISL> hpux -is.

Now type "passwd". -------- change the root password

Then reboot the system. Now see what is status Ur system

BR/
Yaqub

[Sakui]

Hi

This server is not a PA RISC but rather an itanium Server.so the is no ISL or BCH prompt. I have tried single user mode , Maintaintence mode LVM mode and etc.Still the server Can't boot

i also found the following error ync'ing disks (0 buffers to flush): (0 buffers to flush):
0 buffers not flushed
0 buffers still dirty


panic: all VFS_MOUNTROOTs failed: NEED DRIVERS ?????tion 1tgt..




Logical volume 64, 0x2 configured as SWAPmary Boot: 0/1/1/0/4/0.0.0er' returned Su
Logical volume 64, 0x2 configured as DUMPP-UX Primary Boot: 0/1/1/0/4/0.0.0d respo
Swap device table: (start & size given in 512-byte blocks)Hewlett-Packard Development Company, L.P.1/0/4/0.0.0 sdisking t
entry 0 - major is 64, minor is 0x2; start = 0, size = 25165824ontroller

HP-UX Boot Loader for IPF -- Revision 2.028forme
The
Dump device table: (star


HP-UX Boot Loader for IPF -- Revision 2.028oSCSI Port 1: SCSI ID(s) 0 [ Unknown EMU ]

Press Any Key to interrupt Autobootn is 01.03.35.69



Searching fo
0/
\EFI\HPUX\AUTO ==> boot vmunixng the Ultra320 SCSI Controlle
Seconds left till autoboot - 9 versioreturn to MP main menu.)*
Type 'help' for helpP Fibre Channel Mass St

HPUX> boot testvmunix699)r Console Outp
0/
Could not open file testvmunix to check for a directory
0/3/1/0.6 tgturation of de
0/3/1/0.6.0 stapeeveral min
Could not open testvmunix
0/3/1/0.7.0 sctl

loading failed





Could not open /stand/vmunix.prev
iether1: INITIALIZING HP AB352-

loading failed1000Base-T Dua

HPUX> boot vmunix.prev 327
Loaded ACPI revis
Could not open file vmunix.prev to check for a directorye path 0/1/2/14.10 [4631] ºa
Logical volume 64, 0x3 c
Could not open vmunix.prev

MFS is defined: b


loading failedlume 64, 0x2 c

HPUX> boot -is vmunix 4.03 º
> System Memory = 32738 MBme 64, 0x2 configured as D
loading section 0size=
º Drive E








/1/1/0/4/0.0.0 sdisklowing SCSI dri
All rights reservedthe Ultra320 SCSI C

HP-UX Boot Loader for IPF -- Revision 2.028oSCSI Port 1: SCSI ID(s) 0 [ Unknown EMU ]

Press Any Key to interrupt Autobootn is 01.03.35.69



Searching fo
0/
\EFI\HPUX\AUTO ==> boot vmunixng the Ultra320 SCSI Controlle
Seconds left till autoboot - 9 versioreturn to MP main menu.)*
Type 'help' for helpP Fibre Channel Mass St

HPUX> boot testvmunix699)r Console Outp
0/
Could not open file testvmunix to check for a directory
0/3/1/0.6 tgturation of de
0/3/1/0.6.0 stapeeveral min
Could not open testvmunix
0/3/1/0.7.0 sctl

loading failed



Could not open /stand/vmunix.prev
iether1: INITIALIZING HP AB352-

loading failed1000Base-T Dua

HPUX> boot vmunix.prev 327
Loaded ACPI revis
Could not open file vmunix.prev to check for a directorye path 0/1/2/14.10 [4631] ºa
Logical volume 64, 0x3 c
Could not open vmunix.prev

MFS is defined: b


loading failedlume 64, 0x2 c

HPUX> boot -is vmunix 4.03 º
> System Memory = 32738 MBme 64, 0x2 configured as D
loading section 0size=
º Drive E
.......................................................... (complete)ks)fs_link(): File system was registered at index 5. by pressing t

loading section 1jor is 64, minor
.............. (complete)2






tack Trace:i
Booting kernel... F


Memory Class Setup à à à à à à à à à à à à à à à à Ã
-------------------------------------------------------------------------UMP performed.tim
0xe0000000011d6000 im_preinitrc+0x300

Class Physmem Lockmem Swapmemx3a0. Identification String:
End of Stack Tra
------------------------------------------------------------------------- 1.6 GHz Active º






oading
* ROM Date : 08/01/20
Devices/Mouse/USB/Boot/Avocent/KVMAdaptor/1_004.03br
loading symbol tabl
*****************





Calling function e00000000107d1e0 for Shutdown State 8 type 0x2 Success: cachefs_link(): File system was registered at index 5

Stored message buffer up to panic:adcom Gigabit Ethernet Driver'...E
Found adjacent data tr. Growing size. 0x33fc000 -> 0x73fc000.c) 2000-2005 Broadcom Corporation key:ase-T Dual
NOTICE: mod_fs
Pinned PDK malloc pool: base: 0xe000000100c04000 size=118768K7.0.10iven in 1-Kbyte blocks)h 0/1/2/0
ieth


Loaded ACPI revision 2.0 tables.net Driver' returned Success- ma
MMIO on this platform supports Write Coalescing. Identi
Loading 'SysROM:SCSI Bus Driver'...IAL

MFS is defined: base= 0xe000000100c04000 size= 5068 KBr' returned Success1_0daemons-phase 1
(A FULL dump


Unpinned PDK malloc pool: base: 0xe000000110000000 size=524288KNOTICE: USB de
-- 'SysROM:SCSI Tape Driver' returned Successd re
NOTICE: cachefs_link(): File system wa







npinned PDK malloc p
n is 01.03.35.69rnal Bootable DV
0/3/1/0 mpt º º BMC V
Initializing the Ultra320 SCSI Controller at 0/3/1/1. Controller firmware versiosystem was registered at
º Drive Explorer º º MP Version:
n is 01.03.35.69 ºE: nfs3_link
0/3/1/1 mpttem was reg
0/4 lbaat inde
0/4/2/0 td0 0x000009
0/3/1/0.6 tgt-------------
0/3/1/0.6.0 stape º Installed Memo
0/3/1/0.7 tgt ºfigured
0/3/1/0.7.0 sctlom KRS for modul
0/3/1/1.7 tgt





Use < | > to scroll to Select
ware path 0/1/2/1 for Previous Men
Logical volume 64, 0x3 configured as ROOT 27006 MB
Loa
CISS: RAID SA control





Logical volume 64, 0x2 configured as SWAPmary Boot: 0/1/1/0/4/0.0.0er' returned Su
Logical volume 64, 0x2 configured as DUMPP-UX Primary Boot: 0/1/1/0/4/0.0.0d respo
Swap device table: (start & size given in 512-byte blocks)Hewlett-Packard Development Company, L.P.1/0/4/0.0.0 sdisking t
entry 0 - major is 64, minor is 0x2; start = 0, size = 25165824ontroller



HP-UX Boot Loader for IPF -- Revision



Use < | > to scroll to Select
ware path 0/1/2/1 for Previous Men
Logical volume 64, 0x3 configured as ROOT 27006 MB
Loa
CISS: RAID SA control

Starting the STREAMS daemons-phase 1.......................... (complete

System Panic:ce
0/3/1/0.7

panic: all VFS_MOUNTROOTs failed: NEED DRIVERS ?????tion 1tgt..


.............. (complete)ann


How can i revive this Server with this disks without re-installation.

[YAQUB_1]

Have U any Ignite Backup, then U try to restore through Ignite......

[Sakui]

i have no ignite backup. I strongly believe my Smart array is the one causing this problem. I have remove the smart array and attached the disk to a scsi card still i can't boot either to single user mode or multi user mode.

i am still stoping at the crash dump percentage range.

[YAQUB_1]

Have U any additional SmartArray Controller then replace & try to boot the system or U can replace internal HDD.....

BR/
Yaqub

[Sakui]

hi YAQUB

I will try that.wht have you analyzed from all the logs i have posted. I need ur help on this issue.