Deleted files during power outage , Oracle unable to start

[samer1]

Hi all,

we have an HP DL580 oracle 10 server that has a problem starting oracle , because of a power outage, the system goes down without a normal shutdown . When powering up we were unable to mount the oracle disk , the OS (redhat 4 update 5) says: no such file /dev/sda1 . Unfortunately we don't have a backup of the system , how can we find a solution to this issue. We found that the special file /dev/sda1 /dev/sda2 and /dev/sda3 were deleted during the power outage. We were unable to restore these files. I have to precise that this disk is on a n HP EVA6000 , this disk array is configured as a SAN and there is another server (another HP DL580) that is connected to this SAN.
Thanks in advance

[J. Maestre]

Does dmesg say anything at all about that disk?

[samer1]

Here is what does 'dmesg' says :

Thanks for your reply , please ask me if you need more additional informations .

TIA


Output of dmesg:
---------------

Bootdata ok (command line is ro root=LABEL=/1 rhgb quiet)
Linux version 2.6.9-55.0.2.ELsmp
qla2xxx 0000:03:01.0:
QLogic Fibre Channel HBA Driver: 8.02.11-fo
QLogic HP AB429-60001 -
ISP2422: PCI-X Mode 1 (133 MHz) @ 0000:03:01.0 hdma+, host#=0, fw=4.03.02 [IP] [84XX]
scsi: unknown device type 12 ?????????
Vendor: HP Model: HSV200 Rev: 6000
Type: RAID ANSI SCSI revision: 05
qla2xxx 0000:03:01.0: scsi(0:0:2:0): Enabled tagged queuing, queue depth 16.
Vendor: HP Model: HSV200 Rev: 6000
Type: Direct-Access ANSI SCSI revision: 05
qla2xxx 0000:03:01.0: scsi(0:0:2:2): Enabled tagged queuing, queue depth 16.
scsi: unknown device type 12
Vendor: HP Model: HSV200 Rev: 6000
Type: RAID ANSI SCSI revision: 05
qla2xxx 0000:03:01.0: scsi(0:0:3:0): Enabled tagged queuing, queue depth 16.
scsi: Unexpected response from host 0 channel 0 id 3 lun 2 while scanning, scan aborted
ACPI: PCI Interrupt 0000:06:01.0[A] -> GSI 97 (level, low) -> IRQ 233
qla2xxx 0000:06:01.0: Found an ISP2422, irq 233, iobase 0xffffff000001c000
qla2xxx 0000:06:01.0: Configuring PCI space...
qla2xxx 0000:06:01.0: Configure NVRAM parameters...
qla2xxx 0000:06:01.0: Verifying loaded RISC code...
qla2xxx 0000:06:01.0: Allocated (64 KB) for EFT...
qla2xxx 0000:06:01.0: Allocated (1413 KB) for firmware dump...
qla2xxx 0000:06:01.0: Waiting for LIP to complete...
qla2xxx 0000:06:01.0: LIP reset occured (f700).
qla2xxx 0000:06:01.0: LOOP UP detected (4 Gbps).
qla2xxx 0000:06:01.0: Topology - (F_Port), Host Loop address 0x0
scsi1 : qla2xxx
qla2xxx 0000:06:01.0:
QLogic Fibre Channel HBA Driver: 8.02.11-fo
QLogic HP AB429-60001 -
ISP2422: PCI-X Mode 1 (133 MHz) @ 0000:06:01.0 hdma+, host#=1, fw=4.03.02 [IP] [84XX]
scsi: unknown device type 12
Vendor: HP Model: HSV200 Rev: 6000
Type: RAID ANSI SCSI revision: 05
qla2xxx 0000:06:01.0: scsi(1:0:2:0): Enabled tagged queuing, queue depth 16.
scsi: Unexpected response from host 1 channel 0 id 2 lun 2 while scanning, scan aborted
scsi: unknown device type 12
Vendor: HP Model: HSV200 Rev: 6000
Type: RAID ANSI SCSI revision: 05
qla2xxx 0000:06:01.0: scsi(1:0:4:0): Enabled tagged queuing, queue depth 16.
scsi: Unexpected response from host 1 channel 0 id 4 lun 2 while scanning, scan aborted
SCSI device sda: 4292870144 512-byte hdwr sectors (2197950 MB)
SCSI device sda: drive cache: write through
SCSI device sda: 4292870144 512-byte hdwr sectors (2197950 MB)
SCSI device sda: drive cache: write through
sda: unknown partition table ??????????
Attached scsi disk sda at scsi0, channel 0, id 2, lun 2
libata version 2.00 loaded.
Fusion MPT base driver 3.02.73rh
Copyright (c) 1999-2006 LSI Logic Corporation
Fusion MPT FC Host driver 3.02.73rh
Fusion MPT SPI Host driver 3.02.73rh
ACPI: PCI Interrupt 0000:12:08.0[A] -> GSI 16 (level, low) -> IRQ 169
mptbase: Initiating ioc0 bringup
ioc0: 53C1030: Capabilities={Initiator,Target}
scsi2 : ioc0: LSI53C1030, FwRev=01032700h, Ports=1, MaxQ=255, IRQ=169
Vendor: HP Model: SDLT600 Rev: 2929
Type: Sequential-Access ANSI SCSI revision: 03
Fusion MPT SAS Host driver 3.02.73rh
HP CISS Driver (v 2.6.14.RH2)
cciss: Device 0x3230 has been found at bus 14 dev 0 func 0
ACPI: PCI Interrupt 0000:0e:00.0[A] -> GSI 16 (level, low) -> IRQ 169
MSI INIT SUCCESS
cciss: using DAC cycles
blocks= 143305919 block_size= 512
heads= 255, sectors= 32, cylinders= 17562

blocks= 143305919 block_size= 512
heads= 255, sectors= 32, cylinders= 17562

cciss/c0d0: p1 p2 p3
EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting. Commit interval 5 seconds
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
SELinux: Disabled at runtime.
SELinux: Unregistering netfilter hooks
st: Version 20040403, fixed bufsize 32768, s/g segs 256
Attached scsi tape st0 at scsi2, channel 0, id 6, lun 0
st0: try direct i/o: yes (alignment 512 B), max page reachable by HBA 4503599627370495
Attached scsi generic sg0 at scsi0, channel 0, id 2, lun 0, type 12
Attached scsi generic sg1 at scsi0, channel 0, id 2, lun 2, type 0
Attached scsi generic sg2 at scsi0, channel 0, id 3, lun 0, type 12
Attached scsi generic sg3 at scsi1, channel 0, id 2, lun 0, type 12
Attached scsi generic sg4 at scsi1, channel 0, id 4, lun 0, type 12
Attached scsi generic sg5 at scsi2, channel 0, id 6, lun 0, type 1
qla2322: disagrees about version of symbol qla2x00_probe_one
qla2322: Unknown symbol qla2x00_probe_one
qla2322: disagrees about version of symbol qla2x00_probe_one
qla2322: Unknown symbol qla2x00_probe_one
inserting floppy driver for 2.6.9-55.0.2.ELsmp
Floppy drive(s): fd0 is 1.44M
floppy0: no floppy controllers found
Ethernet Channel Bonding Driver: v2.6.3-rh (June 8, 2005)
bonding: MII link monitoring set to 100 ms
divert: allocating divert_blk for bond0
Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.4.43-rh (June 28, 2006)
ACPI: PCI Interrupt 0000:0a:01.0[A] -> GSI 24 (level, low) -> IRQ 201
divert: allocating divert_blk for eth0
eth0: Broadcom NetXtreme II BCM5706 1000Base-T (A2) PCI-X 64-bit 100MHz found at mem fa000000, IRQ 201, node addr 001b784106d6
ACPI: PCI Interrupt 0000:0a:02.0[A] -> GSI 27 (level, low) -> IRQ 209
divert: allocating divert_blk for eth1
eth1: Broadcom NetXtreme II BCM5706 1000Base-T (A2) PCI-X 64-bit 100MHz found at mem f8000000, IRQ 209, node addr 001b784106d8
hw_random hardware driver 1.0.0 loaded
USB Universal Host Controller Interface driver v2.2
ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 16 (level, low) -> IRQ 169
uhci_hcd 0000:00:1d.0: UHCI Host Controller
PCI: Setting latency timer of device 0000:00:1d.0 to 64
uhci_hcd 0000:00:1d.0: irq 169, io base 0000000000001000
uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 1

ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 185
uhci_hcd 0000:00:1d.2: UHCI Host Controller
PCI: Setting latency timer of device 0000:00:1d.2 to 64

uhci_hcd 0000:01:04.4: port count misdetected? forcing to 2 ports


hub 6-0:1.0: 8 ports detected
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
usb 5-1: new full speed USB device using address 2
input: USB HID v1.01 Keyboard [HP Virtual Keyboard] on usb-0000:01:04.4-1
input: USB HID v1.01 Mouse [HP Virtual Keyboard] on usb-0000:01:04.4-1
usb 5-2: new full speed USB device using address 3
hub 5-2:1.0: USB hub found
hub 5-2:1.0: 7 ports detected
NET: Registered protocol family 10
Disabled Privacy Extensions on device ffffffff80408980(lo)
IPv6 over IPv4 tunneling driver
divert: not allocating divert_blk for non-ethernet device sit0
mtrr: type mismatch for e8000000,2000000 old: write-back new: write-combining
ACPI: Power Button (FF) [PWRF]
EXT3 FS on cciss/c0d0p1, internal journal
device-mapper: 4.5.5-ioctl (2006-12-01) initialised: dm-devel@redhat.com

kjournald starting. Commit interval 5 seconds
EXT3 FS on cciss/c0d0p3, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
Adding 32764556k swap on /dev/cciss/c0d0p2. Priority:-1 extents:1

microcode: CPU7 already at revision 0x9 (current=0x9)

cdrom: open failed.
Linux Kernel Card Services
options: [pci] [cardbus] [pm]
ip_tables: (C) 2000-2002 Netfilter core team
ip_tables: (C) 2000-2002 Netfilter core team
bnx2: eth0: using MSI
ADDRCONF(NETDEV_UP): eth0: link is not ready
bonding: bond0: enslaving eth0 as an active interface with a down link.
bnx2: eth0 NIC Link is Up, 1000 Mbps full duplex
ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
bonding: bond0: link status definitely up for interface eth0.
bond0: no IPv6 routers present
eth0: no IPv6 routers present
i2c /dev entries driver
lp: driver loaded but no devices found
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
mtrr: type mismatch for e8000000,2000000 old: write-back new: write-combining


#############################################
[root# fdisk /dev/sda
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.


The number of cylinders for this disk is set to 267218.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help):

[Ivan Ferreira]

This looks really bad, and how come you don't have a backup!

Did the power outage affected the storage? Is the vdisk in correct state or there are invalid data on cache displayed on the command view?

[J. Maestre · Accepted solution]

If your partition table has indeed gone fubar, as it seems to be the case, you could try either Parted (gpart) or TestDisk to try to rebuild it.

Even if the partition table is lost I think you might still be able to dd the disk so you can play it safer on the recovery attempts.

[samer1]

Thank you all for the quick reply, thanks to J. Maestre who suggested to use test-disk (Partition table recovery using TestDisk is really easy) http://www.cgsecurity.org/wiki/TestDisk , then it was the good trick ! I used this tool and the disk /dev/sda has been recovered and fdisk now can sees two partitions (before running this utility fdisk gives this :
fdisk /dev/sda

The number of cylinders for this disk is set to 267218.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Many thanks to all who replied and a great thanks again to J. Maestre.

Bye

[samer1]

I have found a solution to this issue , my disk that was unable to be mounted has now been fixed with the 'TestDisk' utility designed by(Christophe Grenier), thanks a lot to all who responded.