How to restore a filesystem , after disk failure ?

[rveri]

Hi All ,

We have a vg named /dev/vg01_data in which , there was 12 physical disk , consisting of 4 LV , each LV is having 3 disks.

Here is details of lvmtab:
----------------------------
/dev/vg00
/dev/dsk/c1t6d0
/dev/dsk/c2t6d0
/dev/vg01_data
/dev/dsk/c3t0d0
/dev/dsk/c3t1d0
/dev/dsk/c3t2d0
/dev/dsk/c3t4d0
/dev/dsk/c3t5d0
/dev/dsk/c3t6d0
/dev/dsk/c7t0d0
/dev/dsk/c7t1d0
/dev/dsk/c7t2d0
/dev/dsk/c7t4d0
/dev/dsk/c7t5d0
/dev/dsk/c7t6d0

Filesystems are on that vg:
------------------------------
root@mslt01:/data/data03[81]# bdf | grep data03
/dev/vg01_data/lvdata03
53329920 14536320 38490592 27% /data/data03

root# bdf | grep data

/dev/vg01_data/lvdata04
53329920 24742664 28363976 47% /data/data04
/dev/vg01_data/lvdata03
53329920 14536320 38490592 27% /data/data03
/dev/vg01_data/lvdata02
53329920 11590744 41413152 22% /data/data02
/dev/vg01_data/lvdata01
53329920 10672824 42323904 20% /data/data01

------


PROBLEM: The disk c3t5d0 is failed . And hardware engg came and replace the disk.

Now system shows c3t5d0 ok in ioscan. And lvdisplay display of /dev/vg00_data1/data03 , is having two disk.

How to include the 3rd disk (Replaced disk) , i.e c3t5d0 , without loosing any data as we found that the data is intact on that filesystem (/data/data03).

Each disk is 18GB , total filesystem size is 51GB and usuage is 14GB , we can see the 14GB files is intact(There are big oracle files we can see).

Please suggest how to include the 3rd disk (c3t5d0) without loosing any current data of that FS.

Thanks.

[A. Clay Stephenson]

First thing to do is an vgdisplay -v /dev/vg01_data and post that. Next, you need to do an lvdisplay /dev/vg01_data/lvname for each LVOL and post that output.

Don't do anything without fully understanding the consequences. If your LVOL's are mirrored then this should be no big deal; if not, then you are probably toast. Of course, you have a known good backup.

[Victor Fridyev]

Hi,
Do you have a backup of the FS ?
If the FS is intact, that means that before the disk replacing, it was removed from the VG. In this case you have to add the new disk to the VG:
pvcreate /dev/rdsk/c3t5d0
vgextend /dev/vg00_date /dev/dsk/c3t5d0
lvextend -L NEW_SIZE /dev/vg00_data1/data03 /dev/dsk/c3t5d0
umount /data/data03
extendfs /dev/vg00_data1/rdata03
mount /data/data03



HTH

[rveri]

Hi Clay,

Thanks for quick reply.

Here it is :

#vgdisplay -v /dev/vg01_data
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c3t5d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
--- Volume groups ---
VG Name /dev/vg01_data
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 4
Open LV 4
Max PV 25
Cur PV 12
Act PV 11
Max PE per PV 4350
VGDA 22
PE Size (Mbytes) 4
Total PE 47740
Alloc PE 47740
Free PE 0
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c3t5d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
--- Logical volumes ---
LV Name /dev/vg01_data/lvdata01
LV Status available/syncd
LV Size (Mbytes) 52080
Current LE 13020
Allocated PE 13020
Used PV 3

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c3t5d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name /dev/vg01_data/lvdata02
LV Status available/syncd
LV Size (Mbytes) 52080
Current LE 13020
Allocated PE 13020
Used PV 3

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c3t5d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name /dev/vg01_data/lvdata03
LV Status available/syncd
LV Size (Mbytes) 52080
Current LE 13020
Allocated PE 13020
Used PV 2

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c3t5d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name /dev/vg01_data/lvdata04
LV Status available/syncd
LV Size (Mbytes) 52080
Current LE 13020
Allocated PE 13020
Used PV 3


--- Physical volumes ---
PV Name /dev/dsk/c3t0d0
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c3t5d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
PV Status available
Total PE 4340
Free PE 0
Autoswitch On

PV Name /dev/dsk/c3t1d0
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c3t5d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
PV Status available
Total PE 4340
Free PE 0
Autoswitch On

PV Name /dev/dsk/c3t2d0
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c3t5d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
PV Status available
Total PE 4340
Free PE 0
Autoswitch On

PV Name /dev/dsk/c3t4d0
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c3t5d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
PV Status available
Total PE 4340
Free PE 0
Autoswitch On

PV Name /dev/dsk/c3t6d0
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c3t5d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
PV Status available
Total PE 4340
Free PE 0
Autoswitch On

PV Name /dev/dsk/c7t0d0
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c3t5d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
PV Status available
Total PE 4340
Free PE 0
Autoswitch On

PV Name /dev/dsk/c7t1d0
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c3t5d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
PV Status available
Total PE 4340
Free PE 0
Autoswitch On

PV Name /dev/dsk/c7t2d0
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c3t5d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
PV Status available
Total PE 4340
Free PE 0
Autoswitch On

PV Name /dev/dsk/c7t4d0
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c3t5d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
PV Status available
Total PE 4340
Free PE 0
Autoswitch On

PV Name /dev/dsk/c7t5d0
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c3t5d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
PV Status available
Total PE 4340
Free PE 0
Autoswitch On

PV Name /dev/dsk/c7t6d0
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c3t5d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
PV Status available
Total PE 4340
Free PE 0
Autoswitch On


root#:/data/data03[84]#
----------------------------------


2]

# lvdisplay /dev/vg01_data/lvdata03
lvdisplay: Warning: couldn't query physical volume "/dev/dsk/c3t5d0":
The specified path does not correspond to physical volume attached to
this volume group
lvdisplay: Warning: couldn't query all of the physical volumes.
--- Logical volumes ---
LV Name /dev/vg01_data/lvdata03
VG Name /dev/vg01_data
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 52080
Current LE 13020
Allocated PE 13020
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation strict
IO Timeout (Seconds) default

---------------------------------------

thanks.

[rveri]

Victor ,

The faulty disk was part of that LV , as vgdisplay shows 12 PV .

Also lvdisplay -v /dev/vg01_data/lvdata03 shows ???? at the place of this disk.

Here is fraction of the output :

04330 /dev/dsk/c3t4d0 04330 current
04331 /dev/dsk/c3t4d0 04331 current
04332 /dev/dsk/c3t4d0 04332 current
04333 /dev/dsk/c3t4d0 04333 current
04334 /dev/dsk/c3t4d0 04334 current
04335 /dev/dsk/c3t4d0 04335 current
04336 /dev/dsk/c3t4d0 04336 current
04337 /dev/dsk/c3t4d0 04337 current
04338 /dev/dsk/c3t4d0 04338 current
04339 /dev/dsk/c3t4d0 04339 current
04340 ??? 00000 current
04341 ??? 00001 current
04342 ??? 00002 current
04343 ??? 00003 current
04344 ??? 00004 current
04345 ??? 00005 current
04346 ??? 00006 current
04347 ??? 00007 current
04348 ??? 00008 current
04349 ??? 00009 current
04350 ??? 00010 current
04351 ??? 00011 current
04352 ??? 00012 current
04353 ??? 00013 current
04354 ??? 00014 current
..
..
13013 /dev/dsk/c3t6d0 04333 current
13014 /dev/dsk/c3t6d0 04334 current
13015 /dev/dsk/c3t6d0 04335 current
13016 /dev/dsk/c3t6d0 04336 current
13017 /dev/dsk/c3t6d0 04337 current
13018 /dev/dsk/c3t6d0 04338 current
13019 /dev/dsk/c3t6d0 04339 current

-----------------------

Please advice..

[A. Clay Stephenson · Accepted solution]

Well since you didn't mirror, you deserve to be clobbered but if The Force is with you, you may recover but it is very doubtful.

First, we need to restore the LVM structures to the replacement disk.

vgcfgrestore -n /dev/vg01_data /dev/rdsk/c3t5d0

Nest, we need to reactivate the VG to cause any missing disks to be included.

vgchange -a y /dev/vg01_data

At this point, do a vgdisplay -v /dev/vg01_data and that output should appear normal; however, this in no way means the data are intact.

Just because you can ls the files in the filesystem does not mean that the files are intact. It's possible that this was unused space within the VG but I rather doubt it.

About the only thing you can try to do is shutdown the database and backup all the files. Your backup will really only be useful with the database down. Immediately after shutting down, try to copy the files or back them up to tape. If you have 14GB files then tar or cpio will not work; you must use fbackup or a commercial tool like Data Protector.


You can try fsck'ing the filesystem after shutting down the database and unmounting the filesystem but I suspect that is hopeless. The filesystem will need to be recreated and then resored from your copies or backup.

Don't never ever run no databases without mirrors or some form of redundancy like an array.

[Helen French]

Did you unmount the file system before you replaced the disk? Did you remove (or export) this HDD before you replaced the disk? Normally, if you have a good backup of the file system, all you need to do is unmount the FS, replace the disk, run a 'vgcfgrestore' to the new disk and format the FS with 'newfs' command. After this, restore your data from backup.

[rveri]

no we did not unmount the FS , before replacing the disk. , We shutdown the system. And then replaced the disk.

Thanks for all the replies.

[Raj D.]

Hi ,


1] Take a good backup of that FS after DB shutdown as said above. tar can take file size upto 8GB , with latest tar patch.

2] # vgcfgrestore -n /dev/vg01_data /dev/rdsk/c3t5d0

3] # vgchange -a y /dev/vg0a_data


(Make sure you have backup of this FS, mkfs will re-create the FS. )

4] # umount /data/data03 #(unmount the FS)

5] # mkfs -F vxfs /dev/vg01_data/rlvdata03

6] # mountall
check with bdf for mounting of /data/data03

7] # cd /data/data03
# tar -xvf *


Enjoy ,
hth,

Raj.