Bad JBOD disk and need help restoring...

[Coolmar]

Hi folks,

I have a guy who lost a disk on his hpux 11.0 system. It is a jamaica disk and it was part of or all of vg01. The problem is that he has no record of what disk(s) made up vg01. All he has is a map file. He wants me to "fix" it. When I go into SAM I find there are 4 disks that do not belong to a volume group...so I assume that after the one disk died, the whole vg01 disappeared and the other three disks that probably belonged to that volume group are now not belonging to anything. None of this stuff was mirrored either. So can someone tell me what I have to do (step by step) to get these disks back into vg01 so that I can restore from tape. Do I have to just vgimport or pvcreate first....then only the one disk that was replaced, or that and the other three disks that in SAM don't belong to a volume group?

[paolo barila]

http://docs.hp.com/en/5991-1236/When_Good_Disks_Go_Bad.pdf

[Geoff Wild · Accepted solution]

Your vg01 should still be there in /dev

Do a ll /dev/vg01 > /tmp/vg01.listing

To see which disks were in it

Do a strings /etc/lvmtab

Look for vg01 - the disks devs should be there...

Compare to the one that was replaced...

Then do a vgcfgrestore

vgcfgrestore -n /dev/vg01 /dev/rdsk/cXtXdX

Then you should be able to do a vgchange -a y /dev/vg01

Rgds...Geoff

[Patrick Wallek]

To find which disks are in each VG do a:

# strings /etc/lvmtab

(Don't worry about the strange characters that appear. Those are normal).

Another good resource for replacing disks is "Chapter 16 - LVM" of the Software Recovery Handbook.

Software Recovery Handbook:
http://www2.itrc.hp.com/service/iv/node.do?node=prodITRC%2FWW_Start%2FN1%7C16

Chapter 16 - LVM:
http://www2.itrc.hp.com/service/iv/docDisplay.do?docId=prodITRC/DE_SW_UX_swrec_EN_01_E/LVM.pdf

[Coolmar]

Thanks!! That strings command is just what I needed :0)

Now, I doubt he even knows which disk was replaced other than pointing to it physically. I am in another city, so I have to figure it out somehow. Can I just do a vgcfgrestore to all 4 disks in that vg01 rather than trying to figure out which disk it was?

[Geoff Wild]

Sure you can vgcfgrestore to all of the disks if you like.

Rgds...Geoff

[Coolmar]

thanks Geoff!
Well the vgcfgresore worked on all disks, then the vgchange seemed to work. When I try and mount the vg01 stuff (mount -a) I get:

vxfs mount: /dev/vg01/lvol4 is corrupted. needs checking
vxfs mount: /dev/vg01/lvol3 is corrupted. needs checking

[Geoff Wild]

Try to fsck those lvols

fsck -F vxfs -o full -Y /dev/vg01/rlvol3

fsck -F vxfs -o full -Y /dev/vg01/rlvol4

Rgds...Geoff

[Coolmar]

log replay in progress
pass0 - checking structural files
pass1 - checking inode sanity and blocks
pass2 - checking directory linkage
warning - no lost+found directory for fileset 999
pass3 - checking reference counts
pass4 - checking resource maps
OK to clear log? (ynq)y
set state to CLEAN? (ynq)y

Can I just create that lost+found directory? or is that not even the problem?

[Geoff Wild]

You can just mkdir lost+found - but might as well just do your restore.

Can you mount the file systems now?

Rgds...Geoff

[Coolmar]

no...only one mounted...and I can't do an ll on it or anything. It just hangs. Now I am doing an fsck on another lvol and it seems to be hung. I can't even control break out of it or kill -9 the process.

[Coolmar]

I have these two processes and I can't kill them:

ps -ef |grep fsck
root 14639 1 0 09:17:14 pts/ta 0:00 fsck -ofull -Y /dev/vg01/rlvol1
root 14711 1 0 09:20:46 ? 0:00 fsck -ofull -Y /dev/vg01/rlvol3

[Geoff Wild]

Ah man....

Well...I'm afraid we have a problem - with the fsck's hung...only way I know of to clear is a reboot....

I have vgcfgrestored disks before and never any problems (though they were running 11.11)...

Does vgdsiplay -v vg01 work?

If yes, can you
lvdisplay -v /dev/vg01/lvolX |more
on all the lvols?


Rgds...Geoff

[Coolmar]

When I go into SAM I get the following:

The Logical Volume Manager show this device file, /dev/dsk/c6t8d0, with hardware path 10/2/3/0.8.0, as elonging to vg01. SAM has determined that this hardware path isnot currently active because either the device is disconnected, or this is not the priary path to a multiple path physical volume. This results i incorrect disk utilization information in the Disk Devices and Volume group lists.

[Coolmar]

vgdisplay -v vg01
--- Volume groups ---
VG Name /dev/vg01
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 4
Open LV 4
Max PV 16
Cur PV 4
Act PV 4
Max PE per PV 17501
VGDA 8
PE Size (Mbytes) 4
Total PE 69996
Alloc PE 69996
Free PE 0
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg01/lvol1
LV Status available/syncd
LV Size (Mbytes) 69996
Current LE 17499
Allocated PE 17499
Used PV 1

LV Name /dev/vg01/lvol2
LV Status available/syncd
LV Size (Mbytes) 69996
Current LE 17499
Allocated PE 17499
Used PV 1

LV Name /dev/vg01/lvol3
LV Status available/syncd
LV Size (Mbytes) 69996
Current LE 17499
Allocated PE 17499
Used PV 1

LV Name /dev/vg01/lvol4
LV Status available/syncd
LV Size (Mbytes) 69996
Current LE 17499
Allocated PE 17499
Used PV 1


--- Physical volumes ---
PV Name /dev/dsk/c6t8d0
PV Status unavailable
Total PE 17499
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t10d0
PV Status unavailable
Total PE 17499
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t12d0
PV Status unavailable
Total PE 17499
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t14d0
PV Status unavailable
Total PE 17499
Free PE 0
Autoswitch On

[Geoff Wild]

Oh - your PV Status is unavailable for all disks...

How was the disk replaced? was it hot swapped? and does their JBOD support hot swapping?

You may be able to just run an ioscan:

ioscan -fnC disk

insf -e

Otherwise, reboot...if possible, you may want a full harware power cycle of both the server and jbod...

Rgds...Geoff

[Coolmar]

I now get the following with the ioscan:

disk 10 10/2/3/0.8.0 sdisk NO_HW DEVICE HP 73.4GST373307LC
/dev/dsk/c6t8d0 /dev/rdsk/c6t8d0
disk 11 10/2/3/0.10.0 sdisk CLAIMED DEVICE HP 73.4GST373405LC
/dev/dsk/c6t10d0 /dev/rdsk/c6t10d0
disk 12 10/2/3/0.12.0 sdisk NO_HW DEVICE HP 73.4GST373307LC
/dev/dsk/c6t12d0 /dev/rdsk/c6t12d0
disk 13 10/2/3/0.14.0 sdisk NO_HW DEVICE HP 73.4GST373307LC
/dev/dsk/c6t14d0 /dev/rdsk/c6t14d0
disk 14 10/4/4/0.0.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c7t0d0 /dev/rdsk/c7t0d0


So will a reboot make the device "CLAIMED" rather than "NO_HW"?

[Geoff Wild]

Hopefully - but you may need to reboot the jbod as well...

Rgds...Geoff

[Coolmar]

thanks Geoff...I will try and get hold of the guy to do that. thanks again for all your help!