topic Re: Mirrored disk replacement - refresher in Operating System - HP-UX

Mirrored disk replacement - refresher

rvent — Thu, 16 Dec 2010 16:49:19 GMT

Hello,

I havent done this in a while so i am a bit rusty. One of my HDDs crashed and it was replaced. Now this disk is in a HP DS2120 and mirrors to the secondary on a different HP DS2120.

The new disk is /dev/rdsk/c4t8d0 and the disk that i need to mirror data from is /dev/rdsk/c5t8d0.

Going through my notes i have a section named: "replacing Mirrored Disk" and says to do the following:
1.- Remove old disk and insert new disk
2.- Run "vgcfgrestore" to restore LVM config info to the new disk
3.- Run "vgchange -a y vg01" to reactivate the volume to which the disk belongs. No automatic sync occurs since the volume group is currently active.
4.- Run "vgsync /dev/vg01" to manually sync all the extents in the volume.

Now this vg01 is a clustered volume group (Service Guard) and it is where all our ERP data resides.

Reading the man page for vgcfgrestore it says that the restore cannot be done when the volume is activate in shared mode. So i assume that before i can do the restore to /dev/dsk/c4t8d0 i need to deactivate vg01 once done i reactive vg01 and run vgsync, correct??

Now, will the data be offline while the sync occurs?

Also, on the man page says:
vgcfgrestore cannot be performed on devices attached to activated
volume groups. Prior to restoring a backup configuration to a disk,
detach the PV from the volume group using the pvchange(1M) command, or
deactivate the volume group using the vgchange(1M) command.

So since it is a new disk and it hasnt been attached to any VG, could i simply run vgcfgrestore without deactivating the entire VG? Once done i would simply add the disk to the VG and finally run vgsync, correct??

I am just really rusty and havent done this in a while.

Please advise.

Thanks

Re: Mirrored disk replacement - refresher

rajesh_32 — Thu, 16 Dec 2010 17:17:10 GMT

Hello rvent,
Refer http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c01911837/c01911837.pdf
Page 29

else go with lvreduce,vgreduce,pvcreate,vgextend and finally lvextend

Rgds

Re: Mirrored disk replacement - refresher

rvent — Thu, 16 Dec 2010 17:50:39 GMT

Well, i knew i had forgotten something...

I didnt halt LVM access to the disk prior to removing it. Node 1 cannot see the new disk and Node 2 can see the new disk:

Node 1:
Class I H/W Path Driver S/W State H/W Type Description
=========================================================================
disk 0 0/0/2/0.0.0.0 sdisk CLAIMED DEVICE TEAC DV-28E-N
/dev/dsk/c0t0d0 /dev/rdsk/c0t0d0
disk 1 0/1/1/0.0.0 sdisk CLAIMED DEVICE HP 73.4GST373454LC
/dev/dsk/c2t0d0 /dev/rdsk/c2t0d0
disk 3 0/1/1/1.2.0 sdisk CLAIMED DEVICE HP 73.4GST373454LC
/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0
disk 5 0/3/1/1.8.0 sdisk CLAIMED DEVICE HP 73.4GMAT3073NC
/dev/dsk/c5t8d0 /dev/rdsk/c5t8d0

Node 2:
Class I H/W Path Driver S/W State H/W Type Description
=========================================================================
disk 0 0/0/2/0.0.0.0 sdisk CLAIMED DEVICE TEAC DV-28E-N
/dev/dsk/c0t0d0 /dev/rdsk/c0t0d0
disk 1 0/1/1/0.0.0 sdisk CLAIMED DEVICE HP 73.4GST373454LC
/dev/dsk/c2t0d0 /dev/rdsk/c2t0d0
disk 5 0/1/1/1.2.0 sdisk CLAIMED DEVICE HP 73.4GST373454LC
/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0
disk 4 0/3/1/0.8.0 sdisk CLAIMED DEVICE HP 73.4GST373207LC
/dev/dsk/c4t8d0 /dev/rdsk/c4t8d0
disk 3 0/3/1/1.8.0 sdisk CLAIMED DEVICE HP 73.4GMAT3073NC
/dev/dsk/c5t8d0 /dev/rdsk/c5t8d0

The VG is not shared, so it is only active on Node 1.

What can i do for Node 1 to see the disk?

Thanks

Re: Mirrored disk replacement - refresher

rajesh_32 — Thu, 16 Dec 2010 18:00:20 GMT

Try
ioscan
insf -eCdisk in First Node

Re: Mirrored disk replacement - refresher

rvent — Thu, 16 Dec 2010 18:41:20 GMT

Nope, it didnt work:

# ioscan -fnC disk
Class I H/W Path Driver S/W State H/W Type Description
=========================================================================
disk 0 0/0/2/0.0.0.0 sdisk CLAIMED DEVICE TEAC DV-28E-N
/dev/dsk/c0t0d0 /dev/rdsk/c0t0d0
disk 1 0/1/1/0.0.0 sdisk CLAIMED DEVICE HP 73.4GST373454LC
/dev/dsk/c2t0d0 /dev/rdsk/c2t0d0
disk 3 0/1/1/1.2.0 sdisk CLAIMED DEVICE HP 73.4GST373454LC
/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0
disk 5 0/3/1/1.8.0 sdisk CLAIMED DEVICE HP 73.4GMAT3073NC
/dev/dsk/c5t8d0 /dev/rdsk/c5t8d0
[root@unix-1]/roi.backup/roi.12132010
# insf -veC disk
insf: Installing special files for sdisk instance 0 address 0/0/2/0.0.0.0
insf: Installing special files for sdisk instance 1 address 0/1/1/0.0.0
insf: Installing special files for sdisk instance 3 address 0/1/1/1.2.0
insf: Installing special files for sdisk instance 5 address 0/3/1/1.8.0
[root@unix-1]/roi.backup/roi.12132010
# ioscan -fnC disk
Class I H/W Path Driver S/W State H/W Type Description
=========================================================================
disk 0 0/0/2/0.0.0.0 sdisk CLAIMED DEVICE TEAC DV-28E-N
/dev/dsk/c0t0d0 /dev/rdsk/c0t0d0
disk 1 0/1/1/0.0.0 sdisk CLAIMED DEVICE HP 73.4GST373454LC
/dev/dsk/c2t0d0 /dev/rdsk/c2t0d0
disk 3 0/1/1/1.2.0 sdisk CLAIMED DEVICE HP 73.4GST373454LC
/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0
disk 5 0/3/1/1.8.0 sdisk CLAIMED DEVICE HP 73.4GMAT3073NC
/dev/dsk/c5t8d0 /dev/rdsk/c5t8d0

Everytime i do vgdisplay it complaints about the disk, so i know i have a ghost disk. Could this be the reason or should i just ensure that ioscan shows the disk before fixing the ghost disk??

Thanks

Re: Mirrored disk replacement - refresher

rvent — Thu, 16 Dec 2010 21:34:54 GMT

Ok so i got stock on this part. After removing the mirror on all lv's i am trying to remove the disk from the VG. This is the message:

# vgreduce -f /dev/vg01
vgreduce: Couldn't query physical volume "/dev/dsk/c4t8d0":
The specified path does not correspond to physical volume attached to
this volume group
Not all extents are free. i.e. Out of 17499 PEs, only 17474 are free.
You must free all PEs using lvreduce/lvremove before the PV can be removed.
Example: lvreduce -A n -m 0 /dev/vg01/lvol1.
lvremove -A n /dev/vg01/lvol1
Here's the map of used PEs

--- Logical extents ---
LE LV PE Status 1
0000 lvol5 8525 ???
0001 lvol5 8526 ???
0002 lvol5 8527 ???

I tried to:
# lvdisplay -v -k /dev/vg01/lvol5 | pg
lvdisplay: "/dev/vg01/lvol5": No such file or directory

# lvreduce -m 0 -A n -k /dev/vg01/lvol5 0
lvreduce: "/dev/vg01/lvol5": No such file or directory

vgdiplay -v vg01 only shows 4 LVs...

# mount | grep vg01
/roi on /dev/vg01/lvol1 log,nodatainlog on Tue Dec 14 12:36:42 2010
/usr/ud60 on /dev/vg01/lvol2 log,nodatainlog on Tue Dec 14 12:36:42 2010
/usr/redback on /dev/vg01/lvol3 log,nodatainlog on Tue Dec 14 12:36:42 2010
/roi/HST.TRAIN on /dev/vg01/lvol4 log,nodatainlog on Tue Dec 14 12:36:43 2010

Where is this LV coming from??

Any ideas? Force remove??

Thanks

Re: Mirrored disk replacement - refresher

rvent — Fri, 17 Dec 2010 01:00:07 GMT

Ok,

So i got the disk to show up on node 1, but every time i try to do vgcfgrestore -n /dev/vg01 /dev/dsk/c4t8d0 i get a message about not being able to query the physical volume. However, after that it takes a few seconds and then says: "Volume Group cfg restored to disk"

I then try to do pvchange -a y /dev/dsk/c4t8d0 and it complaints about "-a" being an illegal option...

Any thoughts..?

Thanks

Re: Mirrored disk replacement - refresher

R.O. — Fri, 17 Dec 2010 06:49:51 GMT

Is the disk already in the vg01? If so, see if the mirror of lvols has been restored. If not, try to reactivate the volume group (vgchange -a y /dev/vg01) and check whether the logical volumes are being synchronized. If they did not synchronized, do a "vgsync /dev/vg01"

Regards,

Re: Mirrored disk replacement - refresher

R.O. — Fri, 17 Dec 2010 06:53:25 GMT

The message about "pvchange -a y not available" is because you need LVM online replacement (OLR) functionality available. To achieve it you need these patches:

â ¢ For HP-UX 11i version 1, install patches PHKL_31216 and PHCO_30698 or their superseding patches.
â ¢ For HP-UX 11i version 2, install patches PHKL_32095 and PHCO_31709 or their superseding patches.

Regards,

Re: Mirrored disk replacement - refresher

rvent — Fri, 17 Dec 2010 22:39:04 GMT

Nope, vgdisplay -v vg01 only shows the old disk under ---Physical Volumes---

I checked the LVs and no mirrors are present.

The VG is still active and cant take it off line till Sunday. I am gonna try to deactivate it and reactivate it and see if i can get the drive added to the VG.

/etc/lvmtab shows both disks under the vg01 entry.

Re: Mirrored disk replacement - refresher

Terry.giblin — Sat, 18 Dec 2010 17:56:10 GMT

"Cant take it off line till Sunday."

How times have I heard that.

Do you mind answer a few quick questions?

What did the disks look like before the crash?

What do the disks look like now after the crash?

What does the system say the disks look like? - (Now and before)

What does the disk application say the disks look like? - (Now and before)

Without stating the obvious but can you remind me what versions of firmware, hardware, operating systems, applications, are we talking about exactly?

A quick pencile diagram of your two disk config, would be greatly appreciated. - (Now and before, as it is and as it should be.)

If you understand what I mean.

Firmware, Hardware, SAN, Operating system, Application or Database.

Its like looking for a needle, in a Gaint Hay stack.

Its fun when you find it.

Terry Giblin

Re: Mirrored disk replacement - refresher

rvent — Mon, 20 Dec 2010 15:04:27 GMT

Hello,

Ok, before the crash both disks were shown as physical drives for VG01:
# vgdisplay -v vg01
--- Volume groups ---
VG Name /dev/vg01
VG Write Access read/write
VG Status available, exclusive
Max LV 255
Cur LV 5
Open LV 4
Max PV 16
Cur PV 2
Act PV 1
Max PE per PV 17501
VGDA 2
PE Size (Mbytes) 4
Total PE 17499
Alloc PE 9650
Free PE 7849
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg01/lvol1
LV Status available/syncd

LV Name /dev/vg01/lvol2
LV Status available/syncd

LV Name /dev/vg01/lvol3
LV Status available/syncd

LV Name /dev/vg01/lvol4
LV Status available/syncd

--- Physical volumes ---
PV Name /dev/dsk/c4t8d0
PV Status available
Total PE 17499
Free PE 7849
Autoswitch On

PV Name /dev/dsk/c5t8d0
PV Status available
Total PE 17499
Free PE 7849
Autoswitch On

After the crash the c4t8d0 disk was still listed, however, i physically removed the disk before i stopped LVM from writing to it. After reading some HP docs, i found out it is called ghost disk and followed their instruction on how to remove it, however, when the docs say to add it to the VG it fails. The after looks like:
# vgdisplay -v vg01
--- Volume groups ---
VG Name /dev/vg01
VG Write Access read/write
VG Status available, exclusive
Max LV 255
Cur LV 5
Open LV 4
Max PV 16
Cur PV 2
Act PV 1
Max PE per PV 17501
VGDA 2
PE Size (Mbytes) 4
Total PE 17499
Alloc PE 9650
Free PE 7849
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg01/lvol1
LV Status available/syncd

LV Name /dev/vg01/lvol2
LV Status available/syncd

LV Name /dev/vg01/lvol3
LV Status available/syncd

LV Name /dev/vg01/lvol4
LV Status available/syncd

--- Physical volumes ---
PV Name /dev/dsk/c5t8d0
PV Status available
Total PE 17499
Free PE 7849
Autoswitch On

The before and after is the same:

disk 4 0/3/1/0.8.0 sdisk CLAIMED DEVICE HP 73.4GST373207LC
/dev/dsk/c4t8d0 /dev/rdsk/c4t8d0
disk 3 0/3/1/1.8.0 sdisk CLAIMED DEVICE HP 73.4GMAT3073NC
/dev/dsk/c5t8d0 /dev/rdsk/c5t8d0

At the moment, the system sees the disk just like before the crash, with the only difference being that the replacement disk is not in VG01 anymore.

# pvdisplay /dev/dsk/c4t8d0
pvdisplay: Couldn't find the volume group to which
physical volume "/dev/dsk/c4t8d0" belongs.
pvdisplay: Cannot display physical volume "/dev/dsk/c4t8d0".

Then when i try to do vgcfgrestore -n /dev/vg01 /dev/dsk/c4t8d0 i get a message about not being able to query the physical volume. However, after that it takes a few seconds and then says: "Volume Group cfg restored to disk"

# uname -a
HP-UX unix-2 B.11.11 U 9000/800 1807302088 unlimited-user license

VG01 is a mirrored VG running under Service Guard.

Thanks

Re: Mirrored disk replacement - refresher

KapilRaj — Mon, 20 Dec 2010 21:18:39 GMT

I think c4t8 and c5t8 are the same. i.e. c4 and c5 being two paths to the SAN storage.

Can you post lvdisplay output of one logical volume ?. I want to verify if it is really a mirror.

Regards,

Kaps

Re: Mirrored disk replacement - refresher

rvent — Mon, 20 Dec 2010 21:27:42 GMT

No, they arent the same disks (at least physically).

c4 is in DS2120 tray 1 and c5 is in DS2120 tray 2 they are both mirrored, they were mirrored.

# lvdisplay -v /dev/vg01/lvol1 | pg
--- Logical volumes ---
LV Name /dev/vg01/lvol1
VG Name /dev/vg01
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 25000
Current LE 6250
Allocated PE 6250
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation strict
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c5t8d0 6250 6250

--- Logical extents ---
LE PV1 PE1 Status 1
00000 /dev/dsk/c5t8d0 00000 current
00001 /dev/dsk/c5t8d0 00001 current
00002 /dev/dsk/c5t8d0 00002 current
00003 /dev/dsk/c5t8d0 00003 current

Re: Mirrored disk replacement - refresher

KapilRaj — Mon, 20 Dec 2010 21:27:47 GMT

Ignore my request. I see they are indeed different disks.

Post lvdisplay -v | head -30 if you can.

Re: Mirrored disk replacement - refresher

KapilRaj — Mon, 20 Dec 2010 21:30:22 GMT

I think you have gotten rid of the disk from the VG.

So vgcfgrestore if not going to fix.

pvcreate /dev/rdsk/c4t8d0 # Prepare for LVM
vgextend /dev/dsk/c4t8d0 # Add the disk back to the VG #

Now setup the mirrors

lvextend -m 1 /dev//lvol /dv/dsk/c4t8d0

do it for all the LVs.

Re: Mirrored disk replacement - refresher

KapilRaj — Mon, 20 Dec 2010 21:31:37 GMT

Let me rephrase "I think you have" with "You have indeed "

Look at this

"Mirror copies 0 "

Re: Mirrored disk replacement - refresher

rvent — Mon, 20 Dec 2010 21:32:46 GMT

Will this be an offline operation? Or should i be able to do it during business hours?

Thanks

Re: Mirrored disk replacement - refresher

KapilRaj — Mon, 20 Dec 2010 21:35:18 GMT

All he steps I mentioned are online.

Re: Mirrored disk replacement - refresher

KapilRaj — Mon, 20 Dec 2010 21:46:00 GMT

It might put a little bit of load in the system during the mirroring process. So the I/O operations may experience a delay. Do it when there is less production load if you care.