Mirrored disk replacement - refresher

[rvent]

Hello,

I havent done this in a while so i am a bit rusty. One of my HDDs crashed and it was replaced. Now this disk is in a HP DS2120 and mirrors to the secondary on a different HP DS2120.

The new disk is /dev/rdsk/c4t8d0 and the disk that i need to mirror data from is /dev/rdsk/c5t8d0.

Going through my notes i have a section named: "replacing Mirrored Disk" and says to do the following:
1.- Remove old disk and insert new disk
2.- Run "vgcfgrestore" to restore LVM config info to the new disk
3.- Run "vgchange -a y vg01" to reactivate the volume to which the disk belongs. No automatic sync occurs since the volume group is currently active.
4.- Run "vgsync /dev/vg01" to manually sync all the extents in the volume.

Now this vg01 is a clustered volume group (Service Guard) and it is where all our ERP data resides.

Reading the man page for vgcfgrestore it says that the restore cannot be done when the volume is activate in shared mode. So i assume that before i can do the restore to /dev/dsk/c4t8d0 i need to deactivate vg01 once done i reactive vg01 and run vgsync, correct??

Now, will the data be offline while the sync occurs?

Also, on the man page says:
vgcfgrestore cannot be performed on devices attached to activated
volume groups. Prior to restoring a backup configuration to a disk,
detach the PV from the volume group using the pvchange(1M) command, or
deactivate the volume group using the vgchange(1M) command.

So since it is a new disk and it hasnt been attached to any VG, could i simply run vgcfgrestore without deactivating the entire VG? Once done i would simply add the disk to the VG and finally run vgsync, correct??

I am just really rusty and havent done this in a while.

Please advise.

Thanks

[rajesh_32]

Hello rvent,
Refer http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c01911837/c01911837.pdf
Page 29

else go with lvreduce,vgreduce,pvcreate,vgextend and finally lvextend

Rgds

[rvent]

Well, i knew i had forgotten something...

I didnt halt LVM access to the disk prior to removing it. Node 1 cannot see the new disk and Node 2 can see the new disk:

Node 1:
Class I H/W Path Driver S/W State H/W Type Description
=========================================================================
disk 0 0/0/2/0.0.0.0 sdisk CLAIMED DEVICE TEAC DV-28E-N
/dev/dsk/c0t0d0 /dev/rdsk/c0t0d0
disk 1 0/1/1/0.0.0 sdisk CLAIMED DEVICE HP 73.4GST373454LC
/dev/dsk/c2t0d0 /dev/rdsk/c2t0d0
disk 3 0/1/1/1.2.0 sdisk CLAIMED DEVICE HP 73.4GST373454LC
/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0
disk 5 0/3/1/1.8.0 sdisk CLAIMED DEVICE HP 73.4GMAT3073NC
/dev/dsk/c5t8d0 /dev/rdsk/c5t8d0

Node 2:
Class I H/W Path Driver S/W State H/W Type Description
=========================================================================
disk 0 0/0/2/0.0.0.0 sdisk CLAIMED DEVICE TEAC DV-28E-N
/dev/dsk/c0t0d0 /dev/rdsk/c0t0d0
disk 1 0/1/1/0.0.0 sdisk CLAIMED DEVICE HP 73.4GST373454LC
/dev/dsk/c2t0d0 /dev/rdsk/c2t0d0
disk 5 0/1/1/1.2.0 sdisk CLAIMED DEVICE HP 73.4GST373454LC
/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0
disk 4 0/3/1/0.8.0 sdisk CLAIMED DEVICE HP 73.4GST373207LC
/dev/dsk/c4t8d0 /dev/rdsk/c4t8d0
disk 3 0/3/1/1.8.0 sdisk CLAIMED DEVICE HP 73.4GMAT3073NC
/dev/dsk/c5t8d0 /dev/rdsk/c5t8d0


The VG is not shared, so it is only active on Node 1.

What can i do for Node 1 to see the disk?

Thanks

[rajesh_32]

Try
ioscan
insf -eCdisk in First Node

[rvent]

Nope, it didnt work:

# ioscan -fnC disk
Class I H/W Path Driver S/W State H/W Type Description
=========================================================================
disk 0 0/0/2/0.0.0.0 sdisk CLAIMED DEVICE TEAC DV-28E-N
/dev/dsk/c0t0d0 /dev/rdsk/c0t0d0
disk 1 0/1/1/0.0.0 sdisk CLAIMED DEVICE HP 73.4GST373454LC
/dev/dsk/c2t0d0 /dev/rdsk/c2t0d0
disk 3 0/1/1/1.2.0 sdisk CLAIMED DEVICE HP 73.4GST373454LC
/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0
disk 5 0/3/1/1.8.0 sdisk CLAIMED DEVICE HP 73.4GMAT3073NC
/dev/dsk/c5t8d0 /dev/rdsk/c5t8d0
[root@unix-1]/roi.backup/roi.12132010
# insf -veC disk
insf: Installing special files for sdisk instance 0 address 0/0/2/0.0.0.0
insf: Installing special files for sdisk instance 1 address 0/1/1/0.0.0
insf: Installing special files for sdisk instance 3 address 0/1/1/1.2.0
insf: Installing special files for sdisk instance 5 address 0/3/1/1.8.0
[root@unix-1]/roi.backup/roi.12132010
# ioscan -fnC disk
Class I H/W Path Driver S/W State H/W Type Description
=========================================================================
disk 0 0/0/2/0.0.0.0 sdisk CLAIMED DEVICE TEAC DV-28E-N
/dev/dsk/c0t0d0 /dev/rdsk/c0t0d0
disk 1 0/1/1/0.0.0 sdisk CLAIMED DEVICE HP 73.4GST373454LC
/dev/dsk/c2t0d0 /dev/rdsk/c2t0d0
disk 3 0/1/1/1.2.0 sdisk CLAIMED DEVICE HP 73.4GST373454LC
/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0
disk 5 0/3/1/1.8.0 sdisk CLAIMED DEVICE HP 73.4GMAT3073NC
/dev/dsk/c5t8d0 /dev/rdsk/c5t8d0


Everytime i do vgdisplay it complaints about the disk, so i know i have a ghost disk. Could this be the reason or should i just ensure that ioscan shows the disk before fixing the ghost disk??

Thanks

[rvent]

Ok so i got stock on this part. After removing the mirror on all lv's i am trying to remove the disk from the VG. This is the message:

# vgreduce -f /dev/vg01
vgreduce: Couldn't query physical volume "/dev/dsk/c4t8d0":
The specified path does not correspond to physical volume attached to
this volume group
Not all extents are free. i.e. Out of 17499 PEs, only 17474 are free.
You must free all PEs using lvreduce/lvremove before the PV can be removed.
Example: lvreduce -A n -m 0 /dev/vg01/lvol1.
lvremove -A n /dev/vg01/lvol1
Here's the map of used PEs

--- Logical extents ---
LE LV PE Status 1
0000 lvol5 8525 ???
0001 lvol5 8526 ???
0002 lvol5 8527 ???


I tried to:
# lvdisplay -v -k /dev/vg01/lvol5 | pg
lvdisplay: "/dev/vg01/lvol5": No such file or directory

# lvreduce -m 0 -A n -k /dev/vg01/lvol5 0
lvreduce: "/dev/vg01/lvol5": No such file or directory

vgdiplay -v vg01 only shows 4 LVs...

# mount | grep vg01
/roi on /dev/vg01/lvol1 log,nodatainlog on Tue Dec 14 12:36:42 2010
/usr/ud60 on /dev/vg01/lvol2 log,nodatainlog on Tue Dec 14 12:36:42 2010
/usr/redback on /dev/vg01/lvol3 log,nodatainlog on Tue Dec 14 12:36:42 2010
/roi/HST.TRAIN on /dev/vg01/lvol4 log,nodatainlog on Tue Dec 14 12:36:43 2010

Where is this LV coming from??

Any ideas? Force remove??

Thanks

[rvent]

Ok,

So i got the disk to show up on node 1, but every time i try to do vgcfgrestore -n /dev/vg01 /dev/dsk/c4t8d0 i get a message about not being able to query the physical volume. However, after that it takes a few seconds and then says: "Volume Group cfg restored to disk"

I then try to do pvchange -a y /dev/dsk/c4t8d0 and it complaints about "-a" being an illegal option...

Any thoughts..?

Thanks

[R.O.]

Is the disk already in the vg01? If so, see if the mirror of lvols has been restored. If not, try to reactivate the volume group (vgchange -a y /dev/vg01) and check whether the logical volumes are being synchronized. If they did not synchronized, do a "vgsync /dev/vg01"

Regards,

[R.O.]

The message about "pvchange -a y not available" is because you need LVM online replacement (OLR) functionality available. To achieve it you need these patches:

â ¢ For HP-UX 11i version 1, install patches PHKL_31216 and PHCO_30698 or their superseding patches.
â ¢ For HP-UX 11i version 2, install patches PHKL_32095 and PHCO_31709 or their superseding patches.

Regards,

[rvent]

Nope, vgdisplay -v vg01 only shows the old disk under ---Physical Volumes---

I checked the LVs and no mirrors are present.

The VG is still active and cant take it off line till Sunday. I am gonna try to deactivate it and reactivate it and see if i can get the drive added to the VG.

/etc/lvmtab shows both disks under the vg01 entry.

[Terry.giblin]

"Cant take it off line till Sunday."

How times have I heard that.

Do you mind answer a few quick questions?

What did the disks look like before the crash?

What do the disks look like now after the crash?

What does the system say the disks look like? - (Now and before)

What does the disk application say the disks look like? - (Now and before)

Without stating the obvious but can you remind me what versions of firmware, hardware, operating systems, applications, are we talking about exactly?

A quick pencile diagram of your two disk config, would be greatly appreciated. - (Now and before, as it is and as it should be.)

If you understand what I mean.

Firmware, Hardware, SAN, Operating system, Application or Database.

Its like looking for a needle, in a Gaint Hay stack.

Its fun when you find it.

Terry Giblin

[rvent]

Hello,

Ok, before the crash both disks were shown as physical drives for VG01:
# vgdisplay -v vg01
--- Volume groups ---
VG Name /dev/vg01
VG Write Access read/write
VG Status available, exclusive
Max LV 255
Cur LV 5
Open LV 4
Max PV 16
Cur PV 2
Act PV 1
Max PE per PV 17501
VGDA 2
PE Size (Mbytes) 4
Total PE 17499
Alloc PE 9650
Free PE 7849
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg01/lvol1
LV Status available/syncd

LV Name /dev/vg01/lvol2
LV Status available/syncd

LV Name /dev/vg01/lvol3
LV Status available/syncd


LV Name /dev/vg01/lvol4
LV Status available/syncd

--- Physical volumes ---
PV Name /dev/dsk/c4t8d0
PV Status available
Total PE 17499
Free PE 7849
Autoswitch On

PV Name /dev/dsk/c5t8d0
PV Status available
Total PE 17499
Free PE 7849
Autoswitch On

After the crash the c4t8d0 disk was still listed, however, i physically removed the disk before i stopped LVM from writing to it. After reading some HP docs, i found out it is called ghost disk and followed their instruction on how to remove it, however, when the docs say to add it to the VG it fails. The after looks like:
# vgdisplay -v vg01
--- Volume groups ---
VG Name /dev/vg01
VG Write Access read/write
VG Status available, exclusive
Max LV 255
Cur LV 5
Open LV 4
Max PV 16
Cur PV 2
Act PV 1
Max PE per PV 17501
VGDA 2
PE Size (Mbytes) 4
Total PE 17499
Alloc PE 9650
Free PE 7849
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg01/lvol1
LV Status available/syncd

LV Name /dev/vg01/lvol2
LV Status available/syncd

LV Name /dev/vg01/lvol3
LV Status available/syncd


LV Name /dev/vg01/lvol4
LV Status available/syncd

--- Physical volumes ---
PV Name /dev/dsk/c5t8d0
PV Status available
Total PE 17499
Free PE 7849
Autoswitch On

The before and after is the same:

disk 4 0/3/1/0.8.0 sdisk CLAIMED DEVICE HP 73.4GST373207LC
/dev/dsk/c4t8d0 /dev/rdsk/c4t8d0
disk 3 0/3/1/1.8.0 sdisk CLAIMED DEVICE HP 73.4GMAT3073NC
/dev/dsk/c5t8d0 /dev/rdsk/c5t8d0


At the moment, the system sees the disk just like before the crash, with the only difference being that the replacement disk is not in VG01 anymore.

# pvdisplay /dev/dsk/c4t8d0
pvdisplay: Couldn't find the volume group to which
physical volume "/dev/dsk/c4t8d0" belongs.
pvdisplay: Cannot display physical volume "/dev/dsk/c4t8d0".

Then when i try to do vgcfgrestore -n /dev/vg01 /dev/dsk/c4t8d0 i get a message about not being able to query the physical volume. However, after that it takes a few seconds and then says: "Volume Group cfg restored to disk"

# uname -a
HP-UX unix-2 B.11.11 U 9000/800 1807302088 unlimited-user license

VG01 is a mirrored VG running under Service Guard.

Thanks

[KapilRaj]

I think c4t8 and c5t8 are the same. i.e. c4 and c5 being two paths to the SAN storage.

Can you post lvdisplay output of one logical volume ?. I want to verify if it is really a mirror.

Regards,

Kaps

[rvent]

No, they arent the same disks (at least physically).

c4 is in DS2120 tray 1 and c5 is in DS2120 tray 2 they are both mirrored, they were mirrored.


# lvdisplay -v /dev/vg01/lvol1 | pg
--- Logical volumes ---
LV Name /dev/vg01/lvol1
VG Name /dev/vg01
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 25000
Current LE 6250
Allocated PE 6250
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation strict
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c5t8d0 6250 6250

--- Logical extents ---
LE PV1 PE1 Status 1
00000 /dev/dsk/c5t8d0 00000 current
00001 /dev/dsk/c5t8d0 00001 current
00002 /dev/dsk/c5t8d0 00002 current
00003 /dev/dsk/c5t8d0 00003 current

[KapilRaj]

Ignore my request. I see they are indeed different disks.

Post lvdisplay -v | head -30 if you can.

[KapilRaj · Accepted solution]

I think you have gotten rid of the disk from the VG.

So vgcfgrestore if not going to fix.

pvcreate /dev/rdsk/c4t8d0 # Prepare for LVM
vgextend /dev/dsk/c4t8d0 # Add the disk back to the VG #

Now setup the mirrors

lvextend -m 1 /dev//lvol /dv/dsk/c4t8d0

do it for all the LVs.

[KapilRaj]

Let me rephrase "I think you have" with "You have indeed "

Look at this

"Mirror copies 0 "

[rvent]

Will this be an offline operation? Or should i be able to do it during business hours?

Thanks

[KapilRaj]

All he steps I mentioned are online.

[KapilRaj]

It might put a little bit of load in the system during the mirroring process. So the I/O operations may experience a delay. Do it when there is less production load if you care.

[Terry.giblin]

Many things have now happened and you said you have tried many options to fix the problem.

The problem is and from what you have described the you, the system and now everyone else is a little bit confused on the problem.

So can we start from first principles;

You said you tried to fix a ghost disk??

Can you tell me what that means and what exactly you did?

But to be honest, it sounds to me that the kernel has got out of step with LVM.

I would take one step back before doing anything else and refresh your special disk files.

strings /etc/lvmtab
lssf /dev/rdsk/c4t8d0
rmsf -H 0/3/1/0.8.0
rm /dev/dsk/c4t8d0
rm /dev/rdsk/c4t8d0
insf -e


Now we know the special device files are what they should be.

If you do the above commands at least we will have deleted all reference to the old disk and its special files and created new ones with the new disk info.

Now you should try to proceed as normal adding or deleting disks to the volume group.

Under extreme caution.

If this does not fix the problem, the next time the system is 'offline' and you can do some major maintance.

cp /etc/lvmtab to /etc/lvmtab.old
rm /etc/lvmtab

vgscan -av

It works for me everytime.

Terry Giblin

[rvent]

Hello,

Thanks for all the great help. I have one more thing:

will removing the lvmtab and doing vgscan -av fix the issue i am now having:


# lvextend -m 1 /dev/vg01/lvol3 /dev/dsk/c4t8d0
The newly allocated mirrors are now being synchronized. This operation will
take some time. Please wait ....
Logical volume "/dev/vg01/lvol3" has been successfully extended.
vgcfgbackup: /etc/lvmtab is out of date with the running kernel:Kernel indicates 3 disks for "/dev/vg01"; /etc/lvmtab has 2 disks.
Cannot proceed with backup.


Thanks

[Terry.giblin]

Dear rvent,

As I said,

"But to be honest, it sounds to me that the kernel has got out of step with LVM. "

"Under extreme caution.

If this does not fix the problem, the next time the system is 'offline' and you can do some major maintance.

cp /etc/lvmtab to /etc/lvmtab.old
rm /etc/lvmtab

vgscan -av

It works for me everytime.

Terry giblin"

Now lvm has been refreshed the only way I know how to refresh the kernel is with a reboot.

[Terry.giblin]

Dear rvent,

You ran

"vgscan -av

or

vgscan -v"?

"vgcfgbackup: /etc/lvmtab is out of date with the running kernel:Kernel indicates 3"

That's why everyone is confused?

"Some things you just have to do by looking."

Can you do me a big favour, can you go check how many cables are plugged in (connected) to each of your disk array's, please?

And add one more, for good luck.

Then run,

vgscan -v.

Problem solved for good and everyone is happy.

Terry Giblin

[Terry.giblin]

"Problem solved for good and everyone is happy."

Until it happens again, no one ever knowns.

Also long as we have tried and did our best.

Terry Giblin