Failed mirror in VM Host storage allocated to VM Guests

[N Ward]

Hi, we have had a SAS disk failure on a VM Host. The disk is the Primary mirror for a VMs volume group vg_vmos which is the allocated vg00 for each of the VMs.

I have removed the mirror from LVM via lvreduce and vgreduce and the new disk is in its place. I am unable to do the sasmgr replace since its says 'device busy', but nothing is touching the old device, its showing NO_HW.

Also this failed disk is one of 4 in the vmos group that is assigned to each VM Guest to make up vg00. I'm unsure how to proceed with the rebuild. I'm guessing since its a SAS disk I can't just get the target replaced via sasmgr and then mirror vg_vmos again?

[Torsten.]

Consider to use "lsof" to find out why it is busy. Maybe a hung ioscan or something?

[N Ward]

I don't have lsof, but have tried fuser on the dsf, nothing coming back. The error says 'unable to replace target: device busy' so could this be the new H/W its talking about.

At the moment, the new disk just has its H/W path and no DSF since I have not run 'insf' yet.

Once I have this sorted, Can I just remirror and leave it there? I'm just concerned that this disk is part of the allocation to several VMs vg00s and might need other work, e.g. boot stuff for the VM Guest?

[Torsten.]

Did you assigns LVOLs to the guests?

So the guests don't access the disk at disk level.

I have seen this once (without HPVM), a process like ioscan, diskinfo or whatever was still hung, keep the old device "busy".

[N Ward]

The new disk will be one of four that is the mirror for a Volume Group that has a bunch of LVs for the VM Guests. Each LV is the vg00 for a particular VM Guest.

I will have another look for a hanging ioscan or diskinfo..

[N Ward]

Could not find a hanging isocan or diskinfo. There is a process disk_em running which monitors disks and is part of resmon. But fuser is not saying anything is touching the disk so I'm guessing this isn't.

[N Ward]

I think I will have to log a call, I cannot find information to deal with the scenario of disk replacement of VGs on VM Hosts with vg00 LVs assigned to VM Guests. I can also bring up the other problem of the target replace failure.

[Torsten.]

You could try to stop diagnostics, kill any leftover process (e.g. disk_em); try again and finally start diags.

[N Ward]

Tried that, no joy, call it is.. Thanks for trying.

[Chandrahasa s]

HI,
I understand following things from your query.


You have assigned one sas disk (full) to one vm guest

This disk is holding mirror copy of guest vg00
and now this disk is failed(conformed).

Keeping above details in consideration>>>

Now you need to replace this faulty disk with new one

you first do lvreduce in guest

then vgreduce -f vgname

from guest

using hpvmmodify remove disk

add new disk to vmhost

assign new disk to vmguest and remirror it


Chandra

[N Ward]

The disk is part of a mirrored volume group on the VM Host. There are 4 disks in the mirror. There are 4 LVs in the VG and each is assigned out to a guest as a single lun and is the guests vg00.

[Torsten.]

I understand that the command like this one just fails on the host with device busy:

# sasmgr replace_tgt -D /dev/sasd0 -q old_dev=/dev/dsk/c1t4d0 -q new_tgt_hwpath=0/4/1/0.0.0.11.0

[N Ward]

Yes the sasmgr command that needs to be run first fails too with device busy.

[Torsten.]

So I would just forget the old device file, delete it from configuration and add the mirror to the new device file.

[N Ward]

HP have stated its a known problem and I need a later SerialSCSI to fix it. The only way is to reboot for now.

What I'm concerned about with the mirroring is the LVs within the VG on the VM Host are vg00s on the guests. Is a straightforward mirror good enough for the this or is something else needed considering potentially the guest will have to boot from the mirror in a failure scenario.

[Torsten.]

As long as the LVOLS on the host are there, the guest will be happy.

[N Ward]

ok, so from a VM point of view, no associated boot disk tasks are needed for the mirror?

[Torsten.]

Exactly.

The presented LVOL (from host view) remains the same, mirrored or not.

[N Ward]

ok thanks for that...

[SoorajCleris]

Hi N Ward,

As you have already removed the OLD device from VG and LV configuration (lvreduce and vgreduce), now you can use new device file to mirror the LVs.

Just extend the VG with new device file, and mirror the LVs again.

The VG/the VM Guests will not affect with this. They will runfine as they are mapped to the LVs which are still available and running.( As Torsten already mentioned.)

All you can get is avoiding a reboot.

Regards,
Sooraj

[Torsten.]

This looks like your only choice so far, because the device remains "busy" until a reboot, but after a reboot you cannot do the "replace" command...

[N Ward]

This disk is one of 8 (ie 4 mirrored) each disk is mirrored strictly to its corresponding partner. e.g.

c2t3d0 is mirrored to c1t3d0
c2t4d0 is mirrored to c1t4d0

etc, etc. If I use the new disk instead of rebooting, it would change that, however its a consideration to avoid a reboot.

The disk is one of 4 and all 4 disks are in each LV.

e.g.
LV01 has all 4 disks contained within
LV02 has all 4 disks contained within
LV03 has all 4 disks contained within

There are two PVGs vmos_primary and vmos_mirror

The failed disk is from vmos_primary PVG in Volume group vg_vmos.

I'm unsure how to get the new disk into vmos_primary PVG and then exactly mirrored to its device counterpart (currently being used by the VG) in vmos_mirror PVG.

Once I have done this, I presume I can just use lvextend and increase mirror copies to 1 again for each LV presumably specifying all 4 of the mirror disks in the command