EMC VG mystery

[Alan Riggs]

Description:
Single Hyper VG on an EMC 8430 (one of 3).
Powerpath secondary connection exists, but is not extended into the VG as a shadow link.
Power failure on server.
After server comes up, this VG fails to activate.
vgchange -a y fails with "Quorum not present of some PV(s) are missing".
vgexport succeeds.
diskinfo on /dev/rdsk/... succeeds.
dd if=/dev/dsk/... succeeds and appears to show LVM header information.
vgcfgrestore vgsan3 /dev/dsk/... succeeds.
vgimport succeeds.
strings /etc/lvmtab shows the proper disk and VG name.
vgchange -a y fails with same error message.
If I vgexport again and vgimport using *both* primary and secondary paths to the disk, vgimport warns that the VG is suposed to have only one volume, which supports the idea that the LVM information is correct.
But I still cannot activate the VG.

[Sridhar Bhaskarla]

Hi Alan,

Not exactly the scenario but the similar situation happened to me when we did a business copy on XP256. We suspended the pairs and then try to import from the mirror copy. vgimport works but not vgchange. We then had to re-sync, break the mirror and it worked. So, I feel there could be a problem with the LUN itself. If you already have a mirror for this LUN, you may want to resync back and see if it works.

-Sri

[Alan Riggs]

Thanks. Unfortunately this volume does not have any BCV associated with it, so the restore backwards option is not available.

[Animesh Chakraborty]

hi,
Have you tried this one ?
vgchange -q n /dev/vgxx



Regards
Animesh

[Alan Riggs]

Yes. I have tried activating the VG both with and without the "-q n" flag.

[Animesh Chakraborty]

Hi again,
try
#insf -e


best of luck
Animesh

[Alan Riggs]

Ah, left that out of the original problem description. I removed the block and raw device files and recreated them with insf. No change.

[George Petrides]

Do you have only one physical volume in the group (excluding pvlink)? From your description, I suggest HP LVM patches and call EMC, if your EMC rocket scientist made changes to the bin file and is showing you another disk then that might be a problem.

[Carlos Fernandez Riera]

Try vgimport -s.

See: http://forums.itrc.hp.com/cm/QuestionAnswer/1,11866,0x9df8c6af36b7d5118ff10090279cd0f9,00.html

[Alan Riggs]

No changes to the BIN file.
I cannot use the -s flag for vgimport since the VG mapfile was not created with the -s flag, hence I do not know the VGID.

If someone can tell me how to capture the VGID from the disk or from the lvmconf I will try the vgimport -s.

[Alan Riggs]

Forgot to add:
Yes, there is only one disk (with a secondary powerpath link) in the VG.

[James R. Ferguson]

Hi Alan:

As far as looking at the VGID, you can get it this way:

# echo 0x2010?2X|adb /dev/dsk/cXtYdZ|expand|tr -d " "|sed "s/2010:/VGID /"

You will see something like:

VGID 11823E9D33FB2ABF

Regards!

...JRF...

[Carlos Fernandez Riera]

To see VGID:

Dear friend:

vgexport -s -m /tmp/vgap -p /dev/vgxx

or:

VGID is recorded in each disk:

dd if=/dev/rdsk/cxtxtx bs=1024 skip=8 count=9 | dd bs=8 skip=2 count=1 | od -x | head -1 .

Good Luck.

[Alan Riggs]

Thanks for the methods for capturing the VGID from disk. Unfortunately, the vgchange -a y still fails.

More information: I have also tried moving the lvmtab out of the way and recreating it with vgscan. That did not solve this problem and created another problem with a powerpath secondary, so I moved back to the old lvmtab.

[Sridhar Bhaskarla]

Alan,

As you were able to import the volume group the VG information is looking quite right. This should be a problem with the LUN that the EMC is not releasing it. As I mentioned earlier, it didn't work for me as long as the LUN was in suspended sync mode on XP though I could vgimport it.

Being a layman on EMC, I can't say anything on it. But I am afraid, you may have to do something on the EMC itself to get rid of the problem instead trying on the system.

Only my "0" cents.

-Sri

[Carlos Fernandez Riera]

Hi again:

It seems to be like a problem i had a month ago.
Working with a crashed L, vg00 (mirrored) was unrecuperable and we cannot boot after a disk crashed.

We insert a disk from other L, not mirrored to be able to boot and recover crashed vg00.

To be saved, we did a dd from not crased disk to a new disk, and also to replaced crased disk; this is we had 3 images of same disk. Then we unpluged one of these three disk. So we had a boot disk not mirrored, and a duplicated disk - i mean that VGRA was the same on both two disk-.

When we vgimport that vg for reparation ( we needed to run fsck and others), LVM said vg has ONE or THREE disks ( i dont remember exactly. but it was not true, we had only two for that vg):

More or less we had:
disk c1t2d0
disk c1t0d0

LVM-> disk c1t2d0 alternate link c1t0d0 ( remember that on a Lclass these are two different disks, but VGRA were images).

We reduce c1t0d0 because it can not be an alternate link, then LVM said not quorum.

This situation and yours seem to be equal. We had VGRA duplicated as well as you have VGRA duplicated on BCV.

I guess you must sync BCV volumes, to be only one.

After lots of trys, we recover /etc/lvmtab from a backup tape, and ... BINGO.


HTH

[Carlos Fernandez Riera]

Sorry. I remember BCV but it was from Sridhar?s.

Strings /etc/lvmtab must show disks involved.

run pvdisplay each disk.

And try recover /etc/lvmtab from a tape backup. On reboot a vgscan or similar runs and must overwrite the correct lvmtab. vgscan will overwrite /etc/lmvtab too.

Try recover /etc/lvmtab from a tape backup.

[Alan Riggs]

Sridhar,
Unfortunately, the EMC volume is not part of a disk group or associated with a BCV. There isn't any synch/suspend operation for me to clear. It is part of an M1/M2 pair, and I will ask EMC if there is any way to get a view into that relationship.

Carlos,
Strings /etc/lvmtab shows the correct information both when I have imported and exported the disk. I have recreated the file with vgscan and resored an earlier version of the file, both without effect. The warning I get when importing the VG using both paths to the disk seems to clearly indicate that the kernel is "expecting" just a single disk for this VG. I think Sridhar is on the right track that the EMC is not allowing full access to the drive for some reason, but I cannot figure out why (and neither can anyone that I have spoken to at EMC).

[Amado Pichel]

Alan,

Is it possible that this disk is also
presented to other servers also connected to your EMC ??

If so, Maybe it was added to another
VG on another system. If this disk was part of another VG somewhere else The VGDA would contain a VGID and you could still successfully import it but not activate it.

Now, The vgcfgrestore should fix this problem
however that would depend on whether or not the restore file for /dev/lvmconf/vgsan3.conf is valid. Is there another backup file
for vgsan3 like vgsan3.conf.old ??
If so compare the date/time stamps and try the older one.

[Dennis Raudenbush]

Alan, seems at this point it wouldn't hurt to vgextend the alternate disk device in the volume group and retry to activate. If it fails you could then vgreduce the primary to switch to alternate - then vgextend the primary as the alternate and retry.

johnc

[Alan Riggs]

TracFone:
This volume is not published to any other systems. I can also be certain that the VGID is correct since I can query it directly using adb or dd (as supplied above) and import it with the -s flag. I have tried restoring an older copy of the vgcfgbackup. It did not change the results.

Rich:
I have tried activatign the VG with both PVs and with each PV singly. The error remains constant.

[Roger Baptiste]

Alan,

Can you located the devices
of this VG in the commands:
syminq
symdev
and
sympd

If so, go into another
system which is connected
to the EMC and do the same
to ensure whether the
disks are visible from the
other systems.
sympd should show only
the disks which it can use.
(symdev shows all the disks).

Also, are you sure powerpath
is not related to this problem
in any way?

It seems to be an EMC related issue. So,my suggestion would be to use
the sym commands to make
sure.

I hope it is not been
configured as a Gatekeeper device! ;-)

HTH
raj

[Sridhar Bhaskarla]

Alan,

Carlos mentioned a good point. How about trying like this?.

1. Create a Volume of equal size on the EMC.
2. Do a dd from the problematic disk to the new volume.
3. Try to import the volume group from the new volume and see if it works??.

Just curious.

-Sri

[Alan Riggs]

I agree that this is probablly a Symmetrix issue. I just haven't been able to get EMC to own up to it yet (or provide a solution). We have a microcode upgrade scheduled for other reasons, and at this point I am waiting to se ewhether that resolves this condition as well.

[Alan Riggs]

At present I am unable to publish new volumes to the host through Volume Logix (hence the scheduled microcode upgrade). Once I am able to, I will experiment with activating a BCV copy of the device.

That isn't a good solution to my problem, though. My concern is that this condition might recur on a large VG holding production data, which would be (in technical terms) a "very bad thing".