vgimport/vgcreate strange problem?

[Rasheed Tamton]

This seems to be a simple problem but very strange also a long post to read:

I have two systems connected to SAN (EMC). The hostA (source) is split to BCV and from
there it is synced to hostB (destination). This is a DR setup.

We have four VGs on hostA. I vgexported the vgs from hostA, copied the maps to hostB and
tried to vgimport it and it failed.

$vgimport -v -m oravg2.map /dev/oravg2
Beginning the import process on Volume Group "/dev/oravg2".
vgimport: Unable to read the physical volume.

Then I created a txt file (two) including all the disks for the oravg2 (something like below)
/dev/dsk/c6t4d6 /dev/dsk/c6t4d7 /dev/dsk/c6t5d0 /dev/dsk/c6t5d1 etc ...

and used the import command again:
$vgimport -v -m oravg2.map /dev/oravg2 `cat two`

Beginning the import process on Volume Group "/dev/oravg2".
Logical Volume is not defined on any physical volume.
"/dev/oravg2/lvol1b" is missing Physical Volumes.
Warning: Logical Volume number "1" found on physical volume not found in "oravg2.map".
Logical volume "/dev/oravg2/lvol1" has been successfully created
with lv number 1.
Volume group "/dev/oravg2" has been successfully created.
Warning: A backup of this volume group may not exist on this machine.
Please remember to take a backup using the vgcfgbackup command after activating the

volume group.

The volume group was created successfully but when I tried to mount the LV on the VG; the
fsck fails:

$fsck /dev/oravg2/lvol1
fsck: /etc/default/fs is used for determining the file system type
fileset 999, invalid magic number in primary IAU 1
log replay in progress
fileset 1 primary-ilist inode 65 has invalid number of blocks (4234)
fileset 1 primary-ilist inode 65 has invalid block map
fileset 1 primary-ilist inode 97 has invalid number of blocks (4234)
fileset 1 primary-ilist inode 97 has invalid block map
no valid ILISTs for fileset 999
file system check failure, aborting ...
Beginning the import process on Volume Group "/dev/oravg2".
Logical Volume is not defined on any physical volume.
"/dev/oravg2/lvol1b" is missing Physical Volumes.
Warning: Logical Volume number "1" found on physical volume not found in "oravg2.map".
Logical volume "/dev/oravg2/lvol1" has been successfully created
with lv number 1.
Volume group "/dev/oravg2" has been successfully created.
Warning: A backup of this volume group may not exist on this machine.
Please remember to take a backup using the vgcfgbackup command after activating the
volume group.

It happens that the vgimport does not copy the complete vol grp config of the source system, e.g,

the Max PE per PV is 2157 on the source, and when I imported it rounds up to 2178 and the
single LV on the VG (lvol1b) on the source becomes lvol1 on the destination, etc. But it gets correctly the no. of PVs (Max PV) and Cur PV.

SOURCE system: (Pls see the Total PE and Free PE on the source/destination) Is this the cause for my LV failing to mount:

--- Physical volumes ---

PV Name /dev/dsk/c33t10d4
PV Name /dev/dsk/c35t10d4 Alternate Link
PV Status available
Total PE 2157
Free PE 0
Autoswitch On

DESITNATION:
PV Name /dev/dsk/c6t8d5
PV Status available
Total PE 2177
Free PE 0
Autoswitch On

I tried to manually create the VG giving the Max PE/PV as 2157 (below)and still it rounded to 2178.

$vgcreate -e 2157 /dev/oravg2 `cat two`
Increased the number of physical extents per physical volume to 2178.
vgcreate: Volume group "/dev/oravg3" could not be created:
The path does not specify a valid physical volume.

VGCREATE ALSO DID THE 2178 ROUNDUP AS ABOVE

I did all the steps before creating the VGs (vgexport to remove the existing one, mkdir/mknod group/vgcreate, etc). The LUN sizes on the both the systems are same but the EMC boxes are different.

My question is how I can restrict the Max PE/PV to 2157 on the destination system as the
source system. Do I need any patch or any trick. As far as I remember, I did the same tasks successfully on the same system a few months ago.

Thanks
Rasheed.

[Rasheed Tamton]

One thing more:
What I want to achive is to get the vgimport/vgcreate completes successfully and mount the file systems on the destination rather than restricting the PE (it just came to mind as a possible problem - may be it is not). Is there any chance this may something to do with the EMC setup because it is handled by some other persons.
Rgds,
Rasheed.

[Dietmar Konermann]

Rasheed,

something's wrong with your BCV setup. During import LVM reads all structural information from LVMREC which is stored at the beginning of each PV. If your VG is shown different on the destination, then you have either imported from the wrong LUNs or the LUNs don't have the expected content, i.e. their LVMRECs are not in sync.

I would suggest the following:

1) DOUBLEcheck (using syminq, etc.) that you are using the correct LUNs on the destination side.

2) DOUBLEcheck that the Timefinder device groups contain all LUNs necessary.

3) Then, export the VG on the desitnation (to get rid of it), establish and split the group to get the BCVs updated.

4) Try the vgimport again.

Best regards...
Dietmar.

[Steven E. Protter]

You may be successful with vgcreate options.

-p 15

That option in the vgcreate string will limit the vg to 15 physical volumes, allowing for more PE on the physical volumes in the vg.

I'm not sure I totally understand your situation, but hopefully this will help.

SEP

[Michael Duthie]

"vgimport -v -m oravg2.map /dev/oravg2 `cat two"

Are you 100% sure the instanace numbers (c?) match on your 2 systems ?

Try the export/import with the "-s" options rather than your "cat two" see if that makes a difference. "vgexport -s -v -m vgpkt1.map -p vgpkt1"

If not check your BCV split

[renarios]

Can you see the device files with ioscan? I missed the 'insf -e' in your list with commands.
Check your logs and find out from where it went wrong. From that point start over again.

Cheers,

Renarios

[Rasheed Tamton]

Dietmar,
I have already double checked the syminq and compared it with the list provided by EMC for importing. I rely on EMC for this.

I am going to check with the BCV setup as I think I have tried all the LVM things I know to solve this problem. The problem is the EMC guys seems to be pretty new on this. I am not sure whether they are doing it correctly!


Steve,
Max PE per PV is 2157 on the SOURCE, and when I imported it rounds up to 2178 ON DESTINATION where I have problem. In this case, I want to reduce the no. of PEs to match the source box. Also, the no. of PVs are 200 on the source and I get 200 PVs on destination also.

Micahel,
The instance Nos are correct. I already tried the vgexport with -s option even though the disks are physically not shared (just to troubleshoot) and both the disks on source and destination are on different EMC boxes. -s option also did not work.
As said I think I have to check again the BCV setup which is not under my control.

Renarios,
ioscan and insf are all ok. I can see the devices. On the syslog everything is ok except there is (I have seen it yesterday also)
0/0/8/0/1: Fibre channel host port is OFFLINE, can not scan
message for 4 FCs out of 6 FCs. The two FCs are currently used for the disks I am trying to import.

Thanks to all. Expecting more clues and of course I shall allocate the points when I close the thread.
Rasheed.

[Suraj Singh_1]

I tried to re-create the similar setup specified by you (I am not sure though). I did the following things:

1. On the main system:
# vgexport -s -pv -m /tmp/vgxx.map vgxx

2. On the 2nd machine (DR),

Copy the map file created above to say /tmp.

# mkdir /dev/vgyy
# mknod /dev/vgyy/group c 64 0x0y0000 ; 'y' is a unique no.
# vgimport -v -m /tmp/vgxx.map /dev/vgyy
# strings /etc/lvmtab ; Verify
# vgchange -a y /dev/vgyy ; Activate the VG

Now mount the LVs.

NOTE: Here i am assuming that both the systems have a common LUN assigned from SAN storage.

Regards

[Rasheed Tamton]

Thanks Suraj. I did exactly what you did and it does not work. It works on a Service Guard setup because both systems can physically see the disks and shared.

In my case the LUNs are not shared by both the systems. Because it is a DR setup using BCVs.

First, the LUNs are synced with BCV and from BCV it is synced to the destination where I get problem now.

So, physically there is a connection between different LUNs on the different EMC boxes but it is not shared.

The annoying thing is I have done the same tasks few months before without any problems. That is now I start to suspect the EMC setup in this case though not sure.

Regards,
Rasheed.

[Dietmar Konermann]

Rasheed,

just saw this in your message:

Volume group "/dev/oravg2" has been successfully created.

Looks like a vgcreate message... have you ever used vgcreate?? What you are trying to do needs vgimport only! Running vgcreate is destructive.

If this is the case, please follow my suggestions above... and then use vgimport only!

Best regards...
Dietmar.

[Geoff Wild]

Could be a masking issue...

vgexport should be:

vgexport -p -v -s -m /tmp/oravg2.map /dev/oravg2

Instead of relying on EMC - check yourself:

On hostB

symmir -g YOUR_DEVICE_GROUP query

That will show you the devs to bcvs...

On hosta:

syminq |grep cXtXdX

Do that for all the devs...make note of the Sym (actual disk - but drop the leading 2 digits). IE:

# syminq |grep c45t1d2
/dev/rdsk/c45t1d2 M(4) EMC SYMMETRIX 5670 6514F000 35354880

Disk is 14F

Then on hostb

syminq |grep 14F

Make sure hostb "sees" all the disks from hosta

Rgds...Geoff

[Geoff Wild]

BTW - what does:

hosta:
diskinfo /dev/rdsk/c33t10d4

hostb:
diskinfo /dev/rdsk/c6t8d5

reveal?

Rgds...Geoff

[Geoff Wild]

One more item - you can not vgcreate until after you did a pvcreate (with -f)....after that - your bcv's are toast - you will have to vgexport (permanent - not with a -p) and you will have to establish them again, then split after they have sync'd, then you can vgimport...

Rgds...Geoff

[Joe Short]

Rasheed,

You mention that both servers do not share access to the LUNs, and that you are only trying to access the BCVs. I suggest that the BCVs be first spearated from the primary LUNs, the run the vgchid command to change the VGID on the BCV LUNs on the server you are importing them onto. This will remove any VG configuration on those LUNs, this may be where the problem is coming from, as that info may also include how many drives are in the VG. Then manually vgimport the disks into the volume group. Similar to the attached script.

[Michael Duthie]

Rasheed,

Just read your problem again, why are you using "vgcreate". All you should do is vgexport, mkdir, mknod, vgimport.

I fear a vgcreate may have destroyed the LVM structure you are triing to import.

Mike

[Chetan_5]

Rasheed,

I do not have experience with EMC BCVs but I have worked extensively on HP's Business Copy with XP disk arrays. With BC here is the procedure we use...

vgexport maps
sync/split disks
On the machine where the BCVs have to be mounted, do a vgchgid as shown below
vgchgid `cat all_disks_belonging_to_vg01`
vgimport

[Rasheed Tamton]

Hi all,
Thank you very much for all your inputs.
Finally I opened a call with HP and after a few days of thorugh analysis with LVM tools they found out there are some ghost disks in the source system side.

We used the vgreduce -f on the volume groups of the source system.
----
$vgreduce -f /dev/oravg2

skip alternate link /dev/dsk/c35t4d7

skip alternate link /dev/dsk/c35t5d0

skip alternate link /dev/dsk/c35t5d1
..........., etc
---
After that I did all the tasks from scratch again, such as vgexport the vgs from source system, copying to destination, vgimporting, etc. Then it works.

Thanks again
Rasheed Tamton
rasheedp@excite.com