vgimport fails on cluster server

[Jim Mulshine]

We've got a cluster of two servers and a Virtual Array VA7100. A disk was reconfigured with LVM into another volume group and a map file was created using vgexport on server1. Vgimport on server2 fails with the following message:

# vgimport -m /tmp/vg02.map -s -v /dev/vg02
Beginning the import process on Volume Group "/dev/vg02".
Couldn't access the list of physical volumes for volume group "/dev/vg02".

Ioscan on both servers shows CLAIMED and device files for all disks.

Running the armdsp command on server2 results in the following error:

Exception in thread "main" java.lang.NoClassDefFoundError: com/hp/cssi/Licensing/LicenseClient

Trying to check disk information from server2 from SAM also fails with the following message:

The command used to retrieve information about HP SureStore VirtualArray has failed. The stderr is shown below.
armdsp failed

Do I have a CommandView SDM licensing problem or something else?

[Steven E. Protter]

Shalom,

I would assume based on your results that the volume group may be active on the second node or there is a hung lvm command on it hanging it up and preventing access on server2.

SEP

[Geoff Wild]

"A disk was reconfigured with LVM into another volume group and a map file was created using vgexport on server1"

Well - how was that disk reconfigured - a new vg? Is it shared? IE vgchange -c y -S y /dev/vgXX on server1

How did you do the map file?

Should be like:

vgexport -s -v -p -m /tmp/vgXX.map /dev/vgXX

Then import:

mkdir /dev/vgXX
mknod /dev/vgXX/groupd c 64 0xHH0000

Where XX is vg in decimal and HH is HEX equiv.

Rgds...Geoff

[Jim Mulshine]

The volume group is not active on server2 as it was vgexport(ed) previously. It's no longer in the lvmtab. There are no hung lvm commands. The server has even been rebooted with the problem existing both before and after reboot.

The disk was removed from one volume group and added to vg02, a vg that has been cluster enabled for a few years now.

The map file was created like this on server1:

vgexport -s -v -p -m /tmp/vg02.map /dev/vg02

The import attempt looked like this on server2:

mkdir /dev/vg02
mknod /dev/vg02/group c 64 0x020000
vgimport -m /tmp/vg02.map -s -v /dev/vg02

[Geoff Wild]

The vg was in the cluster before - but the disk wasn't.

It sounds like server 2 can't "see" that disk. Make sure it has been presented to server 2 from the Surestore array.

Rgds...Geoff

[nancy rippey]

A way to check to to see if you can verify if you see all the disks is to use the PVIDs. To my knowledge HP does not make it very easy to get the IDs, run something like the following as root (BTW - we use this type of approach with our failover/DR solution):

#!/bin/ksh
echo "Disk\t\t\tPVID\t\t\tVGID"
for i in `ioscan -fnkC disk | egrep rdsk | awk '{print $2}'`
do
PV=`/usr/bin/xd -j8200 -N16 -tx $i`
echo $PV $i | awk '{print $7,"\t",$2$3,"\t",$4$5}'
done

[nancy rippey]

A way to check to to see if you can verify if you see all the disks is to use the PVIDs. To my knowledge HP does not make it very easy to get the IDs, run something like the following as root (BTW - we use this type of approach with our failover/DR solution):

#!/bin/ksh
echo "Disk\t\t\tPVID\t\t\tVGID"
for i in `ioscan -fnkC disk | egrep rdsk | awk '{print $2}'`
do
PV=`/usr/bin/xd -j8200 -N16 -tx $i`
echo $PV $i | awk '{print $7,"\t",$2$3,"\t",$4$5}'
done

Enjoy.
Tom Ferony

[Jim Mulshine]

There is no problem with server2 "seeing" the disks. What do you all think about that licensing error message that comes when I run armdsp?

[Geoff Wild]

"There is no problem with server2 "seeing" the disks. "

Okay - did you try to dd it to say /dev/null - no errors?

Try creating a new vg - with that disk - does it work - or does it say something like - "disk already belongs to another volume group"?

As far as license issue - looks more like a java error - Exception in thread "main"...

That could be something as simple as a corrupt jar file somewhere - try re-installing commandview...

Again, I must reiterate - and error like this:

# vgimport -m /tmp/vg02.map -s -v /dev/vg02
Beginning the import process on Volume Group "/dev/vg02".
Couldn't access the list of physical volumes for volume group "/dev/vg02".

means exactly what it says - it can not access all the physical volumes for that VG - it found some - but not all...

Try a manual import...using the device paths instead - does that work?

vgimport -v /dev/vg02 /dev/dsk/c0t1d0 /dev/dsk/c0t3d0

use the devices that you "know" are the ones in that vg...


Rgds...Geoff

[Jim Mulshine]

Okay, I reinstalled commandview and now all the licensing, java and armdsp-related errors are gone, so commandview is no longer an issue.

Now there is only the problem with vgimport failing. A dd to /dev/null shows that all of the disks in the vg that I want to import are readable from the node where vgimport fails.

[Geoff Wild]

Okay - that's good the java is fixed.

Did you try a manual import?

The only other thing I can think of right now is - the minor number of the group file - is it the same as another vg on that server?

ll /dev/vg*/group

Example:

# ll /dev/vg*/group
crw-r--r-- 1 root sys 64 0x000000 Aug 6 2002 /dev/vg00/group
crw-r--r-- 1 root sys 64 0x010000 Oct 3 2002 /dev/vg01/group
crw-r--r-- 1 root sys 64 0x3c0000 Oct 20 2006 /dev/vg60/group
crw-r--r-- 1 root sys 64 0x3d0000 Oct 19 2006 /dev/vg61/group
crw-r--r-- 1 root sys 64 0x3e0000 Oct 19 2006 /dev/vg62/group
crw-r--r-- 1 root sys 64 0x410000 Oct 19 2006 /dev/vg65/group
crw-r--r-- 1 root sys 64 0x420000 Oct 19 2006 /dev/vg66/group
crw-r--r-- 1 root sys 64 0x430000 Oct 19 2006 /dev/vg67/group
crw-r--r-- 1 root sys 64 0x440000 Oct 19 2006 /dev/vg68/group


Your vg02 should be 0x020000 - make sure that doesn't exist in another one.

Rgds...Geoff

[skt_skt]

I would strongly recommend to try importing by manullay specifying the disk list.

I had a similar error which had happend as i had performed vgchgid on those disks for my some other requierments.

Are u still able to take a fresh map file from server1(it looks like you had already exported the VG on server1 and stuck while importing on server2)

[Yallappa]

I thinking that you are not deactiviting the vg02 on frist cluster node. Try this.




ON CLUSTER NODE 1


Export the VG in the preview mode,
# vgexport -p â s â m /tmp/VG02.map /dev/vg02

Copy the same map file on another cluster node
umount the filesystem & deactivate vg
#vgchange â a n /dev/vg02

.


ON CLUSTER NODE 2

Export the existing configuration of vg02.
#vgexport /dev/vg02

Create the directory and group special file to import the new vg config
#mkdir /dev/vg02
#mknod /dev/vgtrdb/group c 64 0x020000 take care of minor number

Imoprt the new configuration of vg02

#vgimport â s â m /tmp/vg02.map /dev/vg02

[Jim Mulshine]

I ran now the following commands on server2:

# ll /dev/vg*/group
crw-r----- 1 root sys 64 0x000000 Feb 28 2001 /dev/vg00/group
crw-rw-rw- 1 root sys 64 0x010000 Mar 30 2001 /dev/vg01/group
crw-r--r-- 1 root sys 64 0x020000 Aug 23 12:38 /dev/vg02/group
crw-r--r-- 1 root sys 64 0x030000 Aug 26 2003 /dev/vg03/group
crw-r--r-- 1 root sys 64 0x040000 Jan 24 2003 /dev/vg04/group

# vgimport -v vg02 /dev/dsk/c5t0d5 /dev/dsk/c5t0d6 /dev/dsk/c5t0d2 /dev/dsk/c5t0d7
Beginning the import process on Volume Group "vg02".
Volume group "/dev/vg02" has been successfully created.
Warning: A backup of this volume group may not exist on this machine.
Please remember to take a backup using the vgcfgbackup command after activating the volume group.

# strings /etc/lvmtab
/dev/vg00
/dev/dsk/c1t2d0
/dev/dsk/c2t0d0
/dev/dsk/c1t0d0
/dev/dsk/c2t2d0
/dev/vg01
/dev/dsk/c5t0d1
/dev/vg04
/dev/dsk/c5t0d4
/dev/vg03
?K8d
/dev/dsk/c5t0d3
/dev/vg02
/dev/dsk/c5t0d5
/dev/dsk/c5t0d6
/dev/dsk/c5t0d2
/dev/dsk/c5t0d7

# vgchange -a r vg02
Activated volume group
Volume group "vg02" has been successfully changed.
# vgdisplay vg02
--- Volume groups ---
VG Name /dev/vg02
VG Write Access read-only
VG Status available
Max LV 255
Cur LV 0
Open LV 0
Max PV 16
Cur PV 4
Act PV 4
Max PE per PV 6655
VGDA 8
PE Size (Mbytes) 4
Total PE 26616
Alloc PE 0
Free PE 26616
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

The problem that I see is that Alloc PE in the vgdisplay result is 0. It should be the same as on the other node where vg02 was originally created, ie Alloc PE = 15425.

I cannot test yallapa's suggestion which requires deactivation of the vg on server1 (production environment).

Any ideas?

[Mridul Shrivastava]

You should not activate a VG in read only mode in a cluster environment if it is already activated in exclusive mode on other node... this may lead to unwanted results..

[Geoff Wild]

Okay - so the manual import worked - for these disks:

/dev/dsk/c5t0d5 /dev/dsk/c5t0d6 /dev/dsk/c5t0d2 /dev/dsk/c5t0d7

But as you say - they appear to have no data on them.

I don't know commandview (we are an EMC shop) - but can you run a command on each server that shows you the actual LUN? IE the disk id?

Then compare these 4 disks to the 4 that are in vg02 on server1.

Exaample - on my server1 I have this disk in vg60:

c4t4d4

Running an inq shows:

-----------------------------------------------------------------------------
DEVICE :VEND :PROD :REV :SER NUM :CAP(kb)
-----------------------------------------------------------------------------

/dev/rdsk/c4t4d4 :EMC :SYMMETRIX :5771 :52000f7000 : 77760000

My disk is 0f7

On server2:

inq |grep 0f7

/dev/rdsk/c4t4d4 :EMC :SYMMETRIX :5771 :52000f7000 : 77760000
/dev/rdsk/c84t4d4 :EMC :SYMMETRIX :5771 :52000f7000 : 77760000
/dev/rdsk/c90t4d4 :EMC :SYMMETRIX :5771 :52000f7000 : 77760000
/dev/rdsk/c92t4d4 :EMC :SYMMETRIX :5771 :52000f7000 : 77760000

As you see, it just happens to be the same device number - but the important thing is that it is the same serial number - so I know it is the same disk.

At this time, your vg02 on server2 is useless - so it needs to be deactivated and exported on server2.

Recheck your disks - make sure they are really the same - device numbers don't have to be the same - for on my server1, the alternate paths have different devs:

/dev/rdsk/c4t4d4 :EMC :SYMMETRIX :5771 :52000f7000 : 77760000
/dev/rdsk/c74t4d4 :EMC :SYMMETRIX :5771 :52000f7000 : 77760000
/dev/rdsk/c76t4d4 :EMC :SYMMETRIX :5771 :52000f7000 : 77760000
/dev/rdsk/c78t4d4 :EMC :SYMMETRIX :5771 :52000f7000 : 77760000

Once you are positive that the disks on server1 are "seen" on server2, then try importing again with the map file.

Rgds...Geoff

[Jim Mulshine]

I checked the VGID of the disks an all of the disks as seen from each of the two servers have the same VGID on them. However it is not the same VGID that appears in the mapfile created by a vgexport from server1.

Over the weekend I had the opportunity to stop the production system for a while. I then used vgcfgrestore to restore the correct LVM configuration to each of the involved disks. I did this:

On server1:
# vgchange -a n vg02
# vgcfgrestore -n vg02 /dev/rdsk/c4t0d2
# vgcfgrestore -n vg02 /dev/rdsk/c4t0d5
# vgcfgrestore -n vg02 /dev/rdsk/c4t0d6
# vgcfgrestore -n vg02 /dev/rdsk/c4t0d7
# vgchange -a e vg02
# vgexport -s -v -p -m vg02.map vg02
# rcp vg02.map server2:

On server2:
# mkdir /dev/vg02
# mknod /dev/vg02/group c 64 0x020000
# vgimport -s -v -m vg02.map vg02

This solved the problem. Now the VGID on the disks is the same as the one shown in the mapfile and the volume group "works" on both servers. Thank you all for the advice you gave me regarding this problem!