Operating System - HP-UX
1836605 Members
2011 Online
110102 Solutions
New Discussion

Disk already in use by another logical volume

 
Les Harrison
Advisor

Disk already in use by another logical volume

Hi

Question. New disk was presented from our SAN (EMC).
I ran insf –e and we were able to see that disk.

Running ioscan I was able to see the physical volumes assigned to that disk. On server 1 they were:



> /dev/dsk/c14t2d0 /dev/rdsk/c14t2d0

> /dev/dsk/c16t2d0 /dev/rdsk/c16t2d0



I want to add these disks to vgclord volume group.

When I tried to pvcreate these it said they already belonged to a volume group.

I checked /etc/lvmtab but they were not there. These two servers are in a Service Guard cluster (trying to add disks to different packages under that cluster) Could this be a cluster issue?

I checked /etc/lvmtab on Server 2 in the cluster and these volumes are under the vgclwss volume group.

Can I do anything like re-assign different physical volumes to this disk from the SAN? Or is there an alternative?

Thanks

Les
13 REPLIES 13

Re: Disk already in use by another logical volume

Just stop and be careful here...

How have you estableshed that they are the same disks between the two cluster nodes... simply having the same device file name does not mean they are the same disk.

Better to post the output of:

ioscan -funCdisk

and let us figure out which disks are which...

HTH

Duncan

I am an HPE Employee
Accept or Kudo
Les Harrison
Advisor

Re: Disk already in use by another logical volume

Hi Duncan

Long time no hear. See attached for server 1 ioscan.
Les Harrison
Advisor

Re: Disk already in use by another logical volume

And server 2 ioscan attached...

Re: Disk already in use by another logical volume

Les,

OK, looking at these outputs, you need to look at the part of the hardware path that comes after the FC HBA.

From this I can determine that these boxes each have 2 FC ports connected to 2 ports on your DMX.

SO I can see that:

- disk devices which are presented from a port on the DMX which has hardware path 108.102.19 all start with c14 on server 1 and c12 on server 2.

- disk devices which are presented from a port on the DMX which has hardware path 107.110.19 all start with c16 on server 1 and c14 on server 2.

I'd imagine that the two paths are to the same sets of LUNs.

The disks you mention certainly have the highest instance number (77 and 78 on server 1 and 70 and 71 on server 2), so they certainly look like the last 2 that were added - maybe they have already been presented somehwere else before. You can check they're not already in use on these machines by running:

on server 1:
pvdisplay -v /dev/dsk/c14t2d0
pvdisplay -v /dev/dsk/c16t2d0
strings /etc/lvmtab

on server 2:
pvdisplay -v /dev/dsk/c12t2d0
pvdisplay -v /dev/dsk/c14t2d0
strings /etc/lvmtab

Even if they're not in use on these systems, if they already appear to have an LVM structure on them (as the pvcreate serror suggests), I'd at least have your SAN team check again that they haven't presented you with a LUN from another HPUX box.

HTH

Duncan

I am an HPE Employee
Accept or Kudo
Les Harrison
Advisor

Re: Disk already in use by another logical volume

Hello Duncan

Sorry for delay.

Server 1:

root@ugos389# pvdisplay -v /dev/dsk/c14t2d0
pvdisplay: Couldn't find the volume group to which
physical volume "/dev/dsk/c14t2d0" belongs.
pvdisplay: Cannot display physical volume "/dev/dsk/c14t2d0".
root@ugos389# pvdisplay -v /dev/dsk/c16t2d0
pvdisplay: Couldn't find the volume group to which
physical volume "/dev/dsk/c16t2d0" belongs.
pvdisplay: Cannot display physical volume "/dev/dsk/c16t2d0".
root@ugos389# strings /etc/lvmtab
/dev/vg00
Bi\G
/dev/dsk/c2t1d0s2
/dev/dsk/c3t0d0s2
/dev/vgapp
/dev/dsk/c14t0d2
/dev/dsk/c14t1d4
/dev/dsk/c16t0d2
/dev/dsk/c16t1d4
/dev/vgclord
/dev/dsk/c14t0d7
/dev/dsk/c14t5d7
/dev/dsk/c16t0d7
/dev/dsk/c16t5d7
/dev/dsk/c16t1d6
/dev/dsk/c14t1d6
/dev/vgclwss
/dev/dsk/c14t0d3
/dev/dsk/c14t0d4
/dev/dsk/c14t0d5
/dev/dsk/c14t0d6
/dev/dsk/c14t1d0
/dev/dsk/c14t1d1
/dev/dsk/c14t1d2
/dev/dsk/c14t1d3
/dev/dsk/c14t1d7
/dev/dsk/c14t3d5
/dev/dsk/c16t0d3
/dev/dsk/c16t0d4
/dev/dsk/c16t0d5
/dev/dsk/c16t0d6
/dev/dsk/c16t1d0
/dev/dsk/c16t1d1
/dev/dsk/c16t1d2
/dev/dsk/c16t1d3
/dev/dsk/c16t1d7
/dev/dsk/c16t3d5
Les Harrison
Advisor

Re: Disk already in use by another logical volume

Server 2 (also see attached):

root@ugos388# strings /etc/lvmtab
/dev/vg00
/dev/dsk/c2t1d0s2
/dev/dsk/c3t0d0s2
/dev/vgapp
/dev/dsk/c12t0d1
/dev/dsk/c12t1d5
/dev/dsk/c14t0d1
/dev/dsk/c14t1d5
/dev/vgclwss
/dev/dsk/c12t0d3
/dev/dsk/c12t0d4
/dev/dsk/c12t0d5
/dev/dsk/c12t0d6
/dev/dsk/c12t1d0
/dev/dsk/c12t1d1
/dev/dsk/c12t1d2
/dev/dsk/c12t1d3
/dev/dsk/c12t3d5
/dev/dsk/c14t0d3
/dev/dsk/c14t0d4
/dev/dsk/c14t0d5
/dev/dsk/c14t0d6
/dev/dsk/c14t1d0
/dev/dsk/c14t1d1
/dev/dsk/c14t1d2
/dev/dsk/c14t1d3
/dev/dsk/c14t3d5
/dev/dsk/c12t1d7
/dev/dsk/c14t1d7
/dev/dsk/c12t2d0
/dev/dsk/c14t2d0
/dev/vgclord
/dev/dsk/c12t0d7
/dev/dsk/c12t1d6
/dev/dsk/c12t5d7
/dev/dsk/c14t0d7
/dev/dsk/c14t1d6
/dev/dsk/c14t5d7
Matti_Kurkela
Honored Contributor

Re: Disk already in use by another logical volume

If the disk presented from the SAN was previously used by another HP-UX and was not thoroughly wiped, it might still contain the LVM header created by that previous system.

The pvcreate message just means "Someone has issued pvcreate on this disk already." If you wish to avoid this, you should start using the pvremove command before unpresenting old disks.

Check your SAN configuration: if the disks are in use by some other host that does not belong in your cluster, overwriting the disk will corrupt the data used by that other host.

You should install the "syminq" (or sometimes just "inq") tool from EMC: it can find the EMC-specific disk ID numbers from the SAN disks. The ioscan output is of limited utility here: without seeing the EMC identifiers you cannot be 100% sure the disk presented on system A is the same disk as presented on system B. As you've discovered, the hardware paths can (and often will) be different between systems.

If you have the symcli tools installed, "sympd list" can be used instead of "syminq".

You have already confirmed your cluster is not using these disks. If you can make sure the disks are presented to your cluster *only*, you can safely use "pvcreate -f" to get rid of the old LVM information that seems to be on these disks.

MK
MK
Ivan Krastev
Honored Contributor

Re: Disk already in use by another logical volume

Check LVM headers for both disks (c1t2d0 is used for example):
#xd -j8200 -N16 -tu /dev/rdsk/c1t2d0
0000000 2000252410 965817345 2000252410 965817462
PV CPU-ID PV timestamp VG CPU-ID VG timestamp
The above information translates to:
? pvcreate and vgcreate was run on the sytem with systemID (uname -i) 2000252410
? pvcreate was run at timestamp 965817345 (seconds after Jan 1st 1970 0:00 UTC)
? vgcreate was run at timestamp 965817462 (117 seconds later)


You fill find system ID (uname -i) of the system and VG ID as well.


regards,
ivan
Les Harrison
Advisor

Re: Disk already in use by another logical volume

Hi guys,

I ran the xd commands of server 1:

root@ugos389# xd -j8200 -N16 -tu /dev/rdsk/c14t2d0
0000000 2726266024 1208699356 2726266024 1126011188
0000010
root@ugos389# xd -j8200 -N16 -tu /dev/rdsk/c16t2d0
0000000 2726266024 1208699356 2726266024 1126011188
0000010

root@ugos389# uname -i
3752607443

On server 2:

$ uname -i
2726266024

So it looks like they were never pvcreated on server 1, just on server 2.

I'm thinking that these disks must be registered in service guard.
Looking at the Service Guard cluster config I can't see the physical volumes referenced on either server but the volume group vgclwss is:
root@ugos388# grep FIRST_CLUSTER_LOCK_VG *
cluster.ascii:# Lock Disk Parameters. Use the FIRST_CLUSTER_LOCK_VG and
cluster.ascii:# The FIRST_CLUSTER_LOCK_VG is the LVM volume group that
cluster.ascii:FIRST_CLUSTER_LOCK_VG /dev/vgclwss
wssprodcl.conf:# Lock Disk Parameters. Use the FIRST_CLUSTER_LOCK_VG and
wssprodcl.conf:# The FIRST_CLUSTER_LOCK_VG is the LVM volume group that
wssprodcl.conf:FIRST_CLUSTER_LOCK_VG /dev/vgclwss
wssprodcl.conf.011206:# Lock Disk Parameters. Use the FIRST_CLUSTER_LOCK_VG and
wssprodcl.conf.011206:# The FIRST_CLUSTER_LOCK_VG is the LVM volume group that
wssprodcl.conf.011206:FIRST_CLUSTER_LOCK_VG /dev/vgclwss
root@ugos388# grep FIRST_CLUSTER_LOCK_PV *
cluster.ascii:# FIRST_CLUSTER_LOCK_PV parameters to define a lock disk.
cluster.ascii: FIRST_CLUSTER_LOCK_PV /dev/dsk/c8t0d3
cluster.ascii: FIRST_CLUSTER_LOCK_PV /dev/dsk/c10t0d3
wssprodcl.conf:# FIRST_CLUSTER_LOCK_PV parameters to define a lock disk.
wssprodcl.conf:FIRST_CLUSTER_LOCK_PV /dev/dsk/c12t0d3
wssprodcl.conf:FIRST_CLUSTER_LOCK_PV /dev/dsk/c14t0d3
wssprodcl.conf.011206:# FIRST_CLUSTER_LOCK_PV parameters to define a lock disk.
wssprodcl.conf.011206:FIRST_CLUSTER_LOCK_PV /dev/dsk/c8t0d3
wssprodcl.conf.011206:FIRST_CLUSTER_LOCK_PV /dev/dsk/c10t0d3
root@ugos388# grep c14t2d0 *
root@ugos388#

Re: Disk already in use by another logical volume

Les,

So now I'm glad I had you check this out - as you can see from the output from server2, plus the other commands people have asked you to run, the LUN in question is already in use and already part of a different volume group on server2.

It's actually part of the volume group vgclwss on server2. It isn't part of the configuration on server1 - this means you're cluster is currently broken and wouldn't failover correctly.

Luckily the pvdisplay also indicates that there aren't any actual logical volumes on those disks so fixing the problem should be reasonably straightforward. But before going any further you need to tell us which cluster package vgclwss is part of, and on which node that package is running. Once we have that info, we should be able to fix this issue...

HTH

Duncan

I am an HPE Employee
Accept or Kudo
Les Harrison
Advisor

Re: Disk already in use by another logical volume

Cheers Duncan

vgclwss is on pkgwssprod package running on server 2 in the wssprodcl cluster. See cmviewcl:
CLUSTER STATUS
wssprodcl up

NODE STATUS STATE
ugos388 up running

PACKAGE STATUS STATE AUTO_RUN NODE
pkgwssprod up running enabled ugos388

NODE STATUS STATE
ugos389 up running

PACKAGE STATUS STATE AUTO_RUN NODE
pkgordprod up running enabled ugos389

Re: Disk already in use by another logical volume

Les,

OK so the volume group is atcive on the same node where the disk has been incorrectly added to the wrong VG - this makes life easier. To make this LUN available for adding to the other volume group, you just need to take it out of the existing volume group. You do this by vgreduce'ing both paths out of the volume group as follows:

vgreduce /dev/vgclwss /dev/dsk/c12t2d0
vgreduce /dev/vgclwss /dev/dsk/c14t2d0

AFter that the disk is available to be added to the other volume group - -ypu will still have to use "pvcreate -f" on the disk before adding it to the other VG though.

HTH

Duncan

I am an HPE Employee
Accept or Kudo
Les Harrison
Advisor

Re: Disk already in use by another logical volume

Thanks Duncan

Will have to do this on the weekend. Even though this is non destructive in that the devices are not being used, so vgreduce will not lose any data, we still have to schedule it for a weekend.

Thanks for all your help. I will let you guys know how it goes.

Thanks

Les