Error using vgextend with EMC SCSI connections

[John M. Robinson]

The two SCSI cables to our EMC array were accidently disconnected. When they were re-connected, I found that I had only primary links to my disks; this causes a performance problem. They were balanced over the two cables; some primary on one cable and some primary on the other.
I have tried using vgextend to get back my alternate links, but the vgextend doesn't work.

vgextend /dev/vgnksolar002 /dev/dsk/c0t5d0
vgextend: Couldn't open physical volume "/dev/dsk/c0t5d0":
Device busy

What do I do?

John Robinson

[Eileen Millen]

I lost my alternate links at one point also.
You should be able to extend them with vgextend. Did you umount the disks first?

pvcreate -s /dev/dsk/c0t5d0
can be used to set the disk to primary

[John M. Robinson]

I couldn't umount the file system. It is not mounted, but it is used as a raw device by sybase.

[Michael Tully]

Hi John,

What displays in you lvm output?

# vgdisplay -v /dev/yourvg

Are both primary and secondary disks displayed?
If so what exactly does it report?

Regards
Michael

[Varghese Mathew]

Hi,

You have just disconnected the cables physically from the EMC Array, right?, then i don'nt think the entry for the alternate link will disappear just because of that.

What's the output of #strings /etc/lvmtab, search for your vg details, from this you can compare the device files entries of the concerned hard disks and the out put of a command # ioscan -fnC disk

In case you need more help, you need to post additional information about your system.

Hope this throws some light ..

Cheers !!!
Mathew

[Bill McNAMARA_1]

ioscan -fnk

vgdisplay -v vgemc

strings /etc/lvmtab

If you don't send this info, it'll be nearly impossible to guess.. I'd reckon that the disk is showing up as no hw, try reconnecting the cable and running a full ioscan again.

ioscan -f

Take a cup of coffee then

ioscan -fnkCdisk
hopefully it'll show up.

then vgchange -a y vgemc

Later,
Bill

[John M. Robinson]

The devices are in lvmtab.

When I did a vgdisplay I got:

vgdisplay: Warning: couldn't query all of the physical volumes.
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t5d0":
The specified path does not correspond to physical volume attached to
this volume group

This is what happened when I did the vgchange:

vgchange -a y vgnksolar002
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c0t5d0":
Device busy
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c0t14d0":
Device busy
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c0t14d1":
Device busy
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c0t14d2":
Device busy
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c0t14d3":
Device busy
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c0t14d4":
Device busy
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c0t14d5":
Device busy
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c0t14d6":
Device busy
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c0t14d7":
Device busy
Volume group "vgnksolar002" has been successfully changed.

I am still at wits end.

Thanks for your help in advance.

[MANOJ SRIVASTAVA]

Hi John


You need to do a vgcfgrestore using the back oup of the lvmconf file , do a man vgcfgrestore to get the option , I think it is
(- n) but u need to check the contents of old lvmconf file so that it is fine.

Manoj Srivastava

[Bill McNAMARA_1]

>vgdisplay: Warning: couldn't query physical
> volume "/dev/dsk/c0t5d0":
>The specified path does not correspond to
>physical volume attached to
>this volume group

can you post up ioscan -fnk

If you recabled the scsi cables back on different paths the device files will change.. thus the lvm vg information will have to be corrected.(mv /etc/lvmtab /etc/lvmtab.importantbackup; vgscan or vgexport/vgimport after correct device file analysis)

Are you certain that you can access by the all the pv's by their primary link no problem.
A vgdisplay -v vgname IS REQUIRED to verify examining the alternate link status.
The order the PV device files appear in the vgdisplay -v output should be identical to that in the vgdisplay (unless hw has failed/changed since vgcreation time)

Later,
Bill

[Jordan Bean]

What does ControlCenter say about the status of the Symmetrix and the affected directors? Is PowerPath installed? If so, which version and what is the output of `powermt display` and `powermt vgdisplay dev=all`? I did notice that each controller has the same TL sequence, but the TLs between pvlinks are not identical, as one may expect, but are crisscrossed. Devices c0t[0-5]d* are mirrored at c1t[8-14]d*, and c0t[8-14]d* at c1t[0-5]d*. Is this correct? I've attached the sort of your inq by hyper label.

[John M. Robinson]

PowerPath is not installed and the directors are showing no problems on Control Center.

EMC has dialed into the box and has found no problems.

Your assumption about the mirroring is correct.

[Jeff Gyurko]

We've got a server that looses it's PV Links on each reboot. I get the same error as your describing. We resolve it by re-activating the volume group in question, even if the VG is already active, just reactivate. The errors go away and we can see all disks again. Of course you will not get the performance balance as all the primaries will be on the one controller. You can then do your vgreduce/vgextend to balance the load again.
Hope this helps

[John M. Robinson]

How do you reactivate a volume group?

[MANOJ SRIVASTAVA]

Hi John


do


vgchange -a y < vgname > to reactivate the volume group.


Manoj Srivastava

[Jordan Bean]

Let's see what what would be involved by exporting and importing the volume group:

First, let's preview what the export would do:

# vgexport -pv -m vgname.map -f vgname.pv vgname

Check the order of physical volumes in vgname.pv. If they're all there in the right order, STOP! Do not proceed unless you think this will help.

If not, make a new copy with the correct order (vgname.pv.new) and proceed. I had to do this to correct a poorly balanced set of pvlinks across four controllers to our Symmetrix. We are using PowerPath, so performance wasn't an issue... The configuration was just aesthetically grotesque.

# ls -l /dev/vgname/group # remember the major and minor!

# vgchange -a n vgname
# vgexport -m vgname.map -f vgname.pv vgname
# mkdir /dev/vgname
# mknod /dev/vgname/group c
# vgimport -m vgname.map -f vgname.pv.new vgname
# vgchange -a y vgname
# vgcfgbackup vgname
# vgdisplay -v vgname

[John M. Robinson]

I opened tickets on this with HP and EMC.
EMC is saying that if this server has an old host bus adapter, the device busy message will never go away. They said that this is not recoverable.
HP is looking into the possibility of send a hardware tech to our site to help determine the problem.

It looks like plan B or plan C will be to delete the volume group and re-create it using different disks on the EMC array.

I am still looking for ideas, but am beginning to think about hardware problems or plan B or C.

[Jordan Bean]

Time to get dirty. I'd like the wizards to comment on these procedures.

Has the host been rebooted at all? How much downtime can you afford? Let's see how deep this issue goes.

After exporting the volume group (see my previous post), remove all the device files.

# rmsf -vaH 8/12; rmsf -vaH 8/8;

Read the man page first. If the kernel bitches at this point, then HP support is imperative.

Then switch off the ports on the Symmetrix and completely power down the host. We want the HBAs to reset at startup. Power up the host in LVM maintenance mode. Run

# ioscan -fnH 8/8; ioscan -fnH 8/12

to verify that the controllers are available. Switch on the ports on the Symmetrix and run ioscans again. If all devices are present, then

# insf -eH 8/8; insf -eH 8/12

As long as the device paths didn't change, you should be able to safely import the volume group (see previous post).

[Bill McNAMARA_1]

That's really strange.
The problem is just with the identification through 8/0 and everything on 8/12 is fine.
Which means that your data is fine and there's no need to restore.

I don't believe its a problem on the hp-ux side. The problem is access on the 8/8 bus.
Hopefully there is not a pin problem on the SCSI cable.

Try forcing the switch to the alternate link with pvchange. I forget the exact arguement but use the -s or -x option to test your alternate link. Be carefull though because one of them changes the policy of switching.. to yes switch automatically on pri failure (default) or no wait until I pvchange -s or -x to do the switch..
The vgdisplay -v vg will probably indicate that only pv links are missing, but the vg should be fine.

There was a recent lun gap problem on the emc on hp-ux that was fixed in the latest SCSI/LVM cumulative, I'm not sure if that could be related... I would imagine no, because you didn't delete luns.. may be worth installing it just in case.

Later,
Bill

[John M. Robinson]

The HP hardware engineer came and replaced the SCSI adapter card with a new one.

This has solved the problem. I do not get "device busy" and all of the PV on
the EMC array can be seen.
I quickly performed vgreduce and vgextend to get the proper primary and alternate balancing that I wanted.