vgdisplay -v shows "could not attach PV to the volume group"

[Ra Jose]

I have strange probem. Spent some time on it, could not figure out. This is HP-UX 11iv1 on rp4440(uptodate patches with HP).

The /dev/vg02 has 3 LVs and 1 PV. The lvmtab shows correct info on PV. The ctd #s for PV are
c6t10d2 (primary) and c9t10d2 (alternate).

When I do lssf on these, I get
lssf /dev/dsk/c9t10d2
sdisk card instance 6 SCSI target 10 SCSI LUN 2 section 0 at address ??? /dev/dsk/c6t10d2
root@kmhp44#2555 /home/rjoshi/symdisks/Apr26_2007
% > lssf /dev/dsk/c9t10d2
sdisk card instance 9 SCSI target 10 SCSI LUN 2 section 0 at address ??? /dev/dsk/c9t10d2

The vgdisplay -v and lvdisplay -v all show
these PV devices correctly.

As you can see, there is no path to these devices (from lssf output).

But when you check the PVID, VGID from these 2 devices c6t10d2 or c9t10d2, they show correct.

So, I moved lvmtab to lvmtab.s and recreated lvmtab using -a option. This created good lvmtab. Now, I can see that vg02 has c18t1d0 (primary)and c19t1d0 (alt). The lssf shows EMC paths correctly for 2 devices. These h/w paths match correctly with the HBA's that I have.
AFter the new lvmtab was created, now my vgdisplay -v and lvdisplay -v on any vg or lvol, is saying that "could not attach PV " for all the PVs. These pvnames printed are the old ones (c6t10d2, c9t10d2).

Somehow the server seems to function fine.

I am guessing that, the HBA's may have been moved to different slots, which created these new ctd c18t1d0 and c16t0d0 for vg02. If so,
why then vgdisplay -v does not show me these devices.

Most of it, with old lvmtab file, server has been rebooted several times. I am wondering how
it is functioning with old device files which point to no devices. lvdisplay -v also showed ??? under PV1 column.

Can anyone explain, what could have happened.

vgdisplay -v gives could not attach PV to the volume group

Thank you all for your replies.

Rajos.

[Rita C Workman · Accepted solution]

Because the O/S on HPUX is not only smart, it remembers. These disks used to be called what you say, then something changed and now they are called the new c-t-d-. But HPUX remember who they were and who they are...hence you keep working.

You mention EMC, so I wonder if you are running PowerPath. This is not uncommon.

Do you run PowerPath ?

Rita

[Ra Jose]

Thank you Rita.

yes we run powerpath 4.1.0.

Is it that the old CTDs are recorded in ioconfig. Or does powerpath remembers in its config.

Somehow, the lssf on these old devices does not show any HP path. But all other commands on these devices such as diskinfo, dd, xd, work and show correct output.

Rajos

[Rita C Workman]

PowerPath and HPUX sometimes get into little 'tiffs'.

Bet if you run "powermt check" you will find some dead connections. Try it.

If you do, then to research, run "powermt diskplay dev=all" and look down for those disks and you will probably see dead connections and active connections to the same disk. Try it.

If you see the dead/active, you'll notice the dead is the first line. So there is the problem...

Run "powermt check" again and see if you can kill those dead connections. Sometimes you can-sometimes you can't. What you need to do is reboot. Sorry, but that's the easiest way.
Reboot, do not start up your app yet. Keep mountpoints down.
Run powermt check and clean up dead connections. I run till it runs clean with no errors.
Then run "powermt config"
Then run "powermt save"
Now restart apps back up.


...anytime you change things on disk, you need to remember to account for things in PowerPath. Especially when you remove or change things. You have keep the O/S & PowerPath in sync. That's what happened - you got out of sync with O/S to disk.

Let me know how it goes,
Regards,
Rita

[Rita C Workman]

...cursed typo again....

powermt display dev=all

Sorry,
Rita

[Steven E. Protter]

Shalom Rajos,

What happened: A disk died or was improperly removed from the system.

cd /etc
mv lvmtab lvmtab.save

vgscan -a

You may need to vgreduce -f the volume group. If so follow the instructions that follow up the command.

SEP

[Ra Jose]

Thank you Steven.

I have this disk which is good. It has 4 device files. 2 old and 2 new. The 2 new device files (primary and alternate) are good and I can do diskinfo, dd, xd.

Somehow the the kernel recognizes the 2 old device files. If I do pvdisplay on old device file, gives errors that it does not belong to vg. But somehow, lvmtab has this and has been working for months. We reboot every month and it has been working with old ctd devices in lvmtab.

The powermt display dev= shows dead status. I cannot remove this dead device also.

I tried adding these 2 new devices but vgextend gives errors. There is one device in this VG that seems to be missing. The lvdisplay -v lvol in this VG show ??? under PV1.

WOndering how the server seems to function fine with old ctd devices for VGs.

[Rita C Workman]

Hi again Rajos,

It's the PowerPath vs HPUX O/S. If you can not clear the dead connections using "powermt check", then schedule to reboot the box. And like I mentioned, with the box rebooted and quiet (only vg00 mounted) then re-run the command; clean up dead connections till powermt check runs clean. The powermt config; then powermt save.
Remember to run those commands anytime you make changes with disk (add, delete, change). It helps keep things cleaner.

Then with the vg in question 'active' try to vgdisplay now and see if things are cleaner. Sometimes I have to 'tap' them (vgchange) with a -a n and then another -a y.
But let's see if PowerPath and HPUX O/S are playing nice again together.

Rgrds,
Rita

[Ra Jose]

I was able to get rid off most of the dead devices using powermt check and then answering y for removing dead devices.

However, some devices could not be removed as it said "device in use".

Now this is vg10 and has all these devices which are old. The device file exists but no path to the device.

I have given output (cut short as many devices are there) -
# vgdislay -v vg10
--- Volume groups ---
VG Name /dev/vg10
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 32
Open LV 32
Max PV 255
Cur PV 11
Act PV 10
Max PE per PV 5000
VGDA 20
PE Size (Mbytes) 16
Total PE 21560
Alloc PE 19465
Free PE 2095
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c5t15d7":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
--- Logical volumes ---
LV Name /dev/vg10/lvol1
LV Status available/syncd
LV Size (Mbytes) 8000
Current LE 500
Allocated PE 500
Used PV 1

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c5t15d7":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name /dev/vg10/lvol2
LV Status available/syncd
LV Size (Mbytes) 4000
Current LE 250
Allocated PE 250
Used PV 1

I know what c-t-d (new ones which are good) belong to this vg10. I can vgextend vg10 with these new devices and remove the old ones.

When I try vgextendong, I get errors.
e.g.

vgextend: /dev/dsk/c5t15d7 has no correspoding valid raw device file under /dev/rdsk.
Verification of unique LVM disk id on each disk in the volume group
vg10 failed.

Somehow, if you look at the top, vgdisplay vg10 output, it shows current devices as 11 and activate devices as 10. So this one device /dev/dsk/c5t15d7 causing trouble to extend this. I checked some LVs in this vg10,
they show PV status as ???

This VG is used by sybase. All of its LVs are used as raw devices.

Any suggestions, how to add the new devices to it and then remove old ones.

Probably powermt remove dev=these_devices
may succeed.

Thank you all
Rajos

[Rita C Workman]

To finally clear this up the devices must be 'not busy'. So when you can get things to a not busy state, then clear up the dead links. Remember you have Powerpath giving orders at one end and the HPUX O/S giving orders at the other end. It's like a marriage with poor communication where nobody forgets what you said in the past, and nobody is very forgiving.
If you are able to clean them up, then please run the powermt config and then powermt save.

[One last thing-which is a final option- is to mv /etc/powermt.custom to /etc/ and reboot the box. This will cause EMC to recreate this file]
But generally just a reboot is sufficient.

When you get your dead connections cleared up and Powerpath fixed, you should be ok.

Adios,
Rita

[Ra Jose]

Thank you Rita. Your solution worked.

I did take out all the dead paths. As I said above, there was one VG which was complaining and I was unable to remove the dead paths as it was saying incorrectly the device being used.

After checking this VG, there was one device that was in lvmtab but was not there in /dev/{r}dsk/ directories. I had to create this device file using mknod (although I know there is no real path existing to this). I did this after carefully viewing /dev/{r}dsk/ directory to make sure I us ethe right minor/major #s. After I created this device file, put correct permissions/ownerships, then all seem to work fine.

I was able to vgextend all of VGs with new correct devices, then remove the old ones, then remove it from powerpath.

This was done without reboot.

Thx for your help. Hudos...

Rajos