How to clear non-existent PV information from clustered production environment

[arkie]

Hi Experts,

We have a 2-node SG 11.16 cluster (landscape: SAP BW) and in fact it is production.

In Node #1 or the DB node, we are unable to generate fresh cluster configuration ascii file with the following errors like

# cmgetconf > bwdb.ascii

Error: Unable to determine a unique identifier for physical volume /dev/dsk/c12tYdZ on node bwci. Use pvcreate to give the disk an identifier

... 

Errors are for 53 such devices, viz.,

c12t0d0 thru c12t0d7 - 8 devices
c12t1d0 thru c12t1d7 - 8 devices

c12t2d0 thru c12t2d7 - 8 devices

c12t3d0 thru c12t3d7 - 8 devices

c12t4d0 thru c12t4d7 - 8 devices

c12t5d0 thru c12t5d7 - 8 devices

c12t6d1 thru c12t6d5 - 5 devices

--------------------------------------------

Total -53 devices

The cluster configuration ascii files that is getting generated (bwdb.ascii) is of size 0 bytes 

======================================================================

In Node #2 or the CI node (bwci), 'vgdisplay -v' output contains warnings like:-

vgdisplay: Warning: couldn’t query physical volume “/dev/dsk/c12t3d6”:
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn’t query all of the physical volumes

In Node #2, 'strings /etc/lvmtab' also contains PV c12tYdZ information apart from c10tYdZ and C16tYdZ, where it is observed that the c12tYdZ PVs are no longer existing even in 'ioscan -fnCdisk' output. 

Need your expert guidance on how we can go ahead and clear this non-existent PV information from the system while the production cluster is still up and running.

Thanks in advance.

[arkie]

Squeezing out my first requirement:-

/dev/vg01 is a cluster VG. It is currently active in cluster node #1.

Since, in cluster node #2, this vg01 information include /dev/dsk/c12tYdZ, which is unnecessary, and we want to remove this information, can we export the vg01 in node #2 and recreate it using the mapfile generated at node #1.

Can this activity be done online?

[Matti_Kurkela]

>/dev/vg01 is a cluster VG. It is currently active in cluster node #1.

>Since, in cluster node #2, this vg01 information include /dev/dsk/c12tYdZ, which is unnecessary, and we want to >remove this information, can we export the vg01 in node #2 and recreate it using the mapfile generated at node >#1.

Yes, this can be done on node #2 while the VG is active in node #1.

I think this problem can even be fixed on node #1 while it is online.

First, make sure the missing PVs are considered free by the LVM. All LV mirrors on them should have already been removed. If not, use lvreduce -m to remove them. If the missing PVs contain any LVs that are now lost, lvremove them. When the PVs are free, use vgreduce -f vg01 to remove the missing PVs.

(This problem sounds suspiciously like an aftermath of a SAN migration where the sysadmin has missed the last step: cleaning up the old LUNs from the LVM configuration after the migration is done, but before the old SAN is physically removed. If the migration was otherwise properly executed, there should be no harm done... but if you asked me, I would hope for a more professional performance in the future.)