PV defined as missing on unrelated LUNs

[mikewjohnston]

Yesterday, we did some storage maintenance for an AIX 5.3/Oracle server. The server is attached to an EVA 5000 running VCS3.110.

The server had 22 luns presented.

Our AIX admin exported a volume group (one of three on the server) and deleted the associated HDISKs. This VG consisted of 3 luns. The lun IDs were 13,14 and 18. In addition to unpresenting the 3 luns associated with this VG, we were adding two new luns for archive log volumes (different sizes, for those of you asking why we didn't re-purpose two of the luns slated for removal).

At any rate, the removals and additions worked as usual, and since I removed luns 13,14 and 18 first, the two new luns were auto-assigned to IDs 13 and 14.

My problem and question is this....

After the addition and removal were complete, the AIX admin ran cfgmgr to scan for the changes. To our surprise, he noticed a ton of LVM I/O errors on completely different disks. LUNS 16 and 19 were no longer visible to the host even though nothing changed for these two disks. They were never unpresented or touched at all throughout this process.

We then rolled back our change and made everything as it was (since I had not deleted the old VDisks, only unpresented them). Nothing changed. The PVIDs associated with luns 16 and 19 showed as missing no matter what we tried. Even after a reboot, the server still showed the luns missing.

Our final resolution was to unpresent all storage for that host and re-present it in the same lun order. Only then was the host able to see luns 16 and 19 and bring the VG online.

I am in a state of complete confusion as to why two luns that were not touched in the process were made inaccessible to the server.