VxVM powerpath service guard issue

[Todd Bowden]

To all,

Last night we seemed to have a strange problem with one of our packages in a Service Guard (v.11.15) cluster. This particular package is a VxVM filesystem with 4 disks associated with it. When we tried to fail it over to the alternate node the error message below was indicated. When we brought up the GUI for VxVM the disk group would not import so we had to force import it. After forcing it to import 2 (the ones listed below in the error message) disks said it was disconnected, however these same 2 disks were available in the free pool along with the alternate links.

vxvm:vxdg: WARNING: Disk veritas03: Not found, last known location: c34t2d4
vxvm:vxdg: WARNING: Disk veritas04: Not found, last known location: c34t2d5
vxvm:vxvol: ERROR: Volume vol01 has no associated data plexes
ERROR: Function activate_disk_group
ERROR: Failed to vxvol -g veritas startall

After having HP on the phone trying to help us solve this problem we kept playing around with moving these disks out of the free pool and into our DG, which it could do successfully. We tried disabling the alternate paths for power path, but no luck. What we ended up doing was right mouse clicking on the disconnected drives and replacing them with the same ones in the free pool. Now we could deport and import the DG just fine but we had to force start the volume in order to mount that filesystem.

Has anyone had a similar problem?

[Rita C Workman]

Powerpath will remember old information on disks. For whatever reason.

Did you by any chance...after the server could see the disks...
Did you try running ... powermt check
and clean up any dead connections. (i.e. answer yes to kill them). Remember to to run powermt config and then powermt save after.
Then retry your import.

I'm not used to the VxVM lingo, but I've run into 'similar' strange things under LVM when disks change. ESPECIALLY when I break up meta's.

Just a thought,
Rgrds,
Rita

[Todd Bowden]

Thanks but this package had been up for a while. We are just not sure what happened.

[Rita C Workman]

..yes but we're not talking the package. If I read correctly it was having trouble finding disks. Disks that you know are there.

Like I said, I had the issue that disks that the host saw, it couldn't readily get to (sounds crazy I know), because powerpath was holding on to some old disk reference info.
You mention powerpath. And that is an EMC software product that controls I/O to the disks. So if you are, in fact, using Powerpath, then it is possible that Powerpath is the culprit.

Now if your term powerpath is not referencing the EMC product...than..oops sorry for the confusion.

Rgrds,
Rita

[Todd Bowden]

Yes we are talking about powerpath, but doing a powermt check on the devices showed that they were both available and online to both paths. Im wondering now if there is a way to tell which path Powerpath is actually using at the time. We did check ioscan to make sure we didnt have any NO_HW for any disks. This was not the case. We could diskinfo the problem disks. Maybe there are too many pieces to this puzzle.

[Rita C Workman]

You can compare the hardware address on your host to what Powerpath shows.

Run: powermt display dev=all | more

It runs in numerical order of the disk numbers (i.e. EMC's disk number).
You should be able to look down and compare the information between your two sources to confirm they match up.

Rgrds,
Rita

[Todd Bowden]

Thanks, the hardware addresses did match up.

Veritas is the one that seems confused.

[Rita C Workman]

Ahhhh.....well I'll leave you to that, since I haven't gone VxVM yet.

Hope some things help you in your search for resolution....
Rgrds,
Rita