Serviceguard SMS - some files are missing after upgrading VxVM 5.10.00 to 5.10.01

[KathyL1]

In November 2011 I upgraded from Serviceguard SMS CFS version A.4.0 to version A.4.0.1 - swinstall did not report any errors or warnings during either the installation or configuration phases.

In April 2012 I connected some new servers to EMC Symmetrix VMAX storage for the first time (all other servers were connected to HP EVA storage) and Veritas immediately started constantly logging I/O errors in relation to the EMC 'gatekeeper' device.

As soon as I started Serviceguard CVM/CFS modular packages Veritas started disabling and unmounting the disk groups - this could happen within 30 minutes of the package starting.  Unfortunately the CVM/CFS modular packages did not fail/halt when this happened and this left the affected node (or the whole cluster if Veritas had disabled/unmounted the disk group/s on all nodes) in a very unhappy state.  It was not possible to halt the affected CVM/CFS modular packages or the SG-CFS-pkg package - the affected node had to be rebooted (and the server would crashdump on the way down because Veritas had not been shutdown properly).

As a result of this issue I was not able to implement Serviceguard SMS CFS in any clusters connected to EMC storage (and I had to convert Veritas file systems to LVM file systems) - this was a MAJOR setback because it meant I could not migrate the Oracle RAC databases to CFS and that I had to continue using cross-mounted NFS file systems (implemented with the Serviceguard NFS Toolkit).  No issues were experienced in relation to LVM file systems - only Veritas file systems were affected by this issue.

Some time later, and quite by chance, I discovered that the Veritas problem only occurred when the following setting was used: "vxddladm set namingscheme=osn mode={default|legacy}".  If the setting "vxddladm set namingscheme=osn mode=new" was used Veritas did not disable/unmount the disk group/s.  Also, removing the EMC 'gatekeeper' device from DMP control ("vxdmpadm exclude vxvm dmpnodename=cNtNdN") prevented Veritas from disabling/unmounting the disk group/s.  These were workarounds but not solutions.  Also, a couple of months ago I discovered that servers which could not see the 'EMC' gatekeeper device were not affected by this issue - these servers did not need either workaround applied.

Last week I finally received some information from L3 support that shed some light on the cause of this problem.  Analysis of logs I had supplied indicated that the Veritas Array Support Libraries(ASLs) and Array Policy Modules(APMs) for the EMC Symmetrix were not installed.  Apparently, the EMC ‘gatekeeper’ device is not visible in "vxdisk list" output when the proper ASL is installed.  I was advised to download the latest VRTSaslapm package from the Symantec website (https://sort.symantec.com/asl/details/577) and install it.  This fixed the problem - the EMC ‘gatekeeper’ device was not visible in "vxdisk list" output and Veritas did not disable/unmount the disk group/s even when the setting "vxddladm set namingscheme=osn mode={default|legacy}" was configured.

On the servers on which I hadn't applied the fix "swlist" showed that VRTSaslapm was installed but the "vxddladm listsupport all" command (I found about this command in the VRTSaslapm installation notes) returned the message 'No ASL support found with vxddladm listsupport'.  The Serviceguard SMS release notes show that VRTSaslapm is included in all versions of Serviceguard SMS so I couldn't understand why there wasn't any ASL support.  By contrast, on the servers on which I had applied the fix the "vxddladm listsupport all" command returned a list of all currently supported disk arrays.

Curious, I did an internet search for VRTSaslapm and discovered the attached Symantec TechNote (available from http://www.symantec.com/business/support/index?page=content&id=TECH179122).  It describes a known problem of the post-install script for VRTSvxvm removing the Array Support Libraries(ASLs) and Array Policy Modules(APMs) when upgrading from 5.1.100.000 (SG SMS 4.0) to 5.1.100.001 (SG SMS 4.0.1).  The TechNote states that error messages about the missing ASL/APM files will be logged in "/var/adm/sw/swagent.log" but this didn't happen in my environment.  Only the output of the "vxddladm listsupport all" command ('No ASL support found with vxddladm listsupport') confirmed this problem on the affected servers.

I then checked the latest Serviceguard SMS 4.x release notes (March 2012 - http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c02815613/c02815613.pdf) and discovered the following "Known Problem":  'QXCR1001186450 - Some files are missing after upgrading VxVM 5.10.00 to 5.10.01'.  Unfortunately this was not a known problem in November 2011!

The release notes indicate that the details of this problem can be found by logging in to the HPSC and searching the Knowledge Base for QXCR1001186450 but unfortunately the only information I get is "No documents were found for your search."

So, after more than six months of frustration, and a lot of time spent troubleshooting, this issue has finally been resolved and I can now start the process of implementing Serviceguard SMS CFS again and start planning for migrating the Oracle RAC databases to CFS.

I've asked HP to make 'QXCR1001186450' available to logged in HPSC users and to also include the details of the Symantec TechNote in the Serviceguard SMS 4.x release notes - I'm sure this issue would have been resolved much sooner if it had been properly documented in the March 2012 release notes.