HPVM guests as Serviceguard nodes- monitoring critical resources

[RSA]

Given the following environment: HP/UX 11i v3 September 2011 (hosts and guests), Integrity VM 4.30, Serviceguard 11.20, SGeSAP 05.10

Two HPVM guests setup as Serviceguard nodes:
- Whole disks used as backing storage (AVIO devices)
- For each HPVM guest two fibre channel SAN LUNs as backing storage, one for OS and one for application including data
- HA failover capabilities needed like in traditional Serviceguard cluster configurations
- Application failover rather than HPVM guest failover

What is the best approach to monitor critical resources like physical volumes used in the volume groups for the HPVM guest operating systems as well as the application binaries and data configured in the Serviceguard package?

My idea would be to use SFM/WBEM, for example having a script executing the CIMUtil -e root/cimv2 HP_DiskDrive command to get the volume group disk status. This could be added to the package configuration using the sg/service and sg/generic_resource modules. But what happens if the filesystem storing the monitoring script isn't available anymore? IO timeouts...

What happens if for example the storage array providing the LUNs for the HPVM guest operating system and application data goes down? Everything in memory will continue to run, like for example the server is still pingable, cmcld that determines cluster membership is running in memory and is still able to send heartbeat packages. But everything else... from my experience the operating system will be in a "frozen" state. How is Serviceguard running on HPVM guests able to detect these issues if the backing storage isn't available anymore? In my test environment no failover occurred, cmviewcl -v showed the package still up and running even though the critical resource, the volume group disks, weren't available anymore.

I'm looking for any thoughts/ideas on how-to improve the Serviceguard package failover configuration in the described scenario.

Thanks,

RSA

[asghar_62]

Simply monitoring each physical disk in a Serviceguard cluster does not provide

adequate monitoring for volumes managed by Veritas Volume Manager from Symantec

(VxVM), or logical volumes managed by HP-UX Logical Volume Manager (LVM),

because a physical volume failure is not always a critical failure that triggers failover (for example, the failure of a mirrored volume is not considered critical).

For this reason, it can be very difficult to determine which physical disks must be

monitored to ensure that a logical volume is functioning properly. The HP Serviceguard Volume Monitor provides a means for effective and persistent monitoring of storage volumes.

The LVM monitoring capability (cmvolmond) is relatively new for Serviceguard A.11.20,

and requires the September 2010 Patch.

cmvolmond replaces cmvxserviced, combining the VxVM monitoring capabilities

of cmvxserviced with the new capabilities needed to support LVM monitoring.

Although cmvxserviced will still work in A.11.20, HP recommends you use

cmvolmond instead.

[melvyn burnard]

So if I get this right, you have two HP Integrity Virtual Machines, both are set up as Serviceguard nodes.

Am I correct to assume they are on different Hosts?

I also assume that the Guest OS resides on the SAN infrastructure?

The way I see it, you would need to monitor the storage on teh Integrity Virtual Machine Host, and then pass this down to the Serviceguard nodes.

The challenge is as you state, the SG node has now lost it's backing store so how does this help.

Off the top of my head I would say that you are trying to cover from an MPOF rather than an SPOF, which becomes more challenging.

Therefore I would suggest that the Virtual Machine Host should monitor the SAN storage, and if it ALL goes down on that particular Host, the monitoring should somehow trigger off a TC to the Virtual Machine, FORCING a failover.

This may NOT be a "neat" solution, but it is what I can think of right now.

The other option is, of course, is to put the VM Guest OS disks on internal storage or as files on each Host, which would remove the issue of the Guest OS being able unable to respond to any triggers or events that could be set up inside the Guest OS.

Just my current two pence worth

[RSA]

Hello Asghar Riahi_1,

I will definitely look into the new LVM monitoring capability (cmvolmond). My Serviceguard environment is running on A.11.20, I will also make sure that I have at least the September 2010 patch installed.

Thanks,

RSA

Quick update after some initial testing with cmvolmond. I added the following lines to my SG package configuration:

service_name                            lvol_monitor
service_cmd                             /usr/sbin/cmvolmond -O /var/adm/cmcluster/log/lvol1.log -D 1 -t 61 /dev/vg_cluster/lvol1
service_restart                         None
service_fail_fast_enabled               yes
service_halt_timeout                    90

Checked and applied the configuration to the package. Monitoring seems to work:

# tail -f /var/adm/cmcluster/log/lvol1.log
Jul 24 21:19:10 Sleeping 61
Jul 24 21:20:11 Sleeping 61

Nevertheless with both fibre connections disabled, monitoring also "freezes" until I re-enable the connections again:

Jul 24 21:27:54 Timed-out 29159, terminating
Jul 24 21:27:54 Terminated cmcheckvx
Jul 24 21:27:55 Sleeping 61

The package didn't switch from the primary node to the secondary node in this case. Maybe some further fine tuning is required.

[RSA]

Hello Melvyn Burnard,

Yes, you are correct: two HP Integrity VMs, both are set up as Serviceguard nodes and they run on different hosts. The Guest OS resides on the SAN infrastructure and the disks are passed down from the hosts to the guests. I'm trying to cover a full storage array outage and/or redundant SAN switch outage, which is more than a SPOF and it's getting pretty difficult to monitor this properly.

I also had the idea to run a second Serviceguard installation on the Integrity hosts itself to monitor the disks as well as fibre channel connections. This could be a way to force a failover. Would be a little difficult to manage, but this could be a solution... even though it adds a lot of complexity. 

Internal storage for the guest OS would be an alternative too, but this makes it more difficult to move the guest from one host to another in the case of Integrity VM host maintenance.

I'm trying to figure out why the guest OS and Serviceguard is "freezing" and not causing a reboot. Maybe tweaking timeout parameters for PVs and LVs could improve the situation.

Thanks,
RSA

[asghar_62]

RSA,

Are you using the Serviceguard tool kit for Integrity Virtual machines?

It is highly recommended to install the VM guest management software, especially on VM guests functioning as Serviceguard nodes, in order for Serviceguard to determine an optimal io_timeout_extension value (otherwise, Serviceguard would assume the most conservative value of 70 seconds resulting in unnecessarily lengthening the cluster recovery time).

 The io_timeout_extension parameter is set internally by Serviceguard and is not configurable by the user; however its value can be viewed using the Serviceguard cmviewconf, cmviewcl –v –f commands, or can be found in the system log file.

For more information regarding the Serviceguard toolkit for Integrity Virtual Machines look at:

http://h20000.www2.hp.com/bizsupport/TechSupport/SupportTaskIndex.jsp?lang=en&cc=us&taskId=101&prodClassId=10008&contentType=SupportManual&docIndexId=64255&prodTypeId=18964&prodSeriesId=5196848

[RSA]

Asghar,

In this case I'm not using the Serviceguard toolkit for Integrity Virtual Machines. Serviceguard is protecting the application in this environment, not the guests itself. The Serviceguard nodes are the actual Integrity VM guests. Is there a way to utilize the Serviceguard toolkit for Integrity Virtual Machines in my setup? I thought that's only an option if one setups the Integrity VM guests as Serviceguard packages which is not the case here.

I can confirm that the VM guest management software is installed on both HPVM guests and the version matches to the HPVM host software.

# cmviewcl -v -f line |grep io_timeout
io_timeout_extension=40000000
configured_io_timeout_extension=0

Thanks,

RSA

[asghar_62]

RSA,

An important distinction between VM as Serviceguard package and node configurations is that VM as Serviceguard node configurations only support whole disk VM backing stores. One reason for this restriction is that it is not possible to set timeouts on logical volumes or file systems presented as backing stores to the VM guest, and any errors generated from these types of backing stores are not passed through the virtualization layers from the VM host to VM guest that would allow Serviceguard running in the VM to react to these conditions. Another reason relates to disk I/O performance and the speed at which I/O requests can be completed prior to a VM node failure, which can affect cluster reformation time. Please read the document “Designing high-availability solutions with HP Serviceguard and HP Integrity Virtual Machines”. I provided the link in my previous posting.

[RSA]

Asghar,

Yes, you are right: Only whole disks are supported for the VM backing store. I'm sorry if this wasn't clear in my earlier messages. I'm using one whole disk (SAN LUN) for the guest VM operating system and one whole disk (SAN LUN) for the application. I read the document you mentioned and I'm aware of these constraints.

Thanks,

RSA

[jim_curtis]

SG does not support nested clustering where you have a cluster of VM hosts and a second cluster of guests on those hosts running packages.  The problem as you recognized is the complexity of such a configuration.  It would lead to unpredictable failover scenarios as you basically have "2 brains" trying to decide what actions to take but with no real way to communicate between these brains to coordinate actions.

[jim_curtis]

You might consider using the Generic Resource capability of SG to create your own host monitors.  But there are other implications to consider when attempting to monitor root disks that I will let somebody else comment on.

[RSA]

Jim,

That's exactly what I tried first. A generic resource monitor:

Added sg/service and sg/generic_resource, including service_fail_fast_enabled yes, to Serviceguard package configuration, SFM/WBEM monitor for critical resources using CIMUtil -e root/cimv2 HP_DiskDrive command to get vg disk status

  service_name                            sfm_disk

  service_cmd                             /etc/cmcluster/test/sample_generic_resource_disk_monitor.sh sfm_disk vg_cluster 10

  generic_resource_name                   sfm_disk

  generic_resource_evaluation_type        during_package_start

Script sample_generic_resource_disk_monitor.sh gets excuted during package start and every 10 seconds through while loop to check status. I added a couple lines to the example script to get some more details in the logs:

    if [ -z "$status" ]; then

         set_genres_status $STATUS_DOWN

         sg_log 0 "Monitor for generic resource $RES_NAME: STATUS ERROR"

         exit 1

    elif [ $status != 2 ]; then

         set_genres_status $STATUS_DOWN

         sg_log 0 "Monitor for generic resource $RES_NAME: STATUS DOWN"

         exit 1

    else

        set_genres_status $STATUS_UP

        sg_log 0 "Monitor for generic resource $RES_NAME: STATUS UP"

    fi

If OperationalStatus of CIMUtil -e root/cimv2 HP_DiskDrive command for disk in vg does not equal 2 => package dependency fails and package failover should be initiated.

Unfortunately the following happens in my environment: With disabling the backing storage for the HPVM guest containing Serviceguard the complete guest operating system seems to be "frozen". I'm not able to access this Serviceguard node anymore with SSH or anything else.

I executed cmviewcl -v on the other Serviceguard node to check the status of the package. The package itself is marked up and running. The node with the disabled backing storage stays in a "frozen" state and unfortunately the package is not failing over to the other node. Serviceguard doesn’t detect that there is something wrong with the node as well as the package. As long as the fibre channel connections are disabled the monitoring script isn’t executed or to be more clear the process is waiting for IO I guess.

Adjusting the LV and PV timeouts from the defaults might help in this case. Nevertheless it's surprising to me that Serviceguard isn't detecting the issue.

[JBK]

Hi RSA--

What *might* be happening is that the HPVM host (hypervisor) is not failing the LUN explicitly and immediately due to some inherent default timeout values within the HP-UX storage subsystem.  In other words, even though complete connectivity via FibreChannel is gone, HP-UX might have a timing budget to navigate intermittent SAN congestion/hangs . . while queuing I/O and health status.  If this is the case, those monitoring algorithms could probably be set more aggressively . . such that then the status is propogated quickly and the "cmvolmond" can take corrective action, i.e. tell cmcld to TOC the system.

We are forwarded your thread to a few more technical folk that might be able to help unravel and offer insight to the issue and proper setup to accommodate your failure scenario.  Please stay tuned . . .

Regards!

[RSA]

Hi Joseph,

I would appreciate if I could get some more details on how to set those monitoring algorithms more aggressively. It sounds like this is the direction I have to take.

Let me know if you need any additional details and I would be happy to provide them.

Best regards,

Ralph

[Singar]

Hello Ralph,

This is just to keep you informed that we "HP Pro" are investigating this issue internally.

We will let you know the results once we are done with it. Hope to come back to you within this week time.

Thanks

Singar

[RSA]

Hello Singar,

I'm looking forward to your results. Please let me know if you need any additional information on this issue.

Best regards,

Ralph

[Singar]

Hello Ralph,

Thank you for your patience on this. From serviceguard point of view, I see that cmvolmond is the best way to detect this issue. By the way, cmvolmond failure detection mechanism has been further improved thru a fix (QXCR1001227626), which was requested thru similar cable pull test. The fix is expected to be released in Sep/Oct 2012 (subject-to-change) timeframe as part of Serviceguard Patch.

In previous discussions above, you said  that when FC cable is pulled, Guest access is not possible through ssh (But, the existing processes continue to run without any issue.). So, apart from Serviceguard, HPVM configuration and it's storage failure detection mechanism might need to be checked by HPVM experts and provide fix if there is a need.

Therefore, I request you to raise official support case with HP Response Center and you may refer to this thread in there to give more context for RC. RC will reach out to corresponding experts to solve this problem.

Hope this helps.

Singar