package requires cleanup on node

[Ken Englander]

SG v11.19
We have a script that is intended to halt all packages on a specific node and then run cmhaltnode. Depending on the circumstances, I sometimes get error messages when the cmhaltnode command is run similar to the following.

Unable to halt the cluster: package crs_uxprf_clus requires cleanup on node hpux6.
Ensure that all package components are halted and run
cmmodpkg -e -n hpux6 crs_uxprf_clus
to allow the node to halt.
Unable to halt the cluster: package cerner_hpux5 requires cleanup on node hpux6.
Ensure that all package components are halted and run
cmmodpkg -e -n hpux6 cerner_hpux5
to allow the node to halt.

I am not able to guess why SG is selecting certain packages to enable switching, i.e. I cannot figure out the pattern for its logic.

I'm attaching a file showing the cluster status (cmviewcl -v) after the cmhaltnode command was issued. It also contains output from cmviewcl -v -f line command.

Any insights would be appreciated!

Thanks!

[Stephen Doud · Accepted solution]

Ken, Serviceguard requires cleanup of a package whose halt script has failed. Do the package logs show this happened in the course of attempting to halt these packages through the automation script activity?

Since cerner_hpux5 is dependent on crs_uxprf_clus, does your automation script halt the cerner_hpux5 package before attempting to halt the crs_uxprf_clus package?

[Ken Englander]

Hi Stephen,

That appears to be exactly right, at least that is what is unique in the output from cmviewcl -v -f line. I cannot independently verify what happened as this was at one point in multiple tests we were running on this cluster.

Can you clarify one thing, please? Is this new with SG 11.18 or 11.19. I seem to recall reading that something about the last_halt_failed being a new option AND I do not remember ever seeing this error prior to working with 11.19 (we skipped over 11.18).

Thanks!

Ken

[Stephen Doud]

I also find last_halt_ identifiers in A.11.18 'cmviewcl -v -f line' output, so I believe the framework was put in place on A.11.18. I believe the 'requires cleanup' functionality was also in place on A.11.18, because it is used when reporting problems with predecessor and successor packages in SADTA clusters.

I see that you have a case open on this, so I recommend pursueing this question through the support center.

[Ken Englander]

The case opened with HP suggests this was introduced in Serviceguard 11.18. Using cmviewcl -f line on Serviceguard 11.17 does not have a last_halt_failed in the output.

Regardless of the version, the following script will automatically do the cleanup required before executing a cmhaltnode command.

/usr/sbin/cmviewcl -v -f line | grep "last_halt_failed=yes" | \
while read L_FAILED
do
L_TEMP=${L_FAILED#package:}
L_FAILED_PKG=${L_TEMP%%\|*}
L_TEMP=${L_FAILED#*node:}
L_FAILED_NODE=${L_TEMP%%\|*}
L_TEMP=$(/usr/sbin/cmviewcl -v -f line -p $L_FAILED_PKG | \
/usr/bin/grep "node:$L_FAILED_NODE\|available")
L_AVAIL_PKG=${L_TEMP##*available=}
if [[ $L_AVAIL_PKG = "no" ]]
then
echo "Enabling package $L_FAILED_PKG on node $L_FAILED_NODE"
/usr/sbin/cmmodpkg -e -n $L_FAILED_NODE $L_FAILED_PKG
fi
done