Problem with Veritas Cluster Server's (VCS) Application Agent

[Ralph Grothe]

Hi,

before I spell out my problem I would like to find out if some of you use VCS.
Since this is an HP-UX forum I would consider usage of VCS rather unlikely , and the cluster nodes we run VCS on operate on Solaris.
However, my problem isn't OS but VCS related.

Thanks for your attention
Ralph

[RAC_1]

May be you can put it here and may be HP forum moderator will remove it. You must be knowing -- you can put it here.

http://forums.veritas.com/discussions/index.jspa

Anil

[Ralph Grothe]

Thanks Anil,

you're right, my chances of getting response will be much higher in the Veritas Forum.
Thank you for the link.
I haven't known yet that such a forum exists.

[Manish Bharadwaj]

I am using VCS. What are the issues you are having with application?

Regards,
Manish

[Ralph Grothe]

Hello Manish,

I'm glad I found someone who has VCS experience.

In our VCS cluster we mainly use Informix instances as HA resources to be monitored.
To this end we set up the cluster to make use of the extra Informix Agent supplied by Veritas.

However, because the cluster nodes are in a firewalled LAN and because we use Tivoli for general monitoring, I had to set up a Tivoli gateway in the LAN.
An already existing service group (SG) for data exchange (therefore it only contained volumes, mountpoints, nic and ip as resources) lent itself to this purposes.
After having had a quick look through the "VCS Bundled Agents" PDF I came to the conclusion that the Application agent seemed most appropriate.
However, the very terse agent's description still leaves some questions unasnwered
(the "VCS User Guide" isn't more revealing either)

Since I haven't made use of the Application Agent yet, the first thing I don't understand is the agent's NameRule attribute.
For instance if I add a new resource to an existing SG VCS tells me that agent monitor won't work until I set attributes Enabled to 1 (which is clear), and NameRule (whose purpose I don't understand)

e.g.

[root@nemesis:~]
# [[ $(haclus -value ReadOnly) != 0 ]] && haconf -makerw

[root@nemesis:~]
# hares -add test_dummy Application albani
VCS:10245:Resource added
NameRule and Enabled attributes must be set before agent monitors

[root@nemesis:~]
# hares -display test_dummy -attribute Enabled NameRule
VCS:10554:No resource exists with attribute of NameRule
#Resource Attribute System Value
test_dummy Enabled global 0

[root@nemesis:~]
# hatype -display Application -attribute NameRule
#Type Attribute Value
Application NameRule ""


As you can see, NameRule is a type and not a resource attribute.
But from the type definition of this bundled Application Agent NameRule is per default set to an empty string, which I interpret as undef.

Lets first delete test_dummy resource to tidy up the configuration

[root@nemesis:~]
# hares -delete test_dummy
[root@nemesis:~]
# haconf -dump


I extended the cluster's configuration by a resource that I named tivoli_gw.
This Tivoli Gateway daemon, named "gwproxy" is started through a start|stop script that Tivoli automatically supplies on installation.
I installed it after I had created a new VxVM volume in the already existing disk group that is a resource of SG albani, mkfs-ed a VxFS on its raw device and mounted it.

To see what commands I used, here is a current dump from the cluster config grep-ped for tivoli_gw

[root@nemesis:~]
# hacf -cftocmd /etc/VRTSvcs/conf/config -display|grep tivoli_gw
hares -add tivoli_gw Application albani
hares -modify tivoli_gw Enabled 1
hares -modify tivoli_gw Critical 0
hares -modify tivoli_gw User tivoli
hares -modify tivoli_gw StartProgram "/opt/Tivoli/proxy/gwproxy.sh start"
hares -modify tivoli_gw StopProgram "/opt/Tivoli/proxy/gwproxy.sh stop"
hares -modify tivoli_gw CleanProgram "/usr/bin/pkill -KILL -x gwproxy -u tivoli"
hares -modify tivoli_gw PidFiles "/opt/Tivoli/proxy/gwproxy.pid"
hares -modify tivoli_gw MonitorProcesses gwproxy
hares -link tivoli_gw albani_ip
hares -link tivoli_gw tivo_proxy_mnt

[Ralph Grothe]

Was interrupted, so couldn't carry on with my problem.

Problem with this configuration is that I can only online the SG where resource tivoli_gw is part of to a partial state.


[root@nemesis:~]
# hagrp -display albani -attribute State
#Group Attribute System Value
albani State nemesis |OFFLINE|
albani State themis |PARTIAL|


This shows that resource tivoli_gw has faulted

[root@nemesis:~]
# hares -display $(hares -list Group=albani|awk '$2=="themis"{print$1}') -sys themis -attribute State#Resource Attribute System Value
albani_ip State themis ONLINE
albani_nic State themis ONLINE
albanidg State themis ONLINE
datxchg_mount State themis ONLINE
datxchg_vol State themis ONLINE
dbexp_mnt State themis ONLINE
dbexp_vol State themis ONLINE
tivo_proxy_mnt State themis ONLINE
tivoli_gw State themis FAULTED
tivoli_vol State themis ONLINE


Because this resource isn't set as Critical to the whole SG

[root@nemesis:~]
# hares -display tivoli_gw -attribute Critical
#Resource Attribute System Value
tivoli_gw Critical global 0


it didn't fail the whole SG.
Let's clear the failed resource first


[root@nemesis:~]
# hares -clear tivoli_gw -sys themis

[root@nemesis:~]
# hares -state tivoli_gw
#Resource Attribute System Value
tivoli_gw State nemesis OFFLINE
tivoli_gw State themis OFFLINE


When I try to bring the resource back online again it still fails.


[root@nemesis:~]
# hares -online tivoli_gw -sys themis

This is what got written to the engine log with respect to tivoli_gw lately


[root@nemesis:~]
# grep tivoli_gw /var/VRTSvcs/log/engine_A.log |tail -5
TAG_E 2004/11/18 17:13:11 VCS:50135:User root fired command: hares -online tivoli_gw themis from 127.0.0.1
TAG_D 2004/11/18 17:14:00 (themis) VCS:13066:Agent is calling clean for resource(tivoli_gw) because the resource is not up even after online completed.
TAG_D 2004/11/18 17:14:01 (themis) VCS:13068:Resource(tivoli_gw) - clean completed successfully.
TAG_D 2004/11/18 17:14:01 (themis) VCS:13071:Resource(tivoli_gw): reached OnlineRetryLimit(0).
TAG_B 2004/11/18 17:14:01 VCS:10303:Resource tivoli_gw (Owner: unknown, Group: albani) is FAULTED (timed out) on sys themis


Have you any idea why the agent is cleaning the resource, and why it cannot bring it online?