Filesystem unmount- package stays up

[Mike Smith_33]

I have an 11i system running Serviceguard. 2 node with lock disk. In doing some testing on a test environment I unmounted one of the SG filesystems. The package did not fail over, in fact it did not seem to notice.

The syslog file did log the occurence. I unmounted one and then another filesystem in an attempt to test the failover.

Nov 19 15:21:29 gvxxxx EMS [3058]: ------ EMS Event Notification ------ Value: "MAJORWARNING (3)" for Resource: "/storage/events/d
isks/default/0_1_1_0.1.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resd
ata -R 200409095 -r /storage/events/disks/default/0_1_1_0.1.0 -n 200409090 -a
Nov 19 15:25:16 gvxxxx EMS [3058]: ------ EMS Event Notification ------ Value: "SERIOUS (4)" for Resource: "/storage/events/disks/
default/0_1_1_0.1.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -


The system was setup by a consultant, I have not had SG training so I don't have a high level of expertise. Can someone point me to where I should be looking to diagnose this. I was running the list of tests left by the consultants so this is supposed to be configured.

[Johnson Punniyalingam]

Hi Mike,

>>>I have an 11i system running Serviceguard. 2 node with lock disk. In doing some testing on a test environment I unmounted one of the SG filesystems. The package did not fail over, in fact it did not seem to notice.

The syslog file did log the occurence. I unmounted one and then another filesystem in an attempt to test the failover.<<

# more /etc/cmccluster/pkg_name/cnlt.log --> show you much information

# cmviewcl -v

>> I unmounted one of the SG filesystems. The package did not fail over, in fact it did not seem to notice.
<<

never heard of this "you can't directly unmount " file system or Logical volume which used by package-- I suspect the file system which you unmounted was not belong to the Cluster pkg" becoz "that" File system must have mounted under Cluster "should have throw error like "mount point busy"

>>Nov 19 15:21:29 gvxxxx EMS [3058]: ------ EMS Event Notification ------ Value: "MAJORWARNING (3)" for Resource: "/storage/events/d
isks/default/0_1_1_0.1.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resd
ata -R 200409095 -r /storage/events/disks/default/0_1_1_0.1.0 -n 200409090 -a
Nov 19 15:25:16 gvxxxx EMS [3058]: ------ EMS Event Notification ------ Value: "SERIOUS (4)" for Resource: "/storage/events/disks/
default/0_1_1_0.1.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -
<<

for the above you need run beloe

/opt/resmon/bin/resdata -R 200409095 -r /storage/events/disks/default/0_1_1_0.1.0 -n 200409090 -a

output needed to comment on this,

Thanks,
Johnson

[Mike Smith_33]

First off, thanks for the reply.

I am curious as to why if it is impossible to unmount an SG filesystem, why this step would be one of the tests of SG failover? I can tell you for sure the volume groups/filesystems I am referring to are mounted by the package.

I am going to clear ctnl.log file and run the test again so that I can provide output from that command.

The output from the verbose cmviewcl is here:

CLUSTER STATUS
enterprise_cluster up

NODE STATUS STATE
node1 up running

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY up 0/1/2/0 lan0
PRIMARY up 0/6/1/0/7/0 lan5
STANDBY up 0/2/1/0 lan1

PACKAGE STATUS STATE AUTO_RUN NODE
gvlsediprod up running enabled node1

Policy_Parameters:
POLICY_NAME CONFIGURED_VALUE
Failover configured_node
Failback manual

Script_Parameters:
ITEM STATUS MAX_RESTARTS RESTARTS NAME
Service up 2 0 EDIPROD
Subnet up 10.12.99.0

Node_Switching_Parameters:
NODE_TYPE STATUS SWITCHING NAME
Primary up enabled node1 (current)
Alternate up enabled node2

NODE STATUS STATE
node2 up running

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY up 0/1/2/0 lan0
STANDBY up 0/2/1/0 lan1
PRIMARY up 0/6/1/0/7/0 lan5


Also working on gathering other data.

Hi Mike,

>>>I have an 11i system running Serviceguard. 2 node with lock disk. In doing some testing on a test environment I unmounted one of the SG filesystems. The package did not fail over, in fact it did not seem to notice.

The syslog file did log the occurence. I unmounted one and then another filesystem in an attempt to test the failover.<<

# more /etc/cmccluster/pkg_name/cnlt.log --> show you much information

# cmviewcl -v

>> I unmounted one of the SG filesystems. The package did not fail over, in fact it did not seem to notice.
<<

never heard of this "you can't directly unmount " file system or Logical volume which used by package-- I suspect the file system which you unmounted was not belong to the Cluster pkg" becoz "that" File system must have mounted under Cluster "should have throw error like "mount point busy"

>>Nov 19 15:21:29 gvxxxx EMS [3058]: ------ EMS Event Notification ------ Value: "MAJORWARNING (3)" for Resource: "/storage/events/d
isks/default/0_1_1_0.1.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resd
ata -R 200409095 -r /storage/events/disks/default/0_1_1_0.1.0 -n 200409090 -a
Nov 19 15:25:16 gvxxxx EMS [3058]: ------ EMS Event Notification ------ Value: "SERIOUS (4)" for Resource: "/storage/events/disks/
default/0_1_1_0.1.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -
<<

for the above you need run beloe

/opt/resmon/bin/resdata -R 200409095 -r /storage/events/disks/default/0_1_1_0.1.0 -n 200409090 -a

output needed to comment on this,

Thanks,
Johnson

[Mike Smith_33]

Ok, I cleared the cntl.log file and verified the package and cluster were up and running. I did a umount on one of the filesystems and then did a cat on the log file. Still empty.

bdf verified filesystem not there. cmviewcl verified that cluster /package still up and running.

Nothing in syslog.log. This makes sense. I was running several SG reliability tests yesterday around the same time frame. I ran tests such us failure of primary lan, failure of 1 member of boot disk mirror, sudden disapperance of mounted filesystem, etc... The information from syslog was apparently when we pulled out one member of the boot disk mirror set, not related to the umount of the filesystem.

[john123]

Hi Mike,
MC/SG does not monitor the file systems by default unless you specifically set this up. This can be done by configuring a filesystem as a resource and having MC/SG monitor it.

Regards
John

[Mike Smith_33]

Thanks John, here is my problem. According to the consultant this was setup yet it is not working. I had no experience with SG prior to the consultant. Exactly where is this configured?

[Analyst]

Hi Mike,

The SG configuration important purpose to provide High Availability even with failover .

1.There is a concept of Cluster Lock Disk, plz confirm whether its configured properly.

2.Check policy parameter

3.Node switching parameter

4.Heartbeat configuration.

[Mike Smith_33]

Yes. Those things are fine.

[john123]

Hi Mike,
From your cmviewcl o/p it seems like u have not configured any file system monitoring service, So may have to create your own script that monitors the filesystem status and configure that as a service for the package

Regrds
John

[Mike Smith_33]

Ok, apparently even though removal of a filesystem was listed as a test I can find no evidence that this was ever configured. Apparently I have been chasing something that was not there in the first place. Thanks for the help.