system_multi_node is low performance when running at the same time

[LeThien]

Hi,

I have issue with my Serviceguar cluster of 2 node. When 2 node run at the same time but just only node was used by users and the other is not configured for users use it, the performance of the disk is very high ( ~~ 80% - 100% ) but when i halt one of them , it is work fine. 
Can you explain it for me ? 
Thank you !
This is config file and status :

Config 2 Node : 

PACKAGE_NAME SG-CFS-pkg

PACKAGE_TYPE SYSTEM_MULTI_NODE

FAILOVER_POLICY CONFIGURED_NODE

FAILBACK_POLICY AUTOMATIC

NODE_NAME *

AUTO_RUN YES

LOCAL_LAN_FAILOVER_ALLOWED YES

NODE_FAIL_FAST_ENABLED YES

RUN_SCRIPT /etc/cmcluster/cfs/SG-CFS-pkg.sh
RUN_SCRIPT_TIMEOUT 300
HALT_SCRIPT /etc/cmcluster/cfs/SG-CFS-pkg.sh
HALT_SCRIPT_TIMEOUT 300
SCRIPT_LOG_FILE /etc/cmcluster/cfs/SG-CFS-pkg.log

# vx monitor of vxconfigd daemon

SERVICE_NAME SG-CFS-vxconfigd
SERVICE_FAIL_FAST_ENABLED yes
SERVICE_HALT_TIMEOUT 5

# Serviceguard configuration monitor

SERVICE_NAME SG-CFS-sgcvmd
SERVICE_FAIL_FAST_ENABLED yes
SERVICE_HALT_TIMEOUT 5

# Filesystem check daemon monitor

SERVICE_NAME SG-CFS-vxfsckd
SERVICE_FAIL_FAST_ENABLED yes
SERVICE_HALT_TIMEOUT 30

# vx membership coordination daemon

SERVICE_NAME SG-CFS-cmvxd
SERVICE_FAIL_FAST_ENABLED yes
SERVICE_HALT_TIMEOUT 5

# vx ping daemon

SERVICE_NAME SG-CFS-cmvxpingd
SERVICE_FAIL_FAST_ENABLED yes
SERVICE_HALT_TIMEOUT 5

MULTI_NODE_PACKAGES

PACKAGE STATUS STATE AUTO_RUN SYSTEM
SG-CFS-pkg up (1/2) running enabled yes

NODE_NAME STATUS SWITCHING
app1 down

Script_Parameters:
ITEM STATUS MAX_RESTARTS RESTARTS NAME
Service down Unlimited 0 SG-CFS-vxconfigd
Service down Unlimited 0 SG-CFS-sgcvmd
Service down Unlimited 0 SG-CFS-vxfsckd
Service down Unlimited 0 SG-CFS-cmvxd
Service down Unlimited 0 SG-CFS-cmvxpingd

NODE_NAME STATUS SWITCHING
app2 up enabled

Script_Parameters:
ITEM STATUS MAX_RESTARTS RESTARTS NAME
Service up 0 0 SG-CFS-vxconfigd
Service up 5 0 SG-CFS-sgcvmd
Service up 5 0 SG-CFS-vxfsckd
Service up 0 0 SG-CFS-cmvxd
Service up 0 0 SG-CFS-cmvxpingd

Other_Attributes:
ATTRIBUTE_NAME ATTRIBUTE_VALUE
Style legacy
Priority no_priority

PACKAGE STATUS STATE AUTO_RUN SYSTEM
SG-CFS-DG-1 up (1/2) running enabled no

NODE_NAME STATUS STATE SWITCHING
app1 down unknown

Dependency_Parameters:
DEPENDENCY_NAME SATISFIED
SG-CFS-pkg no

NODE_NAME STATUS STATE SWITCHING
app2 up running enabled

Dependency_Parameters:
DEPENDENCY_NAME SATISFIED
SG-CFS-pkg yes

Other_Attributes:
ATTRIBUTE_NAME ATTRIBUTE_VALUE
Style legacy
Priority no_priority

PACKAGE STATUS STATE AUTO_RUN SYSTEM
SG-CFS-MP-1 up (1/2) running enabled no

NODE_NAME STATUS STATE SWITCHING
app1 down unknown

Dependency_Parameters:
DEPENDENCY_NAME SATISFIED
SG-CFS-DG-1 no

NODE_NAME STATUS STATE SWITCHING
app2 up running enabled

Dependency_Parameters:
DEPENDENCY_NAME SATISFIED
SG-CFS-DG-1 yes

Other_Attributes:
ATTRIBUTE_NAME ATTRIBUTE_VALUE
Style legacy
Priority no_priority

[Bill Hassell]

ServiceGuard doesn't perform disk I/O. It is process and resource coordinator. The disk load is coming from processes that are running on each system. Use Glance to determine which processes are using the most disk I/O. Use the o command in Glance to sort the processes by disk.

[LeThien]

I monitored them with glance but everything is nomal when i running in one node. But when i run two node at the same time, the disk i/o is so high and very slow

[Bill Hassell]

What are the names of the processes that use the most disk I/O?

[Mike_Chisholm]

You might consider posting this over in the HP-UX System Adminstration community https://community.hpe.com/t5/System-Administration/bd-p/itrc-156#.Wb_4fdVSxaQ as well since this is very unlikely to be an actual Serviceguard problem. Serviceguard is most likely just the trigger that activates the CFS on multiple nodes, which is the point where you actually begin to have IO performance problems.

I would check to make sure you are up to date on the relevant HP-UX kernel CFS and CVM VRTS type patches for your operating system (PHKL_XXXXX. Also, it is proper to ask "What changed?" Was a new filesystem added? Mount options changed? New node added to the cluster? Patches? Other?

[Matti_Kurkela]

Does your application frequently update the same file, or even the same blocks of the same file, from both nodes? This kind of a "hot block" or "hot file" is one of the trickiest cases for a cluster filesystem.

If the exactness of the modification times of the files is not important to you, mounting the filesystem with the nomtime option might be helpful. This reduces the level of inter-node coordination the cluster filesystem needs to do, at the price that the file modification timestamps seen on one node may not accurately reflect that the other node has modified the data, until at least 60 seconds has passed from the modification.