memory full because of psmctd, srmHPUX and vxsvc processes

[Venky...]

Hi all,

Memory on one of the servers is 100%. I have found from the top command that the processes psmctd, srmHPUX and vxsvc are occupying a lot of memory.

# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 24544 9036 15508 37% 0 - 1 /dev/vx/dsk/rootdg/swapvol
reserve - 15508 -15508
memory 24545 24536 9 100%
total 49089 49080 9 100% - 0 -


System: server Tue Jul 14 10:41:40 2009
Load averages: 1.27, 1.27, 1.19
511 processes: 393 sleeping, 64 running, 54 zombies
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 1.28 94.3% 0.0% 5.7% 0.0% 0.0% 0.0% 0.0% 0.0%
1 1.23 100.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
2 1.26 37.3% 0.0% 62.7% 0.0% 0.0% 0.0% 0.0% 0.0%
3 1.30 79.0% 0.0% 21.0% 0.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 1.27 77.7% 0.0% 22.3% 0.0% 0.0% 0.0% 0.0% 0.0%

Memory: 19865456K (10000232K) real, 47314116K (17086860K) virtual, 330004K free
Page# 1/57

PU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
1 ? 3372 root 221 10 10188K 116K run 172192:11 97.63 97.46 psmctd
2 ? 4046 root 241 20 8040K 1084K run 43:33 96.75 96.58 srmHPUX
2 ? 3380 root 152 20 127M 2104K run 318917:30 96.30 96.13 vxsvc

My ques here are..
How to restart these processes and
what are the effects on the server if we restart those processes.

This server is one of the 2 node veritas cluster servers.

[Roopesh Francis_1 · Accepted solution]

Hi,

psmctd - the Peripheral Status Client/Target daemon used to monitor the state of hardware resources.you can kill this process if it is not configured with cluster.
vxsvc initializes the Domain or Local Controller, loads all registered providers, and listens to connections from clients. It maintains the client context, accepts client requests, and sends back responses and callbacks to the clients. for killing vxscc using vxsvc -k.make sure you are not using vxmvm before you kill them

please check you patch level also.

Thanks

[Ganesan R]

Hi,

For me it looks you are running out of swap not the physical memory. memory column in swapinfo output doesn't refer physical memory usage. It refers to pseudo swap. You have to increase the swap by adding additional swap.

To know the actual physical memory usage, use glance or kmeminfo tool.

However your server is having CPU bottleneck. Three processes which you have specified consumes most cpu cycles. You have to find out the reason why it consumes more cpu resources. If these process really needs that much cpu, then upgrade cpu's.

[Venky...]

Hi,
Can I try restarting the processes psmctd and vxsvc using the below methods:
Will it effect the system?

For psmctd:
===========
/sbin/init.d/ems stop
/sbin/init.d/diagnostic stop

# ps -ef| grep psmctd
# ps -ef| grep diag

# /sbin/init.d/diagnostic start
# /sbin/init.d/ems start

# ps -ef| grep psmctd
# ps -ef| grep diag

For vxsvc:
=========
1. Kill or make sure that the vxsvc process is not running:
# ps -ef | grep vxsvc

If the process is running, do the following:
/opt/VRTSob/bin/vxsvc -k

If, by doing the above command, you encounter errors such as the following:
# ./vxsvc -k
Request is in process..
Server was not running

This means that the kill script did not find the PID of vxsvc. The vxsvc process should be killed:
# ps -ef | grep vxsvc
root 520 1 0 13:09:28 ? 1:50 /opt/VRTSob/bin/vxsvc -r /opt/VRTSob/config/Registry
# kill 520

If the following error occurs, kill the vxsvc process and remove the /var/vx/isis/vxisis.lock file, then proceed to Step #2.
vxsvc: Another instance of vxsvc seems to be running.
Please verify with the command ps(1m).
If no other instance is running, remove the file:
/var/vx/isis/vxisis.lock
And restart vxsvc

2. To restart the isisd and vxsvc, run the following:
/etc/init.d/isisd start

Note: Stopping and restarting isisd or vxsvc will not affect a live running system.

[Venky...]

Frans,

Can you tell me how to check if the service psmctd is being used by the cluster?

[Ganesan R]

Hi Venky,

psmctd is launched at boot by the diagmond daemon. So you can stop and restart the diagnostic daemon.

/sbin/init.d/diagnostic stop
/sbin/init.d/diagnostic start

I have seen in some threats and also true that this kind of issue should be resolved by upgrading the support tool manager. This threat say that..

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1228749

[Ganesan R]

Hi Again,

You can download the latest version of STM here..

http://h20293.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=B6191AAE

[Dennis Handly]

You are using 48 Gb of swap. You shouldn't be worried about a trivial amount of swap for those 3 processes: 10 M, 8 M 127 M.
But they are using lots of CPU.

>Ganesan: it looks you are running out of swap not the physical memory.

They are running out of both.