topic Re: [SIM 6.1] WCCAgents process memory usage in Insight Remote Support

[SIM 6.1] WCCAgents process memory usage

Danilo B — Tue, 14 Sep 2010 09:52:50 GMT

Hi all,

Is it fine that WCCAgents.exe process uses about 700 MB or ram? What about this process? Operating System is Windows Server 2003 x86 Standard with SQL Server Express 2008. SIM version 6.1 is monitoring about 300 objects.
MXDomainmgr process is using about 700 MB of ram too.

Thank you all for help.

Re: [SIM 6.1] WCCAgents process memory usage

BPE — Tue, 14 Sep 2010 14:42:13 GMT

6-8 Mb peak memory is normal at our installation. But perhaps you have a problem with a big windows event log.
Are you sure that your webes is working correct? Open your webes interface at https://your_server:7906
user admin and check the notification.

I guess you have HW service contracts with 300 objects. You should open an HP Insight remote support call at HP over your HW contract.
select HW repair -> HW Type: Remote Support / Tools inside itrc call manager.

Re: [SIM 6.1] WCCAgents process memory usage

Danilo B — Tue, 14 Sep 2010 15:23:32 GMT

Hi bpe,

thank you for help. Eventlog is clear, notification is 'on' for HP SIM HP Remote Support and Email, and they are all 'green'. Other notifications are off.

Any idea?

Regards, Danilo

Re: [SIM 6.1] WCCAgents process memory usage

BPE — Tue, 14 Sep 2010 22:31:04 GMT

Have you tested your installation with wccproxy test? You should get an email and a service event in HP SIM. If you have a big delay (greater an hour) until you see the service event something is wrong with the eventlog handling.

If you get the service event in less than an 15 minutes. You should try a restart of the wccproxy service an check the memory consume over time with the task manager or process explorer.
e.g. I would check the lower plane in the process explorer for possible hints(many handles are generated but never closed).

Do you have devices which are behind firewalls? Perhaps some parts of the communication are blocked and lead to memory allocation until the timeouts happens.

Because mxdomainmgr has a similar behavior
you can perhaps also find some hints in the SIM logfiles.

Re: [SIM 6.1] WCCAgents process memory usage

Danilo B — Wed, 15 Sep 2010 08:55:55 GMT

wccproxy test did not result in any email or service events in HP SIM in an hour. So I restarted the wccproxy service. After restart WCCAgents process was about 5 MB slowly increasing by about 4KB each step.

Most of the devices are behind one or two firewalls but traffic is allowed by any-any rules from/to HP SIM server.

Thank you for support!

Re: [SIM 6.1] WCCAgents process memory usage

BPE — Wed, 15 Sep 2010 18:01:04 GMT

Today I saw a very similar behavior with windows 2008 R2 and windows firewall activated.
The firewall blocked the name resolution inside the WEBES desta_service and could not connect to the WCCAgent. The Agent was working normal but the data could not be passed to WEBES. This could be an explanation for memory consummation. This is also problem if you have name resolution problems. But you should find indication for the problems in your svctools\spesific\desta and webes \logs directory files.

Re: [SIM 6.1] WCCAgents process memory usage

BPE — Wed, 15 Sep 2010 18:01:35 GMT

Re: [SIM 6.1] WCCAgents process memory usage

Danilo B — Thu, 16 Sep 2010 07:17:53 GMT

Hi,

desta service log (P:\Program Files\HP\svctools\specific\desta\logs\HealthCheck.log) report many errors like this:

---------- Webes HealthCheck Problem Reported at gio, 16 set 2010 07:05:04 +0200 ----------

Subject: Webes HealthCheck End-To-End Detection Failed

Webes Health Check reported a problem at gio, 16 set 2010 07:05:04 +0200

Hostname: vsim-01.bitfarm.pri
Version: Web-Based Enterprise Services Common Components for Windows V6.1 (Build 136), member of Web-Based Enterprise Services Suite for Windows V6.1 (Build 136)
Error Code: E2E

Description:

Webes failed to analyze a HeartBeat Event and log a problem report. Webes may not be in a running state.

CallStack:

Recommended Action:

Stop and restart Desta_service. If this problem persists then run "desta sysinfo" on the command line and give the generated Zip file to HP Customer Support.

Thank you for help!

Re: [SIM 6.1] WCCAgents process memory usage

Danilo B — Thu, 16 Sep 2010 07:24:21 GMT

Hi,

please note Operating System is Windows Server 2003 Standard x86 with firewall disabed.

Regards, Danilo

Re: [SIM 6.1] WCCAgents process memory usage

BPE — Mon, 20 Sep 2010 08:00:09 GMT

You can also try "desta status" to see if the desta service is still running. Perhaps the installation of postgresql failed. Postgresql is included with WEBES since 5.6. If the database connection is not possible desta_service will stop after some time.
This would also explain why your wccproxy is increasing. Desta_service / WEBES will be not able to read the logs from wccproxy because the initialization never ended.

Re: [SIM 6.1] WCCAgents process memory usage

Danilo B — Mon, 20 Sep 2010 08:41:13 GMT

Hi,

desta status reports:

Checking for DESTA director with PID 6708
The Director process is running.

Windows Application Eventlog reports some errors related to PostgreSQL:

Source: PostgreSQL
EventID: 0
pg_ctl: another server might be running; trying to start server anyway

Source: PostgreSQL
EventID: 0
pg_ctl: could not start server
Examine the log output.

How can I fix it?

Thank you for help!

Re: [SIM 6.1] WCCAgents process memory usage

BPE — Mon, 20 Sep 2010 14:36:34 GMT

This is not an easy task to fix.
If you have a contract I would open a HW Remote Support case at HP. (Best option because the problem should be known in the mean time). If you like to fix it your self, you can try to install postgresql manually and and configure it the way WEBES wants to see it. The documentation for the requirements are inside the webes / postgresql directory. It is a html file. *postgresql*.htm

As soon as you found a way to run postgresql on your system you can create the DB as dokumented and restart WEBES again.
But please use the offical way because this manual method can not guaranty a future automatic update.

Re: [SIM 6.1] WCCAgents process memory usage

Danilo B — Tue, 21 Sep 2010 10:13:51 GMT

Hi,

I uninstalled Postgres and then reinstalled following instruction in P:\Program Files\HP\Installers\WEBES\PostgreSQL.htm

Installation:

postgresql.exe --mode unattended --superpassword Webe$rules8552 --serverport 7950

Now, service is running and listening on port 7950 but I can't logon!

PostgreSQL.htm says:

If the PostgresSQL 8.3 has successfully been installed, you can now create the Webes Role and Webes databases. Bring up a psql command line window by clicking on psql.exe in C:\Program Files\PostgreSQL\8.3\bin folder.

This is not right because psql uses default port 5432. So I used:

psql -U postgres -p 7950

and password Webe$rules8552

but then it returns this error:

psql: FATAL: password authentication failed for user "postgres"

Thank you for help!

Re: [SIM 6.1] WCCAgents process memory usage

BPE — Tue, 21 Sep 2010 12:47:36 GMT

try netstat -ab to get a list with the ports and the program which are using the ports.

Re: [SIM 6.1] WCCAgents process memory usage

Danilo B — Tue, 21 Sep 2010 13:19:52 GMT

Hi,

well, I see there is another postres.exe process listening on 7950! This is not running as a service and it runs under SYSTEM account!!

Using ProcessExplorer I was able to find his path:

P:\Program Files\HP\svctools\common\database\bin\postgres.exe

and data path is:

P:\Program Files\HP\svctools\specific\desta\database\data\

What does this mean? I'm confused!

Please note this server also have HP SIM 6.1 installed (previously 6.0 upgraded to 6.1)..

History of this server is as follows:

0. Clean installation of Operating System
1. Installed HP SIM 6.0
2. Installed RSSWM
3. Upgraded to HP SIM 6.1

Thank you for help

Re: [SIM 6.1] WCCAgents process memory usage

Danilo B — Thu, 23 Sep 2010 09:21:56 GMT

Hi,

I think we are near the solution.

First of all, I uninstalled WEBES 6.1 using RSSWM interface, then rebooted, checked that there was no process listening on port 7950. Then reinstalled WEBES 6.1.

WCCAgents memory requirements still grows by 4KB steps. But I found this error in WEBES.0.log file:

INFO: 2010-09-23 12:07:47.747 +0200
Thread: Thread-37: com.compaq.svctools.desta.storage.services.monitor.DNSlookupManager DNSlookupManager:

DNSlookupManager: queued lookup job for negative address Eva4400

______________________________.
INFO: 2010-09-23 12:07:47.747 +0200
Thread: Thread-37: com.compaq.svctools.desta.storage.services.monitor.DNSlookupManager DNSlookupManager:

DNSlookupManager: queued lookup job for negative address 10.168.15.180

______________________________.
INFO: 2010-09-23 12:07:50.013 +0200
Thread: pool-1-thread-3

DNSlookup: name lookup failed to find an IP address for host Eva4400
java.net.UnknownHostException: Eva4400
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:849)
at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1200)
at java.net.InetAddress.getAllByName0(InetAddress.java:1153)
at java.net.InetAddress.getAllByName(InetAddress.java:1083)
at java.net.InetAddress.getAllByName(InetAddress.java:1019)
at java.net.InetAddress.getByName(InetAddress.java:969)
at com.compaq.svctools.desta.storage.services.monitor.DNSlookupJob.run(DNSlookupJob.java:83)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)

This is a name resolution problem. Now, EVA4400 in not an host itself, but is of course a storage attached to another server that has CommandView. This server is present in DNS. So If I would add an entry for EVA4400 into DNS which IP address I'll use?

Thank you again for help!!

Re: [SIM 6.1] WCCAgents process memory usage

BPE — Mon, 27 Sep 2010 17:38:22 GMT

Your DNS Error message could be a reason for the memory leak. You should verify that your Storage is configured the right way inside WEBES. Do you see your your storage on the on the left side in System Event Analyzer SEA?

Re: [SIM 6.1] WCCAgents process memory usage

Danilo B — Thu, 30 Sep 2010 11:56:48 GMT

On the left the tree is empty. I have only 'Default Group' and it is empty.

Re: [SIM 6.1] WCCAgents process memory usage

Danilo B — Thu, 30 Sep 2010 12:18:22 GMT

Sorry this GUI is not so clear. I see I can add nodes to the tree but EVA4400 is missing so I can't add it. Does this mean that it cannot be resolved by DNS?

Re: [SIM 6.1] WCCAgents process memory usage

BPE — Thu, 30 Sep 2010 14:35:25 GMT

If you add the system which is running Command View EVA to the default Group you should see after some time your EVA's in the tree on the left.

You should see your EVA if you click on the notebook with pencil icon in the right hand group of icons. If you have orange highlited entries some data are missing. You can click on view full list if you do not see all systems.