Continuous Webes Error mail

[Ivan Rodriguez Sanchez]

Hi all,

A customer has IRSS 05.50 and Webes 6.1 installd, I checked that trap tests reach AMC portal, but the customer is continuously receiving emails like this:

Webes Health Check reported a problem at Mon, 27 Jun 2011 03:05:00 +0200

Hostname: HPMonitor.l-h.es

Version: Web-Based Enterprise Services Common Components for Windows V6.1 (Build 136), member of Web-Based Enterprise Services Suite for Windows V6.1 (Build 136)

Error Code: E2E

Description:

Webes failed to analyze a HeartBeat Event and log a problem report. Webes may not be in a running state.

CallStack:

Recommended Action:

Stop and restart Desta_service. If this problem persists then run "desta sysinfo" on the command line and give the generated Zip file to HP Customer Support.

He has restarted the DESTA service several times, but nothing has changed.

Does anyone know how to resolve this?

Thanks!

[BPE]

You should get the message every 8 hour, if you have real problem. Check with the desta status command if desta_service is realy running.

If your IRSS  is  running on a vm it could be a configuration problem.

Or  your the eventlog gets to many Events at a time.

[Ivan Rodriguez Sanchez]

Thanks for the reply BPE.

I have confirmed that the desta service is running by the command 'desta status', the Host Device is a physical server, so there is no configuration problem about VM.

I saw at the Scheduled Tasks, that the tasks called AtX, that perform the healthcheck, are executed at 6 2 am and 10am, and at the eventviewer, at those times I can see an informational application event whose description is:

Event Type: Information
Event Source: SEA_TEST
Event Category: None
Event ID: 999
Date:  6/29/2011
Time:  10:00:00 AM
User:  N/A
Computer: HPMONITOR
Description:
The description for Event ID ( 999 ) in Source ( SEA_TEST ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: ??Ǵ.

Is this event related to the error mail the customer is receiving? In that case, how can I resolve this?

Thanks!

PD: Here is the result of executing "desta status" and "desta info"

DESTA STATUS:

Checking for DESTA director with PID 4996
The Director process is running.

DESTA INFO:

Web-Based Enterprise Services Common Components for Windows V6.1 (Build 136), me
mber of Web-Based Enterprise Services Suite for Windows V6.1 (Build 136)
Service Tools Home: C:\Program Files\HP\svctools
Service Obligation Start Date: Thu Jun 09 15:02:42 CEST 2011
Service Obligation End Date: Fri Jun 08 15:02:42 CEST 2012
HPSIM notification: disabled.
ISEE notification: enabled.
MOM notification: disabled.
SNMP notification: disabled.
DTCS notification: disabled.
SMTP notification: enabled.
LOG notification: enabled.
INDICTMENT notification: disabled.
VMSMAIL notification: disabled.

PSP 8.70 is installed in the Host Device.

Thanks!!

[BPE]

Yes, the event is related to the webes healthcheck which runs every 8 hours.

It is normal that you will get the

The description for Event ID ( 999 ) in Source ( SEA_TEST ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: ??Ǵ.

information because WIndows has not enough information to display the details of the binary data inside the event.

Are you really sure that your IRSS Server is able to log a HW call at HP.

You can run a simple test.

"wccproxy test" from the cmd

you should get a wccproxy_test event and some time later RSclient events that the data are submitted to HP.

You can also check the progress inside SEA System event analyzer under the notification button.

[Ivan Rodriguez Sanchez]

 have checked the wccproxy test, and the event appears at Webes, and it also appears at AMC website... but I still receive the healthcheck error mails...

Any other suggestion?

Thanks!

[BPE]

UAC could be the issue check the username for the desta_service and the user under which the at webeshl command is executed. wccproxy test  is writing the test event into the Windows eventlog wccproxy service  is reading the event log and desta_service is collection the new entry from wccproxy. I donot remember how the at command svctools desta.bat launchwhc is checking the status for a successfull transport to HP. If launchwhc is not able to read some logs you could be in troubles. check the svctools\specific\desta\logs for more details.

If you get an UAC message browsing to the files this could be the issue.

Did you realy run the wccporxy test from the IRSS?

The AMC entry can be confusing. Check in the SEA the ISEE status for closed at HP to be sure that the webes healthcheck did not discover a Pproxy filter or Firewall issue. If you see a status  pending to other  followed by other to other  you will ave a transport issue.

[shocko]

I have the same issue. Did you ever get this resolved?

[Brad Cunningham]

I would consider upgrading to 5.70 which includes WEBES 6.3 and many of the E2E issues have been resolved.

Since the issues are happening every 8 hours this indicates that WEBES is unable to process it internal self-test. I would bet that WEBES is still running fine and the message is a false indication.

HP Web Based Enterprise Services Software (WEBES) will occasionally generate an End-to End (E2E) failure notification when it encounters that the Microsoft Windows Application event file is not writing new events. The WEBES product will try to write a Heartbeat event into the Application.evt file. It then tries to read and process this event. If it does not find the newly sent event, it will generate an E2E notification.

Use the following command to quickly determine if you have current events in the .evt file:

DOS>wsea sum all

You can also review the time line of events using the Web User Interface.

There are several WEBES logs that you can review to spot the footprint:

• WEBES.x.log
• Prob.log
• destaService.log

One solution is to either clear the Microsoft Windows Application event file or reboot the system. If after clearing the events they are still not being written, there may be a more serious Windows-based issue.

Also check the “log” setting in WEBES. To dso this run wsea status and check the line “LOG notification:”. If it shows disable run c:\wsea log prob on. This issues was common with WEBES 5.5

If the CMS is a virtual server modify the attributes in SIM under edit system properties page. Change the sub type to Proliant and Product Model to Proliant. Open a command line and run net stop desta_service. After it has stop run net start desta_service. Eventually the CMS should show as a managed entity in WEBES.  Open SEA and clink on the Default Group in the left hand frame . Use the drop down and add the CMS to the default group. If this CMS is virtual there is nothing in the default group listed.

There also a way of turning off the heartbeat that generates this but if this happened you may miss a E2E when desta has really died

Again there are multiple things that can cause this “even an entity can cause this”. I’ve list just a few of the common things. I would strongly consider upgrading SIM to at least 6.3 and RSA to 5.70 and hopefully the problem will be resolved.

[shocko]

Great info Brad!

I'm running SIM 7.1 and have been for some time. My IRS version is 'HP Insight Remote Support A.05.70'. When I run the check against the windows app log I get this:

>wsea sum all

================= AppEvent.evt =================

   Qty     Type   Description
 -----    -----   -------------------------------------
    20      999   SEA Test Event
  2049      998   SEA HeartBeat Event
     1      997   SEA Test Event from ELMC
   599      996   SEA Discovered Fault
 57194        0   NT Event

Total entry count : 59863
First entry date  : Thu 15 Sep 2011 22:32:15 GMT+00:00
Last  entry date  : Tue 28 Aug 2012 17:09:49 GMT+00:00

And indeed I can see events with source SEA_TEST in the application event log. My status is as follows:

>wsea status
System Event Analyzer for Windows V6.3 (Build 137)
Service Tools Home: D:\Program Files (x86)\HP\svctools
Service Obligation Start Date: Mon Oct 24 10:42:20 BST 2011
Service Obligation End Date:   Tue Oct 23 10:42:20 BST 2012
HPSIM notification: enabled.
ISEE notification: enabled.
SMTP notification: disabled.
DTCS notification: disabled.
MOM notification: disabled.
SNMP notification: disabled.
LOG notification: enabled.
INDICTMENT notification: disabled.
EVT notification: enabled.
VMSMAIL notification: disabled.

The E2E check seems to be run via a scheduled task that runs this command:

• cmd /c "D:\PROGRA~2\HP\svctools\common\bin\desta.bat launchwhc "

I have started a java de-compiler to try and see exactly what the JVM is doing when it launches/runs via this command file. This has only started last week since we had an unexpected powerdown.

[shocko]

I got a resolution from HP Support. It turns out that when my Application Evtn log (i.e. the windows event log) grew to over 16MB, the E2E failed. It seems like the E2E process reads through the event log and has an issue when it's over 16MB. SOlution was to clear it down and restart the desta services.