Multiple Email messages - where to start troubleshooting?

[Mike Kanakos_1]

I have HP SIM running on a ML570 server (4 CPU's) and montioring 300 servers. I am receiving multiple bogus pages (dates of insight events are from april and may).

Where do I start troubleshooting?

Already checked SQLServer.exe in services and it is not at a high CPU level. But pages are being genreated now.

Any ideas?

[Dan Lynch]

I've had that problem and it's driving me nuts. I haven't found a root cause, but it was suggested by someone else in the forum to assign all events to a 'workaround' user, then filter out that user in the notification task. This has worked for me without incident thusfar. I have spent hours troubleshooting this and am now resigned to using the fix...
I did get this in the sql event log after this happened:

2004-05-28 01:44:51.97 spid10 WARNING: EC 56df23c0, 0 waited 300 sec. on latch 4e1b5db0. Not a BUF latch.
2004-05-28 01:44:51.97 spid10 Waiting for type 0x4, current count 0xa, current owning EC 0x271FD538.
2004-05-28 01:45:00.26 spid82 Time out occurred while waiting for buffer latch type 2,bp 0x170ac80, page 1:7748), stat 0xb, object ID 17:2:0, EC 0x62BAB538 : 0, waittime 300. Not continuing to wait.
2004-05-28 01:45:00.26 spid82 Waiting for type 0x2, current count 0x80002a, current owning EC 0x56DF23C0.

[Mike Kanakos_1]

I spoke with HP support today and they feel that the problem has to do with old events in the log. They suggested that I create a task to delete the old events after XX days.

Since I am on 4.0 and I am runnign MSDE on thr same box, we decided that since it's broke, let's start fresh.

I will be installing 4.1 tonight.

PS - HP says the problems still exists in 4.1 and th eonly fix is to delete old events. I'll let the group know how i make out.

[Jeff Westwood]

Do you use Microsoft Exchange?

We had similar problems that mysteriously stopped after we upgraded to Exchange Server 2003.

Jeff

[Mike Kanakos_1]

No exchange... Notes..

[Mike Kanakos_1]

Just here to give everyone an update. This issue is still live for me.

I am working with HP senior level support. They have been able to reproduce the problem from debug on my server.

They are writing some new code to patch the system which I guess would be included in an upcoming build of HP SIM.

This problem happened for me in SIM 4.0 and 4.1.

[Jeff_335]

Has there been an update to this? We have the same problem but we also get multiple pages. A pager is annoying enough with out getting 100 some irrelevant pages.

[Rob Buxton]

It would be interesting to know what the conditions are that trigger this.
I've not seen it, so I'm wondering what's different between the sites that are and those that are not.

[Jeff_335]

We are running SIM 4.1 with Windows 2K and SQL 2k. Our development and stage servers alert through email and production servers page us. The automated tasks are based on the Server Role field. The times where we have been inundated with emails or pages, much more annoying, are as follows.

1) A server's role changes. If a server was dev and changes to production we get paged with every non-informational event (we only receive critical, major, and minor) that SIM has for that server in the database. Same happens with email if it was listed as production but is down graded to dev or stage.

2) The rules for the automated task change. If someone is added to the list for alerting they receive all the alerts the database has for all the servers that match that rule. Example, a new person joins the group and needs to be notified if something happens will receive all the notifications on all the servers that match that rule, prod or dev.

Itâ s as though SIM reevaluates the rule against the database and decides that it missed sending out a bunch of alerts so it sends them at that time. Itâ s extremely costly, and more annoying than you can imagine, to get paged on every event for every production server since SIM was up and running. We have no problems as long as the alerts are current and relevant, but when ever a change is made SIM is trying to send out over 100 alerts for a server that took place a month

[Mike Kanakos_1]

There is an update to this issue... I posted a response to a question that was similar to this one.. here's the link to the thread:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=687690

Quick answer is that HP was eventually able to reproduce problem in lab. They wrote a software patch for Java. the patch was put in place about 2 weeks ago and so far everything seems fine. I would guess that patch will make it into a future release of SIM.

Some action would trigger a problem (like a change to one of the Auto Event handlers properties or disabling an alert group). Server events would also trigger the alerts as well. In the middle of the night, the server would go nuts. Somehow SIM would get caught in a loop and start sending out old alerts.

To clarify, admin would change a SIM setting or an event would happen (like server unreachable) and the software would get caught in loop and send out barrage of alerts. Alert dates went back as far as two months. We would get anywhere from 150 - 1000 alerts that forwarded to about 15 different two way pagers the admin staff uses. Do the quick math and you'll realize how annoying, frustrating & expensive it was. Only fix was to shut down SIM service.

[Alan Doran]

Mike,

Do you know where I could get this patch?
4,000 + e-mails every night is driving me crazy.

Any help greatly appreciated.

Thanks
Alan.

[Mike Kanakos_1]

Alan;

The absolute first thing that you must do is open a case with HP. Let them work with you for a little bit but ask to escalate it.

The patch I was given is a beta and who knows if it will work the same on your system.

Let HP make sure that this is the same issue first. The HP support team was very helpful. The person I dealt with posts on this board all the time.

I am sure if you open a case and it gets escalated. You will get it resolved.

Keep posting here with details. I would like to know what is happening.

[Guido Koetter]

Hello Mike,

is there a chance to post this patch? I opened a case here in germany several weeks ago where the support was only a redirection to the irish (european?) support-center. It did'nt help.

Greetings

Guido

[Alan Doran]

Mike,

Have opened a case with HP. It has been escalated.

I will keep updating this thread as I find out more.

Rgds
Al.

[Mike Kanakos_1]

Guido;

I can't post the patch here because of size limiltations.

Email me
michael.kanakos@olympus.com

We'll talk seperately.

[Mike Kanakos_1]

Gentlemen;

I am glad to see a buzz generated from my posts. However, I am getting some people emailing me directly, hoping to get the patch.

I am more than happy to help - that's why i post here; but let's be smart about this.

If this is a widespread issue, HP needs to know about it. I urge you to call HP and open cases. In the past I have referenced the threads on this forum to help HP techs with acknoweldgement of issues.

The HP techs were very helpful with me and I think now that more people are calling on related issues, the turnaround time to get a fix should be much shorter than what i had to go through.

Please - follow the process... it frustrating, but it makes the product better for everyone.

[Jan Gunnar Heistad]

Hi All

I assume this is the patch you are looking for:
http://h18007.www1.hp.com/support/files/server/us/download/21987.html

I have one of my customer who is going to install it within the end of the week. So I have no confirmation if this works or not but I assume this will fix the problem with e-mail spam. I will update this tread when I have got some feedback from my customer.

Best Regards
JGH
HP EMEA ISS Competency Center

[Alan Doran]

Hello all,

Recieved the patch from HP last week. It's a bug in HP SIM 4.1 & will be included in V4.2:

"If an HP Systems Insight Manager (HP SIM) Automatic Event Handling task is configured to Send e-mail or Send page as the action on event and the task is subsequently Disabled and then Enabled, all the previous (old) events are resent. This could potentially amount to hundreds of e-mails and/or pager messages being sent to the designated recipients (an e-mail / pager message "storm").
Note: This may also occur (intermittently) when an Automatic Event Handling Task is edited. "

Installed patch on friday & am in the process of testing. So far, so good. No mail storm yet. I will continue testing over the next few days & update with results.

Rgds
Al.

[Lobmeyer]

This problem still exists in 4.2. I just went through several iterations of email storms. The first storm was 1 mailing of the historic 65000 events. The second was 4 mailings of the 65,000 events. This was not fun. It dismounted my stores because it blew out the disk space for the logs and everything else on my exchange 2K3 server. Yesterday was a fun morning. Extending drives so that I could mount the stores so that I could do backups to reduce the log files and then start deleting messages.

Then you get stuck with the outlook behavoir of limiting you to 4000 emails per delete. When you have 240,000+ emails in your inbox this takes a very long time!