Cleaning queued traps in HP SIM

[pgarr]

Hey Folks-

I had a BES server that was blasting traps, several every minute, to SIM. This was causing SIM to hang up, when processing events. Event handling rules, emails, would not fire until well after the trap was received. In working with HP, they advised I upgrade to 6.2 which I did and that solved the problem. During this time I also killed the BES snmp service. However now under SIM 6.2, it still thinks or is processing traps from the BES server however I've confirmed that the BES server isn't actually sending traps. I believe that probalbly thousands of events are queued up within SIMs db and its processing them.

I contacted HP and they suggested rebooting, which I did but that does now work. They then stated that my BES server is sending traps even though I showed them firewall and also SNMPUTIL trap monitor tool on the server that shoes no traps are being sent from the server.

I'm thinking there is a table or something that I need to clear. Anyone have any similiar experience or know the db structure/queue handling?

[Ali]

Hi There,

you can create/schedule a task to delete these event/trap from the queue.

If server is not sending trap but you are recieving it then its something to do with SNMP.

there are two services: SNMP Service and SNMP Trap. SNMP Service is common and handles sending traps. In addition HPSIM will need the SNMP Trap service to handle incoming traps.

Stop SNMP Trap service
You can try reinstalling SNMP, ensuring both services are installed.

thanks,
Aftab

[pgarr]

I tried removing the snmp feature and re-adding under 2008.

However I still get phantom events. I believe these are stored in the db or on disk within the HP SIM installation directory some where and are being processed.

[Ali]

Hi there,

what kind of trap are you getting?

any specific event like disk threshold, memory processor etc...?

thanks,
Aftab

[pgarr]

As far as hpsim is concerned they are generic unregistered.

They're just informational traps from the BES software. Due to the volume, literally upto 4 traps every minute, i disabled BES snmp.

[Ali]

Hi,

thanks for sharing the info

what is the BES server/software and how does HP SIM look at it.

1. can you see the trap name < > under options-> events -> SNMP Trap settings

2. Is the "Enable Trap Handling" Field set to true?
3. Have you assigned a Category and Severity of your choice?
4. Check the SNMP Filter setting options-> Event-> Event Filter Settings. Upper part
and lower part. For Testing click on all the checkboxes in the upper part and check that you have a * in the left lower part
5. Is the IP of the trap Source already discovered?

thanks,
Aftab

[pgarr]

BES = Blackberry Enterprise Server. It is just software on a Windows 2003 server but the software sends traps via the OS's snmp service.

I did not load the blackberry specific mibs so the traps hit HPSIM as Generic Unregistered events.

That said Generic Unregistered are handled/accepted.

The servers themselves are already discovered.


The BES softtware no longer sends traps though- they overloaded the HPSIM and started slowing processing of other events. I did not no this because the few automatic events I had set were configured to ignore things like generic unregistered events. In working with HP we found that the BES servers were flooding traps of type generic unregistered to the SIM server. It was sending several traps per minute for about a over almost a month. I have since disabled the BES software sending traps but they're still showing up on the HPSIM side.

[pgarr]

What I think, essentially, is that the massive flood from the bes servers over such a long period has queued up a poop load of events that HPSIM is now working through which i why I see events on HPSIM with a time stamp that makes no sense.

For example an event from xxxBES-server with a timestap when xxxBES-server had been offline for hours.

Also in my logs I loads of lines refereing to Adding trap of size: 264from x.x.x.249 to queue
Adding trap of size: 264from 10.14.8.245 to queue
Adding trap of size: 264from 10.14.8.249 to queue
Adding trap of size: 273from 10.14.8.227 to queue
Adding trap of size: 271from 10.14.8.227 to queue
Adding trap of size: 263from x.x.x.227 to queue

IPs are that of the bes servers. I know for absolute certainty that the servers are no longer sending traps- confirmed with snmputil sniffer on both ends and also killed trap service on hpsim but it still said it was receiving traps from those servers.

[MrEwald]

Better late than never :-)

I stumbled on this thread and would like to share the possibility how to clean the traps on the SIM server. If you have lots of traps coming in the SIM server store all the traps as files under the "HP\Systems Insight Manager\traps" folder. And even if you restart the SIM or the entire server it will still pickup and process all traps in the folder until they are all done. So how to clean.

- HPSIM stop

- Rename traps folder

- Create a new traps folder

- HPSIM start

- Delete the old traps folder at you convenience

Best of luck