Ever recover from a trap storm?

[Dana Swanson]

Has anyone out there recovered from a snmp trap storm?

We had several servers sending traps all at one time filling the database beyond the hp sim consoles ability to manage.

Spoke to hp, was shown how to remove the pending events from the database and what to run to fix the database.

Just curious if anyone went through this before.

Thanks

[Ervinck Kevin]

I haven't seen this before. When snmp trap storm occured had you been updating the particular servers in any way (windows updates, psp)?

[Dana Swanson]

Hi Kevin,

We had a number of servers unknowingly configured to send test traps at a high rate, the admin who configured the server was unaware.

SNMP was setup wrong on a few server as well, that added to it.

I'll reply later with some info on how to check if something is going wrong.

Dana

[Dana Swanson]

Hi All,

Sorry for replying so late on this one.

1. Find a copy of SNMPUTIL.EXE

SNMPUTIL.EXE is a one of the tools on Windows NT 4 Resource Kit. You should be able to find a copy by searching the web on "download SNMPUTIL.EXE", make sure to scan it with your AV software and some anti-malware software just to be safe. It's still popular to use out there.

2. You first want to create a folder on the hpsim server to place SNMPUTIL.EXE in. Open a CMD box and browse to that folder.

3. You can then type "SNMPUTIL.EXE trap" to display the traps on the screen as a quick check. All the traps being sent to the hpsim server are now being displayed. If it's super busy with lots of entries passing by you may have a trap storm.

Stop SNMPUTIL.exe from running (close the cmd box or control-c 3 or 4 times)

4. Type "SNMPUTIL.EXE trap>trapreport.txt" and let it run for enough time to get a good log. 30 minutes may be good to start with.

5. With the cmd window open mouse click the cmd box, press Control-C 3 or 4 times (until the the app stops running)

6. You should now have a trapreport.txt to open and look through.

7. If you see several traps with the same source IP within a quick amount of time (seconds or less than seconds) and the traps are identical the source IP of the trap should be looked at.

An example of a server causing part of a trap storm:

snmputil: trap generic=4 specific=0
from -> 192.168.1.105

snmputil: trap generic=4 specific=0
from -> 192.168.1.105

snmputil: trap generic=4 specific=0
from -> 192.168.1.105

snmputil: trap generic=4 specific=0
from -> 192.168.1.105

snmputil: trap generic=4 specific=0
from -> 192.168.1.105

snmputil: trap generic=4 specific=0
from -> 192.168.1.105

snmputil: trap generic=4 specific=0
from -> 192.168.1.105

snmputil: trap generic=4 specific=0
from -> 192.168.1.105

8. The trap type "generic=4 specific=0" means it's not in hp sim mibs and an unknown trap.

9. The cause is most likely snmp mis-configured on more than just one server.

10. Check snmp on the server after the "from ->" field

11. If everything looks fine remove the check box from "send authentication trap", this was from hp, not sure why.

I don't think this is a common issue but if your database seems to be filling quickly or the size of the database getting way too large, this is one issue you can check on.

Dana

[mark q]

I just went through it for 4200+ nodes..

built a reg file that will uncheck it on all of the nodes and then pushed it out through psexec..

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SNMP\Parameters]
"EnableAuthenticationTraps"=dword:00000000

my traps went from 300 a minute down to about 1 every 10 minutes.. big help

[Dana Swanson]

I'm glad it worked out. I'm usually searching for some fix here on the forum.