Alarm system detected alarms from unmanaged devices

[weird_harold]

I continue to get these alarms on a daily basis. Any ideas on what's causing this?

----------------------

Alarm Details
    Name        Alarm system detected alarms from unmanaged devices
    Level        Warning
    OID        1.3.6.1.4.1.25506.4.2.2.2.6.2
    Alarm at        2014-01-20 06:49:26
    Alarm Source        
    NMS(127.0.0.1)
    Type        iMC
    Alarm Category        NMS Alarm (Partial)
    Recovery Status        Unrecovered
    Acknowledgement Status        Unacknowledged
    Description        iMC alarm system has received 1000 alarms from the unmanaged device from 2014-01-19 06:27:37 to 2014-01-20 06:49:26.
    Alarm Cause        iMC alarm system has received a large amount of alarms from unmanaged devices.
    Remediation Suggestion        Please add the unmanaged devices to the system for management.
    Maintenance Experience        
    Note        -- [Modify]

Alarm Parameter        
    Parameter Name    Parameter Value
    *Start Time    2014-01-19 06:27:37
    *Stop Time    2014-01-20 06:49:26
    Times    1000

[LindsayHill]

You're getting SNMP traps from devices that aren't managed by IMC.

With that volume, the most likely source is from some system that has SNMP Authentication Failure traps enabled, and IMC set as its destination.

You've got three choices:

1/ Fix the offending system

2/ Add the offending system to IMC

3/ Turn off notification for alarms from Unknown systems.

The problem with 1 & 2 is identifying the offending systems. The easiest way is probably to go to Alarm -> Trap Management -> Filtering Trap. Click Modify next to "Unknown Trap Filter". Uncheck "Filter Unknown Traps" - now you will at least see the alarms in Alarm -> Trap Management -> Browse Trap. You can then see which system you need to go and fix, or add to IMC.

If you don't care that systems are sending you large numbers of unnecessary traps, and you just want to stop the alarm about it, then you could uncheck "Escalate to alarms" from the above Filter. 

[weird_harold]

Thanks Lindsay,

I think your steps to uncheck "filter unknown traps" is what I  was looking for. I'd prefer to track down what system(s) are causing this, but the original alert I posted had localhost 127.0.0.1 as the source.

I'll post back if this does the trick, in case others have the same issue.

Thanks again.

[LindsayHill]

The reason you see "127.0.0.1" in the original alert is that it's raised against the NMS itself. The message is saying that it received a total of 1,000 messages from unknown sources - that includes many different sources, so it just rolls the alert up into one against the NMS itself.

(Yes, it would be nice if it had some more analysis - e.g. 100 from 10.1.1.11, 50 from 10.1.1.2, etc).

Hopefully you can now track down the problematic systems.

[avnero]

Hello,

Can I change the alarm that I received via email to show me the source IP (instead of 127.0.0.1)?

Thanks

[LindsayHill]

Not in this case, because there is no one single source IP. Instead, there could be many source IPs. The alarm is saying "I got 1000 alarms from unknown sources" - it's not saying "I got 1000 alarms from this specific source"

[swinkster46]

Hello, I was happy to see this post; I've been frustrated with the numbers of these alarms being generated. I unclicked "Filter Unknown Traps" and waited a while. I am still seeing alarms "iMC alarm system has received 1000 alarms from the unmanaged device from 2014-05-03 15:29:58 to 2014-05-03 15:35:18." but when I go to Alarm/Trap Management/Browse Trap, I don't see any traps that correspond to this alarm. I am running 7.0 E0202. Is there somewhere else I can look for this information? TIA.

[GiroSinTornillo]

I have the same issue.

 I cant track down the source of unknow alarms.

What steps do you recommend Lindsay to get source identified?

Thanks in advance.

Hugo

[LindsayHill]

Disable "Filter Unknown Alarms"

Also try using tcpdump to look for syslog & SNMP traps, and check your sources.

[flibo]

Hi,

i think you can see device ip on the menu "Alarm\Trap Management\Browse Trap"

François

[aroman]

Thanks a lot to all that contributed on this post. We also had the same warning alarms for a device that was trying to report SNMP traps to iMC every minute (we had hundreds of these alarms). Even if you specify the OID of the alarm in iMC by going to Alarm > Trap Management > Trap Definition (by adding the OID in the "Trap OID" field and doing a Query), click on Modify, and try to include the source IP address in the trap description with the $a macro as indicated in the ? at the right of the description, the source IP address that you will get is the local IP address 127.0.0.1, not the source IP address from the device that is sending the SNMP traps

The only way where we could identify the source IP address of these SNMP traps, is by installing tcpdump in the iMC server (for Windows, you can use TCPDUMP for Windows that you can download from http://www.microolap.com/products/network/tcpdump/), and capture packets for some time using this filter:

tcpdump -D //To identify the interface ID on which you need to capture packets

tcpdump -i <Interface ID> udp port 161 -w <Filename where you want to save the packet capture>.pcap //To capture the packets

Then you can copy the packet capture file to a device with Wireshark installed (you can download Wireshark from https://www.wireshark.org/download.html), open the capture file, and on the menu at the top go to Statistics > Conversations. On the tab called "IPv4", on the Address A column, identify any unknown IP addresses. These should be most likely the ones that are trying to report SNMP traps to iMC, and have not being added in iMC. You can then define a filter in Wireshark for these unknown IP addresses by doing a right-click on those Conversations, and select Apply as filter > Selected > A - B, then on the filtered packets, you can identify the SNMP version, the SNMP community string, and the OID (indicated as enterprise)

Once the IP address from the device that is sending the SNMP traps is identified, you can:

1. If you manage this device and want to receive SNMP traps in iMC: Add it to iMC

2. If you manage this device but you do not want to receive SNMP traps in iMC, or manage it using iMC: Configure the device to stop sending SNMP traps

3. If this is a rogue device or you do not manage it: Configure the firewalls between this device and iMC to drop these packets

I hope that this helps solving similar issues