SNMP Secuirty Access Violation

[swaize]

Our trap receiver is getting flooded with the error stated in the title. It seems to be caused when out Procurve Manager server tries to communicate with the switch.

If I do a 'test communication parameters' in PCM, on that switch, then everything comes back as successful.

We are using SNMPv3, and it only seems to effect certain switches, although they are all set up the identically.

Full copy and paste of error below:
hpSwitchJ8165A.0.2 [1] eventDescription.235 (DisplayString): W 08/18/10 09:38:15 snmp: SNMP Security access violation from 192.168.255.200 [2] snmpTrapEnterprise.0 (Object ID): hpSwitchJ8165A

FYI - 192.168.255.200 is our PCM server.

Has anyone come across this or know how to resolve it?

[Tore Valberg]

Hi Swaize

I see you get the error on a 2650.

I would recommend you to install H.10.83 as it contains some snmpv3 related fixes.

Should fix your issue.

http://h10144.www1.hp.com/customercare/support/software/summarypages/h-j4900-c.htm

Tore

[swaize]

Thanks for replying, however I'm already running that version i'm afraid.

[Tore Valberg]

Hmm

Is it only happening with 2600 switches?

Could you post the switch configuration?

It should work just fine with H.10.83

Could there be any other snmp software running in the background on the PCM server?

Tore

[Tore Valberg]

I have set up a simple isolated test with a PCM 3.1 server and a 2650 running H.10.83.

I will let you know what i see.

Tore

[swaize]

Only see this problem on 2600's. Not all seem to be affected though, despite identical setup.

While it is happening, I'm getting exactly 3 traps sent every minute, so for some reason PCM is polling this switch at this interval all day.

I have just used wireshark on PCM to have a look at what is going on, and snmp conversation looks different coming from problem switch than to one working normally.

Normal switch:
http://i7.photobucket.com/albums/y255/paaaatrick/norm.jpg

Problem switch:
http://i7.photobucket.com/albums/y255/paaaatrick/prob.jpg

So, seem to be two issues:
1. PCM is polling this switch all day and all night, 3 times a minute.
2. Said switch is not replying correctly and causing the error.

Config below:
; J8165A Configuration Editor; Created on release #H.10.83

hostname "WE 1A01 Switch 2650-PWR"
snmp-server contact "networkfaults@xx.com"
snmp-server location "1st Floor, Cabinet A"
time daylight-time-rule Western-Europe
cdp run
no web-management
web-management ssl
no telnet-server
interface 15
disable
name "Faulty Port"
no power
exit
ip default-gateway 10.202.1.253
sntp server 192.168.255.200
timesync sntp
sntp unicast
snmpv3 enable
snmpv3 only
snmpv3 user "wireworks"
snmpv3 group ManagerPriv user "initial" sec-model ver3
snmpv3 group ManagerPriv user "wireworks" sec-model ver3
snmp-server community "public" Unrestricted
snmp-server host 192.168.255.200 "public" Not-INFO
snmp-server host 10.1.11.144 "public" Not-INFO
vlan 1
name "DEFAULT_VLAN"
untagged 1-50
ip address 10.202.2.12 255.255.0.0
exit
fault-finder bad-driver sensitivity high
fault-finder bad-transceiver sensitivity high
fault-finder bad-cable sensitivity high
fault-finder too-long-cable sensitivity high
fault-finder over-bandwidth sensitivity high
fault-finder broadcast-storm sensitivity high
fault-finder loss-of-link sensitivity high
fault-finder duplex-mismatch-HDx sensitivity high
fault-finder duplex-mismatch-FDx sensitivity high
aaa authentication login privilege-mode
aaa authentication console login radius local
aaa authentication console enable radius local
aaa authentication web login radius local
aaa authentication web enable radius local
aaa authentication ssh login radius local
aaa authentication ssh enable radius local
radius-server dead-time 5
radius-server timeout 3
radius-server retransmit 2
radius-server host 10.1.10.96 key xxxxx
radius-server host 10.1.12.28 key xxxxx
ip ssh
ip ssh filetransfer
ip ssh version 1-or-2
no tftp client
no tftp server
password manager

[Tore Valberg]

Hi

For retries every 3 minutes, that is simply PCM retrying as it initially failed. And keeps doing that.

The snmp report you have highlighted on the non working switch "usmStatsNotInTimeWindows" comes right before the switch sends the snmp access violation trap.

It could simply be the config on the pcm side. But looks to be related to an issue with SNMP API.

Have you tried rebooting the switch? Looks like it might be the solution:

http://kbase.zohocorp.com/kbase/SNMP/SNMPv3/file_110761.html

If that is no go, could you take a problematic device and do the following:

Right click device > device access > communication parameters in PCM.

Then run trough the wizard and (re)set the SNMPv3 credentials.

This will only affect PCM's access settings for that spesific device.

Tore

[swaize]

Have checked the settings on the PCM side. The SNMPv3 details are the default setting, and already seemed to be set correctly, however I re-applied them anyway. This made no difference. Strange thing is test comms parameters report shows everything including SNMPv3 as testing ok, but the same test re-produces the error. I noticed some of the captured packet appeared to be snmpv2, maybe PCM is testing with both causing both the error and the reported success.

Weirdly some other switches of the same type stared doing the same thing earlier. So I'm starting to think PCM is the issue. We have had issues with PCM forgetting comms parameters in the past, although this seems to be something a bit different.

Anyway, will try and give it a reboot out of hours, and see if this helps.

[Tore Valberg]

Hi

I dont see any issues in my test bed so far.

Could you check excatly what version of PCM you are running?

I am running PCM 3.1 AutoUpdate 3.

[swaize]

Running C.03.1.201

We have had some other issues and actually had a guy from HP PCM team dialling in and fixing things. So It's possible we don't have even have an official version running, as he has been changing things 'on the fly'.

Issue was specifically with switches forgetting comms parameters, they would appear blank if you checked what was stored. He did seem to have fixed that issue, but not surprised if there is now another issue along same lines. Correct prams appear to be stored for these switches.

I might get him to take a look at this also.

[Matt Hobbs]

I'm having the exact same issue with SNMPv3. Test communication parameters is fine, but bogged down in these access violation errors. Also getting HP Support to remote in to my PCM server to take a look when I can find the time.

[Jay671]

Same thing here. I am running on server 2008 R2 though. Refer to my thread here

http://forums13.itrc.hp.com/service/forums/questionanswer.do?threadId=1442355

It's all my edge switches that are doing this. The higher end switches (2910, 3500, 5400, 8200) they all work fine, but I also had to set the MIB manually in each of those to get them to stop with the errors.

Had a case open with HP, did all kinds of troubleshooting, packet captures, trace files, and in the end they closed the case saying they don't support server 08 R2 and used that as the reason.
Not sure if that is similar in your case or not.

-Jay

[Jay671]

I guess I should have mentioned this in my previous post. The reason I talk about the fact that it's not reporting traffic stats is, if you go into PCM, click on one of the switches filled with these errors, then click on the traffic tab at the top, right click any active port and choose "port summary", then click on the "log" tab, you should notice that it errors out every time it tries to retrieve traffic stats from an edge switch with something similar to the below text....

06/28/10 12:27:35p: Poll port[id=2816772|ifindex=49] error (verify device is connected correctly)
06/28/10 12:27:33p Poller port [id=2816772|ifindex=49] timer expired (attempting to poll device again)
06/28/10 12:17:33p: Poll port [id=2816722|ifindex=49] error (verify device is discovered correctly)


Now compare those times with the log in the switch where you are receiving the errors of

W 06/28/10 12:27:35p snmp: SNMP Security access violation from 10.x.x.x

And you should notice that the times match up, or are at the very least close enough to point at what PCM is trying to do when the switch receives the error message. Even though right clicking in PCM and testing parameters all passes.

As I said, I had a case open with HP, which was eventually escalated to the engineers, all of this info was given to them along with multiple trace files, screen shots, show tech all's, a sample of my blood, my first born child, and in the end they came back with the fact that I was running it on an unsupported Server OS. So if this is the case for you as well, get used to your edge switches filling up with the errors.

Best you can do is change the setting in the agent manager to poll less often or not at all.

[swaize]

Was kind of expecting this might be the case in the end!

Not enough people running v3 to make them fix it I guess.

[Tore Valberg]

06/28/10 12:27:35p: Poll port[id=2816772|ifindex=49] error (verify device is connected correctly)
06/28/10 12:27:33p Poller port [id=2816772|ifindex=49] timer expired (attempting to poll device again)
06/28/10 12:17:33p: Poll port [id=2816722|ifindex=49] error (verify device is discovered correctly)


That errors is quite interesting. Did you try deleting , and then manually discover the switch?

Did any of you open a case with procurve support?

At least i can confirm this issue have not been reported to L2 support in EMEA.

If you have a open case with front line support, they should be able to elevate the case to L2 if they receive the correct data needed.

I not already done, i would collect the following data, and insist that the case is elevated further: (also refer to this thread)

- Server, client and agent logs from the PCM server. (Tools - Preferences - Archive logs) And chose to collect all logs.
- MSINFO32.exe infor] from the server. (Export as nfo or txt)
- System and application logs from event viewer.

Tore

[Jay671]

Tore,

Yep did that, done that and so much more. I had an open case, was dealing heavily through email. Received a response stating it was getting handed off to "Eric Knight" which I assumed was an escalation. after several more emails back and forth, this was his response.

"Hi Jason,

Thanks for clarifying. Heres where we are going to run into problems. We currently are not supporting R2. HP will probably support it in the future but its still in the development phase. You can try to enable Sflow manually on the 2610,5400,3500 etc. using the commands from the document you sent me. If that works then that would be the workaround for now. But until we have developed the correct hooks to intergrate with 2008 R2. We are probably not going to see any consistant results with Traffic Monitor. When full support for R2 is available, I can let you know.

Eric"

If you have access the case number is 4614699099

I have also tried several tweaks on the server side of things all unsuccessful.
If you want more info, or every single stpe I have tried, including rebuilding from scratch, let me know.

-Jay