SNMP Does not respond when server is busy

[Jason Martens]

Hey All,
I am monitoring and tracking things like CPU utilization via snmp which works great most of the time. However, whenever the box gets busy, my snmp queries time out, leaving gaps in my graphs. Has anyone else had this problem? Is it possible to increase the priority of the snmp daemons? What is the best way to do something like that?

[rick jones]

You might first want to make sure that it is indeed simply delays in responses and not lost SNMP queries. Depending on where/how the queries/replise are lost, doing something like "nice" or "rtprio" (be _really careful there) may not help.

For example, does lanadmin show dropped packets for your interfaces? Does netstat -p ip show packets dropped for other reasons? Does netstat -p udp show socket buffer overflows?

[Jason Martens]

lanadmin, netstat -p ip and netstat -p udp all show no errors. I am pretty sure that the problem is it just takes too long to process the snmp queries, and the app I am using (nagios/apan) times out after 10 seconds. I have observed slow response when doing a manual query as well. I guess I could just increase the timeout, but it seems like it should not take more than 10 sec to query 4 oids.

[SAHA]

One thing clear you don't have network bottleneck and no UDP packet lost.
So, it remains your box.
I am sure there is high cpu Qlengths and snmpd is not getting enough CPU cycles to finish its job.
Increaing priority will definitely solve the problem but you land up into other problems in other applications.

For SNMP there are 3 important parameters:
1)polling frquency.
2)Timeout
3)Retry count

If your polling freq is 5 minutes (which is reasonably good),
try increase Time out and retry count.

If polling freq is too small, increase it, if it is acceptable.

I saw similar problem in busy server and this solved the problem ...with acceptable limit.

Thanks,

[rick jones]

well, if indeed there are no losses then nice/renice/rtprio would be the things to consider. must be a _really_ busy box

[Jason Martens]

Thanks for the input.

My current polling interval is 5 min. I will try changing the timeout to 15 sec, but much longer than that and it will really hold up my other checks.

It's not that the box is super busy, but when there is a big job (backups for instance) I just have big holes in my graphs for as much as several hours, which is right when it would be interesting to look at that data.

Any thoughts on the best way to renice the snmpd process? Can I just modify the startup script so it is always niced when the box boots up?

Thanks again.