net-snmp on HP-UX 11.31

[SyedHamid]

And there is no firewall in between, it is just two different VLANS

172.16.5.57 -- > Solarwinds server

172.16.184.34 -- > HP UX box

[Matti_Kurkela]

> Received SNMP packet(s) from UDP: [172.16.184.34]:64617->[0.0.0.0]:0

The messages say the received packet is coming from the HP-UX server itself, not from the Solarwinds server.

Is it possible that these messages are from your earlier testing on the HP-UX server locally?

If so, then it indicates that the net-snmp server has not received anything from the Solarwinds server at all.

Since you don't have IPFilter, the most likely explanation is that the packets from the Solarwinds server did not arrive to the HP-UX system. There must be something at the Solarwinds server or in the network between Solarwinds and HP-UX that stops the packets or routes them the wrong way.

To confirm this theory, the next step would be to take a network trace on the HP-UX system while running snmpwalk from the Solarwinds server. Here are instructions for taking a network trace using native HP-UX tools only:

http://www.compute-aid.com/nettl.html

Your filter file for the nettl trace should include the IP address of the Solarwinds server:

filter ip_saddr 172.16.5.57
filter ip_daddr 172.16.5.57

(If you have tcpdump installed on the HP-UX, "tcpdump -i lan0 -s1500 -n -vvv host 172.16.5.57" should produce a nice, readable trace from all the traffic between the HP-UX and the Solarwinds server.)

Start the trace on the HP-UX system, then run snmpwalk from the Solarwinds server to the HP-UX box. Then stop the trace and examine the results. If the trace contains nothing, that is because the HP-UX system has received nothing from the Solarwinds server. You now have confirmed that the problem is not in the net-snmp software, and most likely not in the HP-UX OS. If the NIC on the HP-UX box works normally otherwise, it is unlikely to be a hardware problem in the HP-UX box.

You might want to double-check the software firewall settings at the Solarwinds server. Since your net-snmpd on the HP-UX box is in a custom port (1161/UDP), the normal "allow SNMP traffic" rules might not cover traffic to UDP port 1161 on the HP-UX box. You may have add an extra rule to allow that.

If that is not the problem, the next step is to get help from your network administrator. You should be trying to get a similar network trace (also known as "traffic dump") at some point in the network that is roughly half-way between the Solarwinds server and the HP-UX server. The router that passes traffic from one VLAN to another (i.e. probably your default gateway, 172.16.184.1) might be a good first choice.

If that trace indicates that snmpwalk packets from the Solarwinds server are not reaching the point where you're tracing, the problem is between the Solarwinds server and that point; if you see snmpwalk packets at that point, the problem is somewhere between that point and the HP-UX box.

Keep splitting the problem area into smaller and smaller pieces by taking network traces in different points of the network while running snmpwalks from the Solarwinds server to the HP-UX, and you will eventually find something that should be passing the snmpwalk packets but isn't. There is the cause for your problem.

[SyedHamid]

Hi Matti,

How you are well, sorry for such a delayed response. I set up the native packet capture on HP-UX as advised by you. I have captured packets using filters. Following are the filters i'm using

---------------------- SUBSYSTEM FILTERS IN EFFECT -----------------


        ---------------- LAYER  1 -----------------

        ---------------- LAYER  2 -----------------

        ---------------- LAYER  3 -----------------
        filter ip_saddr          172.16.184.34
        filter ip_saddr          172.16.5.57

        ---------------- LAYER  4 -----------------
        filter udp_sport         1161

        ---------------- LAYER  5 -----------------

---------------------- END SUBSYSTEM FILTERS -----------------------

And ,whats intresting is that the packets ( UDP : generated by solarwinds server ) are reaching the HP-UX server however the HP-UX server is not resposnding back. From the packet capture, I guess I now know the problem . The packet capture shows that port 1161 on HP-UX server upon snmp poll is sending data to tiself on a different higher port. i.e source and destination ip are the same.

vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvARPA/9000 NETWORKINGvvvvvvvvvvvvvvvvvvvvvvvvvv@#%
  Timestamp            : Mon Jun 17 BST 2013 09:54:06.614198
  Process ID           : 19332              Subsystem        : NS_LS_IP
  User ID ( UID )      : 0                  Trace Kind       : PDU OUT TRACE
  Device ID            : -1                 Path ID          : 0
  Connection ID        : 0
  Location             : 00123
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
================================ IP Header (outbound -- pid: 19332) ==========
Source:  172.16.184.34(B) Dest:  172.16.184.34(B)
       len:  84      ttl: 64    proto: 17     cksum:  0xb822      id: 0xf27a
     flags:  NONE    tos: 0x0  hdrlen: 20    offset:  0x0     optlen: 0
-------------------------------- UDP Header ----------------------------------
sport:   1161   -->   dport:  50947     data len: 56    chksum: 0x40
-------------------------------- NET-SNMP ------------------------------------
   0: 30 36 02 01 01 04 0b 46 75 73 41 66 69 4c 32 49  06.....FusAfiL2I
  16: 51 73 a2 24 02 02 57 1b 02 01 00 02 01 00 30 18  Qs.$..W.......0.
  32: 30 16 06 08 2b 06 01 02 01 01 02 00 06 0a 2b 06  0...+.........+.
  48: 01 04 01 bf 08 03 02 0e -- -- -- -- -- -- -- --  ................

[SyedHamid]

The issue is of routing the traffic. When I look at the packet header of UDP traffic from 161 it has a FLAG ( DF ) and the routing is  proper. However when it comes to net-snmp the routing is not proper.

Any help with creating and deleting routes manually would be appreciated.

Thanks

Syed

[SyedHamid]

The issue is now resolved. My special thanks to Matti_Kurkela for his time and valuable feedback.

Coming to the issue after going through the documentation of NET-SNMP 5.7  the behaviour i was experiencing is a known "regression". I downgraded to Net-SNMP 5.6.1.1 and it works as a charm.

Once again thank you all for your awesome support.

Thanks

Syed

[sbrews]

I seem to be having a similar problem - I would like to try the 5.6.1.1 version, but have not been able to find a link for it.  do you have a link you can share?

[sbrews]

Disregard my question - I was able to get a copy of the 5.6.1.1 netsnmp and it resolved the issue I was seeing.