cancel
Showing results for 
Search instead for 
Did you mean: 

net-snmp on HP-UX 11.31

SyedHamid
Advisor

net-snmp on HP-UX 11.31

Hi All,

 

I'm trying to monitor a HP-UX server with an application Orion Solarwinds

 

Following are the version details

 

NET-SNMP version:  5.7.2

 

HP -UX  :  B.11.31

 

I have disabled HP-UX's snmp and have installed  it from HP recommended depot ( https://h20392.www2.hp.com/portal/swdepot/searchProducts.do )

 


I'm able to snmpwalk on version 1 via public community on the HP-UX server. However I'm not able to get the server to be snmpwalked via a community from the monitoring system.

 

I have attached my snmpd.conf file and netstat details.

 

Any help would be highly appreciated.

 

Thanks

Syed

26 REPLIES
SyedHamid
Advisor

Re: net-snmp on HP-UX 11.31

Also following is my lsof output

 

lsof -i udp:161

 

COMMAND  PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
snmpd   4878 root    5u  IPv4 0xe0000003139bbc80      0t0  UDP *:snmp (Idle)
snmpd   4881 root    5u  IPv6 0xe00000036f71c980      0t0  UDP *:snmp (Idle)

Matti_Kurkela
Honored Contributor

Re: net-snmp on HP-UX 11.31

Your lsof output and your previous netstat output don't match:
according to the "agentaddress" setting in the configuration you posted, the net-snmp daemon should be listening in port 1161, as confirmed by netstat. Your ps -ef indicates that the net-snmp daemon is running as PID 2022.

 

Yet your lsof indicates snmpd is running in port 161 and with PIDs 4878 (for IPv4) and 4881 (for IPv6).

 

What is going on?

MK
SyedHamid
Advisor

Re: net-snmp on HP-UX 11.31

I was testing,I have changed my agentaddress to 161. My new snmpd.conf ( relating to my lsof output ) has been uploaded.

Matti_Kurkela
Honored Contributor

Re: net-snmp on HP-UX 11.31

Is 172.16.5.57 the IP address of your monitoring system? At the moment, it seems to be the only remote host that is authorized to query your net-snmp daemon for anything at all.

 

If your monitoring system has multiple NICs and/or IP addresses, are you sure its requests to this HP-UX hosts are sent out via the 172.16.5.57 interface?

 

With your current configuration, queries from 172.16.5.57 will only succeed if community "FusAfiL2IQs" is used, and even then with read access only.

Any other non-local queries will fail.

MK
SyedHamid
Advisor

Re: net-snmp on HP-UX 11.31

Hi,

 

True, on the HP-UX server

 

/opt/iexpress/net-snmp/bin/snmpwalk -v 2c -c FusAfiL2IQs 172.16.184.34 system

 

and

 

/opt/iexpress/net-snmp/bin/snmpwalk -v 2c -c FusAfiL2IQs 172.16.5.57 system

 

Work absolutely fine. However when I try and poll the HP UX server from the Orion Solarwinds applications it fails.

 

Thanks

Syed

Matti_Kurkela
Honored Contributor

Re: net-snmp on HP-UX 11.31

So the IP address 172.16.5.57 belongs to the HP-UX server itself? And maybe 172.16.184.34 too?

 

Then you need to find out the IP address the Orion Solarwinds applications send their SNMP requests from, and add it to a line like this:

com2sec mynetwork  aa.bb.cc.dd          FusAfiL2IQs

 Replace aa.bb.cc.dd with the IP address the Orion Solarwinds uses.

MK
SyedHamid
Advisor

Re: net-snmp on HP-UX 11.31

com2sec mynetwork  172.16.5.57          FusAfiL2IQs

 

is my entry

 

172.16.5.57   - Is the Solarwinds server

 

172.16.184.34 - Is the ip of the HPUX box

Matti_Kurkela
Honored Contributor

Re: net-snmp on HP-UX 11.31

Try restarting snmpd with full debug logging enabled:

 

/opt/iexpress/net-snmp/sbin/snmpd -LF7 /tmp/snmp-troubleshooting.log -a -p /var/tmp/net-snmpd -c /opt/iexpress/net-snmp/etc/snmpd.conf

This should produce a log file at /tmp/snmp-troubleshooting.log.

 

Then send some queries locally, using first a correct community name and then an incorrect one. Both queries should produce messages in the log.

 

Then send some queries from the Solarwinds server and look at the log again.

 

I think there are three possibilities:

  • no new messages in the log = something is blocking the queries from the Solarwinds server from reaching the HP-UX system
  • the messages say that snmpd is rejecting the queries from the Solarwinds server = most likely the Solarwinds server is not actually using the community name you expect it to use, or
  • the log file says snmpd is receiving the query and responding to it = something is blocking the response from reaching the Solarwinds server.

 

 

MK
SyedHamid
Advisor

Re: net-snmp on HP-UX 11.31

no new messages in the log = something is blocking the queries from the Solarwinds server from reaching the HP-UX system.

 

 

SyedHamid
Advisor

Re: net-snmp on HP-UX 11.31

Here is the output from the log

Warning: no access control information configured.
(Config search path: /opt/iexpress/net-snmp/etc/snmp:/opt/iexpress/net-snmp/share/snmp:/opt/iexpress/net-snmp/lib/snmp://.snmp)
It's unlikely this agent can serve any useful purpose in this state.
Run "snmpconf -g basic_setup" to help you configure the snmpd.conf file for this agent.
Matti_Kurkela
Honored Contributor

Re: net-snmp on HP-UX 11.31

>Warning: no access control information configured.

 

If there is nothing else in the log, then the snmpd is not reading the configuration file at all???

 

Wait... the command in my previous reply was supposed to be a single long line. If you typed it as two separate commands, that might explain this error.

 

>(Config search path: /opt/iexpress/net-snmp/etc/snmp:/opt/iexpress/net-snmp/share/snmp:/opt/iexpress/net-snmp/lib/snmp://.snmp)

 

According to this, you should probably put your snmpd.conf file to /opt/iexpress/net-snmp/etc/snmp/ directory. If the directory does not exist, create it. Then you should not need to explicitly tell snmpd the configuration file location with the -c option.

MK
SyedHamid
Advisor

Re: net-snmp on HP-UX 11.31

Hi Matti,

 

Thank you for your patience, still doesnt work

 

I have snmp(HP's) running on udp 161 and net-snmp on 1161

 

HPUX-root-:/#>lsof -i udp:1161
COMMAND   PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
snmpd   14558 root    6u  IPv4 0xe00000036a2ba100      0t0  UDP *:netsnmp (Idle)

 

HPUX-root-:/#>lsof -i udp:161
COMMAND   PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
snmpdm  14986 root    6u  IPv4 0xe00000038ead7080      0t0  UDP *:snmp (Idle)

 

 

/opt/iexpress/net-snmp/bin/snmpwalk -m ALL -v 2c -c FusAfiL2IQs 172.16.184.34:1161 .1

 

Does not work

 

/opt/iexpress/net-snmp/bin/snmpwalk -m ALL -v 2c -c FusAfiL2IQs 172.16.184.34:161 .1

 

Works

 

/opt/iexpress/net-snmp/bin/snmpwalk -m ALL -v 2c -c public localhost:1161 .1

 

Works

 

 

 

Any idea's

Matti_Kurkela
Honored Contributor

Re: net-snmp on HP-UX 11.31

> /opt/iexpress/net-snmp/bin/snmpwalk -m ALL -v 2c -c FusAfiL2IQs 172.16.184.34:1161 .1

> Does not work

 

What is the error message? Is it "Timeout: No Response from 172.16.184.34:1161" or something else?

 

If you do this with snmpd debug logging enabled, do any new messages appear in the log? What do they say?

MK
SyedHamid
Advisor

Re: net-snmp on HP-UX 11.31

/opt/iexpress/net-snmp/bin/snmpwalk -m ALL -Le -v 2c -c FusAfiL2IQs 172.16.184.34:1161 .1

 

Timeout: No Response from 172.16.184.34:1161

 

Log output  is ( Verbose Mode)

 

NET-SNMP version 5.7.2
Received SNMP packet(s) from UDP: [172.16.184.34]:59733->[0.0.0.0]:0
  GETNEXT message
    -- ccitt.1
Received SNMP packet(s) from UDP: [172.16.184.34]:59733->[0.0.0.0]:0
  GETNEXT message
    -- ccitt.1
Received SNMP packet(s) from UDP: [172.16.184.34]:59733->[0.0.0.0]:0
  GETNEXT message
    -- ccitt.1
Received SNMP packet(s) from UDP: [172.16.184.34]:59733->[0.0.0.0]:0
  GETNEXT message
    -- ccitt.1
Received SNMP packet(s) from UDP: [172.16.184.34]:59733->[0.0.0.0]:0
  GETNEXT message
    -- ccitt.1
Received SNMP packet(s) from UDP: [172.16.184.34]:59733->[0.0.0.0]:0
  GETNEXT message
    -- ccitt.1

 

 

SyedHamid
Advisor

Re: net-snmp on HP-UX 11.31

Ok, Now I have made a minor chane to the snmpd.conf file

 

com2sec mynetwork 172.16.184.34     FusAfiL2IQs

 

Where the ip 172.16.184.34   belongs to the HP BOX and now when I do

 

/opt/iexpress/net-snmp/bin/snmpwalk -m ALL -Le -v 2c -c FusAfiL2IQs 172.16.184.34:1161 .1

 

it works

 

following is the log

 

Received SNMP packet(s) from UDP: [172.16.184.34]:60782->[0.0.0.0]:0
  GETNEXT message
    -- ccitt.1
Received SNMP packet(s) from UDP: [172.16.184.34]:60782->[0.0.0.0]:0
  GETNEXT message
    -- SNMPv2-MIB::sysDescr.0
Received SNMP packet(s) from UDP: [172.16.184.34]:60782->[0.0.0.0]:0
  GETNEXT message
    -- SNMPv2-MIB::sysObjectID.0
Received SNMP packet(s) from UDP: [172.16.184.34]:60782->[0.0.0.0]:0
  GETNEXT message
    -- DISMAN-EVENT-MIB::sysUpTimeInstance
Received SNMP packet(s) from UDP: [172.16.184.34]:60782->[0.0.0.0]:0
  GETNEXT message
    -- SNMPv2-MIB::sysContact.0
Received SNMP packet(s) from UDP: [172.16.184.34]:60782->[0.0.0.0]:0
  GETNEXT message
    -- SNMPv2-MIB::sysName.0
Received SNMP packet(s) from UDP: [172.16.184.34]:60782->[0.0.0.0]:0
  GETNEXT message
    -- SNMPv2-MIB::sysLocation.0
Received SNMP packet(s) from UDP: [172.16.184.34]:60782->[0.0.0.0]:0
  GETNEXT message
    -- SNMPv2-MIB::sysORLastChange.0
Received SNMP packet(s) from UDP: [172.16.184.34]:60782->[0.0.0.0]:0
  GETNEXT message
    -- SNMPv2-MIB::sysORID.1
Received SNMP packet(s) from UDP: [172.16.184.34]:60782->[0.0.0.0]:0
  GETNEXT message
    -- SNMPv2-MIB::sysORID.2
Received SNMP packet(s) from UDP: [172.16.184.34]:60782->[0.0.0.0]:0
  GETNEXT message
    -- SNMPv2-MIB::sysORID.3
Received SNMP packet(s) from UDP: [172.16.184.34]:60782->[0.0.0.0]:0
  GETNEXT message
    -- SNMPv2-MIB::sysORID.4
Received SNMP packet(s) from UDP: [172.16.184.34]:60782->[0.0.0.0]:0
  GETNEXT message
    -- SNMPv2-MIB::sysORID.5
Received SNMP packet(s) from UDP: [172.16.184.34]:60782->[0.0.0.0]:0
  GETNEXT message
    -- SNMPv2-MIB::sysORID.6

 

 

However Solarwinds ( windows server) still fails to snmpwalk the machine  to the HP box on 1161 port.

 

 

Matti_Kurkela
Honored Contributor

Re: net-snmp on HP-UX 11.31

OK.

 

If I understood correctly, your snmpd.conf file should now have both IP addresses on com2sec lines:

com2sec mynetwork 172.16.5.57       FusAfiL2IQs
com2sec mynetwork 172.16.184.34     FusAfiL2IQs

 

Now, if you try snmpwalk from the Solarwinds server, with the logging enabled at the HP-UX server, what happens?

 

Do you see messages like "Received SNMP packet(s) from UDP: [172.16.5.57]:<some port number>->[0.0.0.0]" in the log at the HP-UX server, or not?

 

If not, it may be a network communication issue in getting the packets from the Solarwinds server to the HP-UX server.

 

If the messages are appearing in the log and you are using the correct community name at the Solarwinds server, then it is probably another network communication issue: something is preventing the answers of the HP-UX server from getting back to the Solarwinds server.

 

If you see lines like this:

Connection from UDP: [172.16.5.57]:<some port number>->[0.0.0.0] REFUSED

 ... then your net-snmpd is receiving the query from the Solarwinds server, but rejecting it because the source IP address is not in the allowed list. The net-snmpd may be compiled with the "libwrap support" feature enabled: in that case, you should add a line like this to your /etc/hosts.allow file (or create the file if it does not exist):

snmpd:  127.0.0.1 172.16.184.34

 

If the log seems to indicate that net-snmpd is receiving and accepting the query from Solarwinds, or if no query from Solarwinds seems to be received at all, it might be a firewall issue.

 

SNMP is an UDP-based protocol. With UDP, allowing communication from A to B does not automatically guarantee that replies from B back to A are allowed too. This is because some UDP-based protocols are truly one-directional and won't ever reply. (For example, the syslog remote logging protocol.)

 

With TCP-based protocols, sending something from A to B always requires that B can send acknowledge messages back to A, so if communication from A to B is allowed, replies from B to A must be automatically allowed too.

 

Most modern protocols use TCP, so sometimes firewall administrators assume that the TCP behavior is the standard for all protocols, and forget that UDP-based protocols can be different. Some firewalls can automatically allow the replies with UDP protocols too, but that requires that the firewall "knows" how the UDP-based protocol is supposed to behave. A SNMP protocol in standard port 161 might have its responses handled automatically, but using SNMP with a custom port 1161 might need some special settings, or just a separate firewall rule like:

allow UDP packets from port 1161 of 172.16.184.34 to any port of 172.16.5.57

The usual firewall rules are like:

allow <protocol> from any port of <source IP> to port <number> of <destination IP>

so the net-snmp in a custom port might need a rule that works in a reverse sense. Most firewalls allow this, but the firewall administrator might not be able to use his/her "standard template" in defining the rule for net-snmp replies, so mistakes are more likely than usual.

 

The firewall might be implemented in software too: an IPFilter on the HP-UX system or a Windows firewall on the Solarwinds system might be causing the problem. If a HP-UX system has IPFilter installed and configured, its configuration files should be in /etc/opt/ipf. If that directory does not exist or is empty, you can be reasonably sure you don't have IPFilter active on your system.

MK
SyedHamid
Advisor

Re: net-snmp on HP-UX 11.31

Thanks for detailed explanation, really appreciated. I have check with IP filtering

 

######-root-:/#> ipf -V
ipf: IPFilter is currently disabled

 

I will talk to security team today and see if there is any rule in place or if required will create one.

 

Thanks and will keep you posted

SyedHamid
Advisor

Re: net-snmp on HP-UX 11.31

Ok, now i have some real data.

 

When I snmpwalk from the Solarwinds server ( 172.16.5.57) to HP-UX box 172.16.184.34  on SNMPv2 and the defined community, following is what is captured in the log file

 

NET-SNMP version 5.7.2
Received SNMP packet(s) from UDP: [172.16.184.34]:64617->[0.0.0.0]:0
  GET message
    -- SNMPv2-MIB::sysObjectID.0
Received SNMP packet(s) from UDP: [172.16.184.34]:64617->[0.0.0.0]:0
  GET message
    -- SNMPv2-MIB::sysObjectID.0
Received SNMP packet(s) from UDP: [172.16.184.34]:64617->[0.0.0.0]:0
  GET message
    -- SNMPv2-MIB::sysObjectID.0

 

The server fails detection on Solarwinds

SyedHamid
Advisor

Re: net-snmp on HP-UX 11.31

Hi Matti,

 

Here is my routing table

 

Routing tables
Destination           Gateway            Flags Refs Interface  Pmtu
127.0.0.1             127.0.0.1          UH    0    lo0       32808
172.16.184.34         172.16.184.34      UH    0    lan0      32808
172.16.184.0          172.16.184.34      U     2    lan0       1500
127.0.0.0             127.0.0.1          U     0    lo0       32808
default               172.16.184.1       UG    0    lan0       1500

SyedHamid
Advisor

Re: net-snmp on HP-UX 11.31

And there is no firewall in between, it is just two different VLANS

 

172.16.5.57 -- > Solarwinds server

 

172.16.184.34 -- > HP UX box

Matti_Kurkela
Honored Contributor

Re: net-snmp on HP-UX 11.31

> Received SNMP packet(s) from UDP: [172.16.184.34]:64617->[0.0.0.0]:0

 

The messages say the received packet is coming from the HP-UX server itself, not from the Solarwinds server.

Is it possible that these messages are from your earlier testing on the HP-UX server locally?

 

If so, then it indicates that the net-snmp server has not received anything from the Solarwinds server at all.

Since you don't have IPFilter, the most likely explanation is that the packets from the Solarwinds server did not arrive to the HP-UX system. There must be something at the Solarwinds server or in the network between Solarwinds and HP-UX that stops the packets or routes them the wrong way.

 

To confirm this theory, the next step would be to take a network trace on the HP-UX system while running snmpwalk from the Solarwinds server. Here are instructions for taking a network trace using native HP-UX tools only:

http://www.compute-aid.com/nettl.html

 

Your filter file for the nettl trace should include the IP address of the Solarwinds server:

filter ip_saddr 172.16.5.57
filter ip_daddr 172.16.5.57

 

(If you have tcpdump installed on the HP-UX, "tcpdump -i lan0 -s1500 -n -vvv host 172.16.5.57" should produce a nice, readable trace from all the traffic between the HP-UX and the Solarwinds server.)

 

Start the trace on the HP-UX system, then run snmpwalk from the Solarwinds server to the HP-UX box. Then stop the trace and examine the results. If the trace contains nothing, that is because the HP-UX system has received nothing from the Solarwinds server. You now have confirmed that the problem is not in the net-snmp software, and most likely not in the HP-UX OS. If the NIC on the HP-UX box works normally otherwise, it is unlikely to be a hardware problem in the HP-UX box.

 

You might want to double-check the software firewall settings at the Solarwinds server. Since your net-snmpd on the HP-UX box is in a custom port (1161/UDP), the normal "allow SNMP traffic" rules might not cover traffic to UDP port 1161 on the HP-UX box. You may have add an extra rule to allow that.

 

If that is not the problem, the next step is to get help from your network administrator. You should be trying to get a similar network trace (also known as "traffic dump") at some point in the network that is roughly half-way between the Solarwinds server and the HP-UX server. The router that passes traffic from one VLAN to another (i.e. probably your default gateway, 172.16.184.1) might be a good first choice.

 

If that trace indicates that snmpwalk packets from the Solarwinds server are not reaching the point where you're tracing, the problem is between the Solarwinds server and that point; if you see snmpwalk packets at that point, the problem is somewhere between that point and the HP-UX box.

 

Keep splitting the problem area into smaller and smaller pieces by taking network traces in different points of the network while running snmpwalks from the Solarwinds server to the HP-UX, and you will eventually find something that should be passing the snmpwalk packets but isn't. There is the cause for your problem.

MK
SyedHamid
Advisor

Re: net-snmp on HP-UX 11.31

Hi Matti,

 

How you are well, sorry for such a delayed response. I set up the native packet capture on HP-UX as advised by you. I have captured packets using filters. Following are the filters i'm using

 

---------------------- SUBSYSTEM FILTERS IN EFFECT -----------------


        ---------------- LAYER  1 -----------------

        ---------------- LAYER  2 -----------------

        ---------------- LAYER  3 -----------------
        filter ip_saddr          172.16.184.34
        filter ip_saddr          172.16.5.57

        ---------------- LAYER  4 -----------------
        filter udp_sport         1161

        ---------------- LAYER  5 -----------------

---------------------- END SUBSYSTEM FILTERS -----------------------

 

And ,whats intresting is that the packets ( UDP : generated by solarwinds server ) are reaching the HP-UX server however the HP-UX server is not resposnding back. From the packet capture, I guess I now know the problem . The packet capture shows that port 1161 on HP-UX server upon snmp poll is sending data to tiself on a different higher port. i.e source and destination ip are the same.

 

 


vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvARPA/9000 NETWORKINGvvvvvvvvvvvvvvvvvvvvvvvvvv@#%
  Timestamp            : Mon Jun 17 BST 2013 09:54:06.614198
  Process ID           : 19332              Subsystem        : NS_LS_IP
  User ID ( UID )      : 0                  Trace Kind       : PDU OUT TRACE
  Device ID            : -1                 Path ID          : 0
  Connection ID        : 0
  Location             : 00123
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
================================ IP Header (outbound -- pid: 19332) ==========
Source:  172.16.184.34(B) Dest:  172.16.184.34(B)
       len:  84      ttl: 64    proto: 17     cksum:  0xb822      id: 0xf27a
     flags:  NONE    tos: 0x0  hdrlen: 20    offset:  0x0     optlen: 0
-------------------------------- UDP Header ----------------------------------
sport:   1161   -->   dport:  50947     data len: 56    chksum: 0x40
-------------------------------- NET-SNMP ------------------------------------
   0: 30 36 02 01 01 04 0b 46 75 73 41 66 69 4c 32 49  06.....FusAfiL2I
  16: 51 73 a2 24 02 02 57 1b 02 01 00 02 01 00 30 18  Qs.$..W.......0.
  32: 30 16 06 08 2b 06 01 02 01 01 02 00 06 0a 2b 06  0...+.........+.
  48: 01 04 01 bf 08 03 02 0e -- -- -- -- -- -- -- --  ................

SyedHamid
Advisor

Re: net-snmp on HP-UX 11.31

The issue is of routing the traffic. When I look at the packet header of UDP traffic from 161 it has a FLAG ( DF ) and the routing is  proper. However when it comes to net-snmp the routing is not proper.

 

Any help with creating and deleting routes manually would be appreciated.

 

Thanks

Syed

SyedHamid
Advisor

Re: net-snmp on HP-UX 11.31

The issue is now resolved. My special thanks to for his time and valuable feedback.

 

Coming to the issue after going through the documentation of NET-SNMP 5.7  the behaviour i was experiencing is a known "regression". I downgraded to Net-SNMP 5.6.1.1 and it works as a charm.

 

Once again thank you all for your awesome support.

 

Thanks

Syed