topic Re: MRTG and snmp woes in Operating System - HP-UX

MRTG and snmp woes

Geoff Wild — Tue, 09 May 2006 14:47:55 GMT

Currently monitoring about 60 servers with mrtg.

Mrtg running on AIX 5.2 server.
Monitoring over 40 HP-UX servers.

for some reason, 4 HP servers are not displaying...

In /var/adm/mrtg.log

2006-05-09 13:35:01 -- Started mrtg with config '/htdocs/mambo/mrtg/mrtg.cfg'
2006-05-09 13:35:02 -- SNMP Error:
Received SNMP response with error code
error status: noSuchName
index 1 (OID: 1.3.6.1.4.1.11.2.3.1.1.15.0)
SNMPv1_Session (remote host: "server7.mydomain.com" [192.168.2.144].161)
community: "public"
request ID: 236718950
PDU bufsize: 8000 bytes
timeout: 2s
retries: 5
backoff: 1)
at /usr/local/mrtg-2/bin/../lib/mrtg2/SNMP_util.pm line 490
2006-05-09 13:35:02 -- SNMPGET Problem for .1.3.6.1.4.1.11.2.3.1.1.15.0 .1.3.6.1.4.1.11.2.3.1.1.13.0 sysUptime sysName on public@server7.mydomain.com::::::v4only
at /usr/local/mrtg-2/bin/mrtg line 1846
2006-05-09 13:35:02 -- SNMP Error:
Received SNMP response with error code
error status: noSuchName
index 1 (OID: 1.3.6.1.4.1.11.2.3.1.1.16.0)
SNMPv1_Session (remote host: "server7.mydomain.com" [192.168.2.144].161)
community: "public"
request ID: 236718951
PDU bufsize: 8000 bytes
timeout: 2s
retries: 5
backoff: 1)
at /usr/local/mrtg-2/bin/../lib/mrtg2/SNMP_util.pm line 490
2006-05-09 13:35:02 -- SNMPGET Problem for .1.3.6.1.4.1.11.2.3.1.1.16.0 .1.3.6.1.4.1.11.2.3.1.1.14.0 sysUptime sysName on public@server7.mydomain.com::::::v4only
at /usr/local/mrtg-2/bin/mrtg line 1846

That server is part of a cluster - the other node is fine and snmpd.conf is identical...

That server is 11.11, but also on 3 11.23 servers I get the same response...

On one of those servers (rp4440) it has an identical prod system - which works.

Same OS OE and yet 1 works and the other doesn't...

snmpwalk appears to work...

Here what s in my mrtg.cfg:

#------------------------------------------

Target[pc0710.cpu]:.1.3.6.1.4.1.11.2.3.1.1.15.0&.1.3.6.1.4.1.11.2.3.1.1.13.0:public@server7.mydomain.com + .1.3.6.1.4.1.11.2.3.1.1.16.0&.1.3.6.1.4.1.11.2.3.1.1.14.0:public@server7.mydomain.com
Title[pc0710.cpu]: CPU Load on server7.mydomain.com
MaxBytes[pc0710.cpu]: 100
Ylegend[pc0710.cpu]: CPU Load
ShortLegend[pc0710.cpu]: %
PageTop[pc0710.cpu]:

CPU Load on server7

Options[pc0710.cpu]: integer, growright, nopercent
WithPeak[pc0710.cpu]: wmy
LegendO[pc0710.cpu]: User/System:
LegendI[pc0710.cpu]: Idle/Nice:
Legend1[pc0710.cpu]: Idle/Nice CPU Time
Legend2[pc0710.cpu]: User/System CPU Time
Legend3[pc0710.cpu]: Maximal Idle/Nice CPU Time
Legend4[pc0710.cpu]: Maximal User/System CPU Time

Target[pc0710.mem]: .1.3.6.1.4.1.11.2.3.1.1.7.0&.1.3.6.1.4.1.11.2.3.1.1.8.0:public@server7.mydomain.com
Title[pc0710.mem]: Memory on server7.mydomain.com
MaxBytes[pc0710.mem]: 1000000000
Ylegend[pc0710.mem]: Memory Usage
ShortLegend[pc0710.mem]:
PageTop[pc0710.mem]:

Memory on server7

Options[pc0710.mem]: integer, gauge, growright, nopercent
kilo[pc0710.mem]: 1024
WithPeak[pc0710.mem]: wmy
kMG[pc0710.mem]: k,M,G
LegendO[pc0710.mem]: Total:
LegendI[pc0710.mem]: Free:
Legend1[pc0710.mem]: Free System Memory
Legend2[pc0710.mem]: Total Available System Memory
Legend3[pc0710.mem]: Maximal Free System Memory
Legend4[pc0710.mem]: Maximal Available System Memory

#------------------------------------------

Any ideas?

Rgds...Geoff

Re: MRTG and snmp woes

Jannik — Tue, 09 May 2006 15:13:18 GMT

Hey Geoff,

Then you say that the configuration files are identical you meen identical "diff". So this is probably a wild guess:
DNS is it working correctly?
compare /etc/nsswitch.conf
compare /etc/resolv.conf

nslookup work both ways (normal and reverse)

The duplex could not be an issue but anyway "lanadmin -sx <1...>"

Hope you get it to work!
Best Regards,
Jannik

Re: MRTG and snmp woes

Geoff Wild — Tue, 09 May 2006 15:21:40 GMT

diff is clean:

# diff /etc/nsswitch.conf /tmp/nsswitch.conf

# diff /etc/resolv.conf /tmp/resolv.conf

# diff /etc/snmpd.conf /tmp/snmpd.conf

speed/duplex the same...

Rgds...Geoff

Re: MRTG and snmp woes

Steven E. Protter — Tue, 09 May 2006 15:34:51 GMT

Shalom Geoff,

How goes it?

I'm thinking about corrupt installation. swverify the depot components and consider remove/reinstall of non depot componenets on effected servers.

I'm also thinking of switch configuration blocking the traffic. Check with the Cisco admins.

I'm thinking of putting a packet sniffer like tcpdmp or ethereal and see if there is a networking problem.

SEP

Re: MRTG and snmp woes

Geoff Wild — Wed, 10 May 2006 08:35:11 GMT

Hey SEP!

Nope - not a networking issue - as I can snmpwalk the machines.

Nothing interesting in the swverify (IE same on working and non working systems).

Rgds...Geoff

Re: MRTG and snmp woes

Ralph Grothe — Thu, 11 May 2006 03:44:55 GMT

Hm, you said snmpwalk appears to work.
Can you snmpwalk sysName and sysUptime on the nodes that fail?
Maybe a broken or not loaded MIB?
You could also try to run the MRTG script of the manager in the Perl debugger and especially around line # 490 in SNMP_util.pm inspect how variables are set (OID mapping, contents of PDU etc., use |x in the debugger to see the full object)

Re: MRTG and snmp woes

Geoff Wild — Thu, 11 May 2006 08:31:10 GMT

I can't seem to snmpwalk sysName on any node...

What I can do is:

On a node that works:

# snmpwalk -v2c -c public svr1104
iso.3.6.1.2.1.1.1.0 = STRING: "HP-UX svr1104 B.11.23 U 9000/800 3394666127"
iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.11.2.3.2.3
iso.3.6.1.2.1.1.3.0 = Timeticks: (305990635) 35 days, 9:58:26.35
iso.3.6.1.2.1.1.4.0 = ""
iso.3.6.1.2.1.1.5.0 = STRING: "svr1104.mydomain.com"
iso.3.6.1.2.1.1.6.0 = ""
iso.3.6.1.2.1.1.7.0 = INTEGER: 72
iso.3.6.1.2.1.1.8.0 = Timeticks: (0) 0:00:00.00
iso.3.6.1.2.1.2.1.0 = INTEGER: 13
iso.3.6.1.2.1.2.2.1.1.1 = INTEGER: 1
iso.3.6.1.2.1.2.2.1.1.2 = INTEGER: 2
iso.3.6.1.2.1.2.2.1.1.3 = INTEGER: 3
iso.3.6.1.2.1.2.2.1.1.4 = INTEGER: 4
iso.3.6.1.2.1.2.2.1.1.5 = INTEGER: 5
iso.3.6.1.2.1.2.2.1.1.6 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.1.7 = INTEGER: 7
iso.3.6.1.2.1.2.2.1.1.8 = INTEGER: 8
iso.3.6.1.2.1.2.2.1.1.9 = INTEGER: 9
iso.3.6.1.2.1.2.2.1.1.10 = INTEGER: 10
iso.3.6.1.2.1.2.2.1.1.11 = INTEGER: 11
iso.3.6.1.2.1.2.2.1.1.12 = INTEGER: 12
iso.3.6.1.2.1.2.2.1.2.1 = STRING: "lan0 HP PCI-X 1000Base-T Release B.11.23.0505.01"
iso.3.6.1.2.1.2.2.1.2.2 = STRING: "lan1 HP PCI-X 1000Base-T Release B.11.23.0505.01"
iso.3.6.1.2.1.2.2.1.2.3 = STRING: "lan2 HP PCI-X 1000Base-T Release B.11.23.0505.01"
iso.3.6.1.2.1.2.2.1.2.4 = STRING: "lan3 HP PCI-X 1000Base-T Release B.11.23.0505.01"
iso.3.6.1.2.1.2.2.1.2.5 = STRING: "lan4 HP PCI-X 1000Base-T Release B.11.23.0505.01"
iso.3.6.1.2.1.2.2.1.2.6 = STRING: "lan5 HP PCI-X 1000Base-T Release B.11.23.0505.01"
iso.3.6.1.2.1.2.2.1.2.7 = STRING: "lan7 HP PCI-X 1000Base-T Release B.11.23.0505.01"
iso.3.6.1.2.1.2.2.1.2.8 = STRING: "lan8 HP PCI-X 1000Base-T Release B.11.23.0505.01"
iso.3.6.1.2.1.2.2.1.2.9 = STRING: "lan9 HP PCI-X 1000Base-T Release B.11.23.0505.01"
iso.3.6.1.2.1.2.2.1.2.10 = STRING: "lan10 HP PCI-X 1000Base-T Release B.11.23.0505.01"
iso.3.6.1.2.1.2.2.1.2.11 = STRING: "lan6 HP PCI 1000Base-T Release B.11.23.0505.01"
iso.3.6.1.2.1.2.2.1.2.12 = STRING: "software loopback"
iso.3.6.1.2.1.2.2.1.3.1 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.2 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.3 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.4 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.5 = INTEGER: 6

On a node that isn't working:

# snmpwalk -v2c -c public svr1103
iso.3.6.1.2.1.1.1.0 = STRING: "HP-UX svr1103 B.11.11 U 9000/800 2221170464"
iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.11.2.3.2.3
iso.3.6.1.2.1.1.3.0 = Timeticks: (16699510) 1 day, 22:23:15.10
iso.3.6.1.2.1.1.4.0 = ""
iso.3.6.1.2.1.1.5.0 = STRING: "svr1103.mydomain.com"
iso.3.6.1.2.1.1.6.0 = ""
iso.3.6.1.2.1.1.7.0 = INTEGER: 72
iso.3.6.1.2.1.1.8.0 = Timeticks: (0) 0:00:00.00
iso.3.6.1.2.1.2.1.0 = INTEGER: 4
iso.3.6.1.2.1.2.2.1.1.1 = INTEGER: 1
iso.3.6.1.2.1.2.2.1.1.2 = INTEGER: 2
iso.3.6.1.2.1.2.2.1.1.3 = INTEGER: 3
iso.3.6.1.2.1.2.2.1.1.4 = INTEGER: 4
iso.3.6.1.2.1.2.2.1.2.1 = STRING: "lan0 HP PCI Core I/O 1000Base-T Release B.11.11.18"
iso.3.6.1.2.1.2.2.1.2.2 = STRING: "lan1 HP PCI-X 1000Base-SX Release PHNE_31153 B.11.11.08"
iso.3.6.1.2.1.2.2.1.2.3 = STRING: "lan2 HP PCI-X 1000Base-SX Release PHNE_31153 B.11.11.08"
iso.3.6.1.2.1.2.2.1.2.4 = STRING: "software loopback"
iso.3.6.1.2.1.2.2.1.3.1 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.2 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.3 = INTEGER: 6
iso.3.6.1.2.1.2.2.1.3.4 = INTEGER: 24
iso.3.6.1.2.1.2.2.1.4.1 = INTEGER: 1500
iso.3.6.1.2.1.2.2.1.4.2 = INTEGER: 1500
iso.3.6.1.2.1.2.2.1.4.3 = INTEGER: 1500
iso.3.6.1.2.1.2.2.1.4.4 = INTEGER: 4136
iso.3.6.1.2.1.2.2.1.5.1 = Gauge32: 1000000000

Re: MRTG and snmp woes

Geoff Wild — Thu, 11 May 2006 10:12:47 GMT

I think Ralph is onto something - about mibs not being loaded....

For example, on the node that does NOT work,

# snmpwalk -v1 -c public pc0710 |wc
42 175 1672

only has that much info - on the other - it just spews and spews

# snmpwalk -v1 -c public pc0711 |wc
Error: OID not increasing: iso.3.6.1.2.1.6.13.1.1.127.0.0.1.57982.127.0.0.1.5798
>= iso.3.6.1.2.1.6.13.1.1.127.0.0.1.1476.0.0.0.0.0

4767 23872 271452

How to get mibs loaded and/or check that they are loading?

Rgds...Geoff

Re: MRTG and snmp woes

rob nolen — Thu, 11 May 2006 17:30:12 GMT

Geoff,

Sorry for offtopic, but can you paste your full mrtg config for one system? I've been looking all over the place for a way to set this up for my hpux systems and have been kind of lost.

Thanks... Rob

Re: MRTG and snmp woes

Geoff Wild — Fri, 12 May 2006 08:29:58 GMT

Rob - see:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=707417

Rgds...Geoff

Re: MRTG and snmp woes

rob nolen — Fri, 12 May 2006 11:04:16 GMT

Hey Geoff,

Thanks for the reply... last ot post for this thread I promise. I'm trying to figure out what those counters mean. When I snmpwalk them, it just says counter32 and returns a value, but I cant make heads or tails of it.

What am I getting back, and are there any docs that explain them?

Thanks,

ROb

Re: MRTG and snmp woes

Geoff Wild — Fri, 02 Jun 2006 08:00:06 GMT

Solution was to restart snmp on the HP servers.

For HPUX 11.23:

/sbin/init.d/SnmpHpunix stop
/sbin/init.d/SnmpHpunix start
/sbin/init.d/SnmpMib2 stop
/sbin/init.d/SnmpMib2 start

For 11.11, just re-run the daemon:

/usr/sbin/snmpd

Rgds...Geoff