topic Re: problem with nagios client config in Operating System - Linux

problem with nagios client config

Sunny Jaisinghani — Thu, 12 Nov 2009 11:19:26 GMT

Hello All,

I have configured a nagios client (Red Hat Enterprise Linux AS release 4 )

when i run the command it gives me an error

# /usr/local/nagios/libexec/check_nrpe -n -H
CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages.

On remote(client) server, i get foll in messages

Nov 11 07:53:29 nrpe[29524]: Host 172.16.128.54 is not allowed to talk to us!
Nov 11 07:54:45 nrpe[25475]: Host 172.16.128.54 is not allowed to talk to us!
Nov 11 07:55:23 nrpe[25643]: Host 172.16.128.54 is not allowed to talk to us!

what am i missing here.....

I have checked following things

1. ping is successful between nagios server and nagios client

2. [root@cleint ~]# netstat -an | grep 5666
tcp 0 0 0.0.0.0:5666 0.0.0.0:* LISTEN
[root@cleint ~]# netstat -lp | grep 5666

3. [root@client ~]# telnet 5666
Trying 10.18.24.35...
Connected to xxxxxxx (10.18.24.35).
Escape character is '^]'.
Connection closed by foreign host.

4. [root@server ~]# telnet 5666
Trying 10.18.24.35...
Connected to xxxxxxx (10.18.24.35).
Escape character is '^]'.
Connection closed by foreign host.

5. [root@client]# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

6. # ldd /usr/local/nagios/nrpe
libssl.so.4 => /lib/libssl.so.4 (0x00168000)
libcrypto.so.4 => /lib/libcrypto.so.4 (0x00cee000)
libnsl.so.1 => /lib/libnsl.so.1 (0x00c6e000)
libwrap.so.0 => /usr/lib/libwrap.so.0 (0x009fe000)
libc.so.6 => /lib/tls/libc.so.6 (0x00734000)
libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2 (0x00dd9000)
libkrb5.so.3 => /usr/lib/libkrb5.so.3 (0x00101000)
libcom_err.so.2 => /lib/libcom_err.so.2 (0x00cdf000)
libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0x0019e000)
libresolv.so.2 => /lib/libresolv.so.2 (0x009c8000)
libdl.so.2 => /lib/libdl.so.2 (0x00861000)
libz.so.1 => /usr/lib/libz.so.1 (0x009b6000)
/lib/ld-linux.so.2 (0x00716000)

Also, please note, i was stuck up while configuring nagios plugins on the client. On searching a workaround i came across following post

./configure appears to hang
If you find that the configure script appears to hang on this line:
checking for redhat spopen problem...

Then you probably have a badly configured DNS server. This part of configure is testing for a pthread problem in Bind that is a kernel problem on some Red Hat derived versions of Linux (around kernel 2.6.9-11). It runs 10 x 100 nslookup calls to see if your kernel has this problem. If it does, then at least 1 of those calls will fail. Failure rate could be anywhere between 1% and 50%.

To force the workaround and ignore the test, run ./configure with the --enable-redhat-pthread-workaround switch.

You can run ./configure with --disable-redhat-pthread to ignore this test.

Hence i configured nagios plugins with the option --enable-redhat-pthread-workaround

Thanks for your help

Sunny

Re: problem with nagios client config

Ralph Grothe — Thu, 12 Nov 2009 11:57:13 GMT

Hi Sunny,

> Nov 11 07:55:23 nrpe[25643]: Host 172.16.128.54 is not allowed to talk to us!

Have you configured to have your nrpe started through xinetd?

Then probably you have an only_from clause in it that doesn't contain the IP of your Nagios Server.
You can add a list of allowed host or network IPs (CIDR notation honored). See "man xinetd.conf" and serach for "only_from" for details.
Don't forget to include the loopback and IP of the localhost to the list so that you can run check_nrpe checks locally for testing as well.

After having edited /etc/xinetd.d/nrpe
send the xinetd a SIGHUP
(# pkill -1 xinetd)
and try check_nrpe with any of the check commands you have defined.
It's easiest to first start locally,
e.g.

$ /usr/local/nagios/libexec/check_nrpe -H localhost -c check_load

If that works run a check_nrpe check from your Nagios server, specifying hostname or IP of remote NRPE host after the -H.

If already the local check fails or hangs
look what's wrong mostly in

# grep nrpe /var/log/messages | tail

You may also want to raise the verbosity of nrpe by setting debug=1 in nrpe.cfg

Re: problem with nagios client config

Sunny Jaisinghani — Fri, 13 Nov 2009 05:37:21 GMT

i started nrpe with the foll command

/usr/local/nagios/nrpe -c /usr/local/nagios/etc/nrpe.cfg -d

After doing this, things were not working for me. on searching the web, and seeing the suggestions, i also configured nrpe under xinetd and restarted xinetd to no success.

locally the check_xxx commands work

@client
# /usr/local/nagios/libexec/check_load -w 15,10,5 -c 30,25,20
OK - load average: 3.01, 2.25, 0.97|load1=3.010;15.000;30.000;0; load5=2.250;10.000;25.000;0; load15=0.970;5.000;20.000;0;

@server
# /usr/local/nagios/libexec/check_nrpe -H client
CHECK_NRPE: Error - Could not complete SSL handshake.

# /usr/local/nagios/libexec/check_nrpe -n -H client
CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages.

I have started the debug mode

Re: problem with nagios client config

Sunny Jaisinghani — Fri, 13 Nov 2009 05:40:14 GMT

Nov 12 22:14:28 nrpe[17221]: Caught SIGHUP - restarting...
Nov 12 22:14:28 nrpe[17221]: Warning: Daemon is configured to accept command arguments from clients!
Nov 12 22:14:28 nrpe[17221]: Listening for connections on port 5666
Nov 12 22:14:28 nrpe[17221]: Allowing connections from: 127.0.0.1
Nov 12 22:15:08 nrpe[22596]: Host 172.16.128.54 is not allowed to talk to us!
Nov 12 22:15:47 nrpe[22598]: Host 172.16.128.54 is not allowed to talk to us!
Nov 12 22:15:57 nrpe[22600]: Host 172.16.128.54 is not allowed to talk to us!

Re: problem with nagios client config

Sunny Jaisinghani — Fri, 13 Nov 2009 05:42:34 GMT

This is how /etc/xinetd.d/nrpe looks like

# default: on
# description: NRPE
service nrpe
{
flags = REUSE
socket_type = stream
wait = no
user = nagios
server = /usr/local/nagios/nrpe
server_args = -c /usr/local/nagios/etc/nrpe.cfg --inetd
log_on_failure += USERID
disable = no
only_from = 172.16.128.54,127.0.0.1,10.18.24.35
}

Server IP:172.16.128.54
Client IP:10.18.24.35

Re: problem with nagios client config

Ralph Grothe — Fri, 13 Nov 2009 12:51:21 GMT

Hm, the first way you had started it as a standalone daemon with the -d option.

Are you sure you have no tcpwrapper in effect?

If you for instance run an nmap scan against the port from your Nagios server does it show the port as "open" or do you get a "closed" (or even "filtered" (like when routing doesn't work or a firewall drops the packets)?

e.g.

$ nmap -P0 -p 5666 munsaz|grep -A1 ^PORT
PORT STATE SERVICE
5666/tcp open unknown

However, when you start it through xinetd and when your /etc/xinetd.d/nrpe is applied,
then I'm not sure if this line is syntacticly correct:

nly_from = 172.16.128.54,127.0.0.1,10.18.24.35

I have the list elements (i.e. IP addresses) always delimited by a mere whitespace character but never commas.
What happens when you substitute the commas by space and restart your xinetd (or signal it HUP)?

Re: problem with nagios client config

Sunny Jaisinghani — Mon, 16 Nov 2009 11:21:36 GMT

What do i derive from this?

[root@server ~]# nmap -P0 -p 5666 client|grep -A1 ^PORT
PORT STATE SERVICE
5666/tcp open unknown

I replaced the comma in /etc/xinetd.d/nrpe with white space and restarted nrpe ... but it did not make any difference.

Should i first kill the process
nagios 17221 1 0 Nov11 ? 00:00:01 /usr/local/nagios/nrpe -c /usr/local/nagios/etc/nrpe.cfg -d

and then restart xinetd

Thanks for your efforts

Sunny

Re: problem with nagios client config

Ralph Grothe — Mon, 16 Nov 2009 13:58:10 GMT

> Should i first kill the process
nagios 17221 1 0 Nov11 ? 00:00:01 /usr/local/nagios/nrpe -c /usr/local/nagios/etc/nrpe.cfg -d

> and then restart xinetd

Sunny, it looks you are confusing the possible modes of operating the nrpe here.
If you fire it up with the "-d" option
(as the process is listed in your process table)
this means that it is run in daemon mode and doesn't require to be started through the internet super daemon (x)inetd.
As long as this process is running it doesn't matter whatever you configure in either /etc/inetd.conf (classic inetd) or /etc/xinetd.d/nrpe (xinetd) or how often you restart the (x)inetd.
Probably, since the port 5666 is already in use by your PID 17221 you just have received an error message from (x)inetd in /var/log/messages that it couldn't start the nrpe service.

So you need to decide whether you want to run nrpe standalone in which case you most likely have to start nrpe through some init script like /etc/init.d/nrpe restart (or on RHEL "service nrpe restart"),
or if you want it to be started through xinetd.
In the latter case you need to change the option from "-d" to "-i" (stands for inetd run), and you should have a line like this in the nrpe file of your xinetd's config:

$ grep server_args /etc/xinetd.d/nrpe
server_args = -c /etc/nagios/nrpe.cfg -i

When you decide for the xinetd operation the advantage is that you can always make changes to your nrpe's config file /etc/nagios/nrpe.cfg (e.g. add new commands)
without being required to restart xinetd or care for anything other to be restarted since xinetd will restart the nrpe daemon on your behalf whenever a new connection is targeting port 5666 and at the time will reread /etc/nagios/nrpe.cfg

However, when you go the xinetd way
you definetly must first kill the init started nrpe process (i.e. "kill 17221" or under RHEL "service nrpe stop").
Then you also should run "chkconfig nrpe off" and recheck with "chkconfig --list nrpe".
Only after this should you "service xinetd reload".

If on the other hand you decide to run nrpe standalone you must make sure that you have this line in xinetd's nrpe config file:

$ grep disable /etc/xinetd.d/nrpe
disable = yes

and make sure to have

# chkconfig --level 345 nrpe on

I hope this wasn't all too confusing.

Re: problem with nagios client config

Sunny Jaisinghani — Mon, 16 Nov 2009 15:21:47 GMT

It was not confusing :-)

I want to run nrpe in standalone manner as it is running in all the clients.
So i will disabled nrpe through xinetd.

BUt, Please note i was doing the same thing before. SInce it wasn't working for me i configured nrpe in xinetd.

NOw going back to standalone mode of nrpe, i m still at the same error/issue.

PLease suggest further.

Sunny

Re: problem with nagios client config

Ralph Grothe — Tue, 17 Nov 2009 11:05:11 GMT

Hi again,

Ok, now that I know that you are running nrpe in daemon mode you have to be aware that the nrpe.cfg is its main configuration file, and that unlike when run under (x)inetd, all entries therein do apply.

So, please check what your allowed_hosts line reads.
(as I only run my nrpe hosts via (x)inted in my nrpe.cfg files this is commented out)

Here's what my entry (including the explanitory comments) typically looks like:

$ grep -B12 -A1 allowed_hosts /etc/nagios/nrpe.cfg

# ALLOWED HOST ADDRESSES
# This is an optional comma-delimited list of IP address or hostnames
# that are allowed to talk to the NRPE daemon.
#
# Note: The daemon only does rudimentary checking of the client's IP
# address. I would highly recommend adding entries in your /etc/hosts.allow
# file to allow only the specified host to connect to the port
# you are running this daemon on.
#
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd

#allowed_hosts=127.0.0.1,192.168.0.2

As you can see, here (opposed to the similar thing in /etc/xinetd.d/nrpe) the allowed hosts lists is comma separated.

Please, make sure that the list contains at least the IP addresses of localhost (127.0.0.1), the IP address which equals the nagios substituted $HOSTADDRESS$ of this nrpe host when the server runs a check on it, and last but most importantly the IP address of your Nagios server.

After you have applied any changes to the file restart the nrpe,
e.g.

# service nrpe restart

or

# service nrpe stop && service nrpe start

You may want to parse the PID to make sure that it has changed before and after the restarting.

e.g.

# pgrep -fl -P1 nrpe

You may also have a look at the nrpe host's syslogd logfile to see if during the restart it encountered any problems.

e.g.

# grep nrpe /var/log/messages | tail -20

Then try to run an nrpe check locally first.

e.g.

$ /usr/local/nagios/libexec/check_nrpe -H 127.0.0.1 -c check_load

(note, your path to the plug-ins dir may vary
e.g.
/usr/lib/nagios/plugins/check_nrpe
or
/usr/lib64/nagios/plugins/check_nrpe
)

Did this local check work?

Re: problem with nagios client config

Sunny Jaisinghani — Tue, 17 Nov 2009 13:02:46 GMT

I changed nrpe.cfg as said. Now it has 3 IP addresses. loopback, server and client

[root@client etc]# grep allowed nrpe.cfg
# that are allowed to talk to the NRPE daemon.
allowed_hosts=127.0.0.1,172.16.128.54,10.18.24.35

I sent a SIGHUP to nrpe PID. Logs from syslog

Nov 17 05:27:27 client nrpe[4885]: Host 172.16.128.54 is not allowed to talk to us!
Nov 17 05:29:33 client nrpe[17221]: Caught SIGHUP - restarting...
Nov 17 05:29:33 client nrpe[17221]: Warning: Daemon is configured to accept command arguments from clients!
Nov 17 05:29:33 client nrpe[17221]: Listening for connections on port 5666
Nov 17 05:29:33 client nrpe[17221]: Allowing connections from: 127.0.0.1,172.16.128.54,10.18.24.35
Nov 17 05:29:56 client nrpe[11456]: Error: Could not complete SSL handshake. 1
Nov 17 05:29:58 client nrpe[11458]: Error: Could not complete SSL handshake. 1

local check fails

[root@client etc]# /usr/local/nagios/libexec/check_nrpe -H 127.0.0.1
Connection refused by host

Re: problem with nagios client config

Ralph Grothe — Tue, 17 Nov 2009 13:34:22 GMT

> Nov 17 05:29:33 client nrpe[17221]: Listening for connections on port 5666
> Nov 17 05:29:33 client nrpe[17221]: Allowing connections from: 127.0.0.1,172.16.128.54,10.18.24.35

At least it is now accepting connections from the said IP addresses.

> Nov 17 05:29:33 client nrpe[17221]: Warning: Daemon is configured to accept command arguments from clients!

This warning reminds you that you most likely have configured your nrpe to accept command args which can pose a security risk.
If you don't need to define nrpe commands that use arguments (and this is almost never necessary) then you should better deactivate them and set this directive to 0:

$ grep ^dont_blame /etc/nagios/nrpe.cfg
dont_blame_nrpe=0

> Nov 17 05:29:56 client nrpe[11456]: Error: Could not complete SSL handshake. 1

Now, you only have an SSL issue.

You need to know with what options your nrpe daemon was started.

Either look it up in the process table:

what does a

$ pgrep -fl nrpe

for instance show?

Or look into the init script that starts it.

In mine, which was part of a Dag Wieers nagios-nrpe RPM package the start block looks like this:

prog="nrpe"
desc="Nagios NRPE daemon"

start() {
echo -n $"Starting $desc ($prog): "
daemon $prog -c "$CONFIG" -d
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && touch /var/lock/subsys/$prog
return $RETVAL
}

So there should be a process like:

/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -d

Btw., have you installed from a prebuilt RPM or compiled NRPE yourself?
If the latter applies, have you compiled in SSL support?

Provided your nrpe binary has compiled-in SSL support and was started as assumed above (i.e. without the "-n" optione) then you must use a check_nrpe client that likewise has built-in SSL support (remember the possible platform difference and thus check_nrpe binaries between remote nrpe host and your nagios server),
and it also must be invoked without the "-n" option or vice versa.

Re: problem with nagios client config

Sunny Jaisinghani — Tue, 17 Nov 2009 14:06:38 GMT

Yes Ralph... I had compiled NRPE myself.

i restarted nrpe daemon with -n option and it has resolved my problem.

I had tried all these option before also and i wonder how i missed to see this. Probably i was confused in trying so many options.

[root@server ~]# /usr/local/nagios/libexec/check_nrpe -n -H client
NRPE v2.12

Thanks for helping me clean up the mess in my brain :-)
Appreciate your efforts towards good, meaningful and detailed replies.

Sunny