topic DNS monitoring in Operating System - HP-UX

DNS monitoring

2xyo — Wed, 06 Jan 2010 16:33:48 GMT

Dear,

I run BINDv920 B.11.11.01.014 on a HP-UX 11iv1.

Sometimes i have this messages in logs :
Jan 04 13:37:41.301 client: warning: client x.x.x.x#35699: no more recursive clients: quota reached

That's why i increase the value of recursive-clients in bind configuration. No problems here.

Now, I want to monitor the number of recursive clients (ideally in snmp) in real time.
The problem is that the command "rndc stats" does not give much information :

+++ Statistics Dump +++ (1020033800)
success 13
referral 0
nxrrset 0
nxdomain 10
recursion 22
failure 5
--- Statistics Dump --- (1020033800)

similarly for rndc status :
hpux1st->rndc status
number of zones: XX
debug level: 0
xfers running: 0
xfers deferred: 0
soa queries in progress: 0
query logging is ON
server is up and running

Are there other ways to monitor the number of recursive clients? (without update of Bind of course...)

I already find some scripts like this : http://forums.cacti.net/about302-15.html
I always need to calculate a delta. But it's quite dirty.

Any helps is appreciated

Re: DNS monitoring

VK2COT — Thu, 07 Jan 2010 09:10:43 GMT

Hello,

The message means that the name server has
reached its maximum number of simultaneous
recursive query requests. Any other recursive
queries will have to wait to be processed.

You should only ignore the message if you
do not care about performance or the service
levels you are providing to your clients.

If the DNS performance is of importance,
the choices are:

1. Increase the quota. For example,
in named.conf (default is 1000):

options {
recursive-clients 2000;
};

2. Find a way to reduce the load on your
DNS services. For example,

spread the query load across more servers,
limit recursive queries to some subnets,
eliminate searchlists,
and similar.

3. You can also apply a limit to the number
of concurrent TCP connections your name
server will process (for zone transfers and
TCP-based queries) with the "tcp-clients"
statement in named.conf. TCP connections
consume considerably more resources than UDP
because the host needs to track the state of
the TCP connection. The default limit is 100.

But before you do anything, you need to find out who is generating this excess traffic.

You should turn on query logging. That would
log each query that is being made and show
you who is generating the high load.

If you are looking for a monitoring tool, there are two interesting tools that can hook into DNS logs:

Afterglow:
http://afterglow.sourceforge.net/

DNStop:
http://dns.measurement-factory.com/tools/dnstop/

There are others too.

Finally, there is nothing wrong with using
tcpdump or other packet sniffer, getting a
trac of the packets to see how many are
incoming, from where, and for what are they
querying.

By the way, Is this an Internet- or
Intranet-facing DNS server?

If it is Internet-facing server,
I strongly suggest to disable recursive
queries to all but your own subnets.

You might be experiencing denial-of-service
attack. This is one of the possible reasons
for "no more recursive clients: quota
reached" errors.

b) Another possibility is that some of the
servers cannot reach internet (maybe another DNS server or mail server) and they are
asking for names that cannot be resolved when the Internet link is down. Partial
and unqualified names cause real problems
when used with search list and loss of
external connectivity.

Your "rndc status" seems incomplete.
I expect to see more lines in the report, like:

recursive clients: 50/35/1000
tcp clients: 48/100

Cheers,

VK2COT

Re: DNS monitoring

VK2COT — Thu, 07 Jan 2010 09:22:30 GMT

Forgot to add...

Another global option in named.conf:

recursing-file "bind.recurse";

This statement defines the file name to which
data will be written when the command
"rndc recursing" is issued. May be an absolute
or relative (to directory) path.

VK2COT

Re: DNS monitoring

2xyo — Thu, 07 Jan 2010 10:03:44 GMT

> Hello,
Hello,

To begin, thank you for your reply

> 1. Increase the quota. For example,
> in named.conf (default is 1000):
>
> options {
> recursive-clients 2000;
> };
As I indicated in my first post, I have already done that.

> 2. Find a way to reduce the load on your
> DNS services. For example,
I just have many users :-)
I already use filters.

> 3. You can also apply a limit to the number
> of concurrent TCP connections your name
> [...]
Je n'ai pas pensÃ© Ã cette situation. Comment monitorer le nombre de session tcp avec un bind 9.2.0 ?

> But before you do anything, you need to find out who is generating this excess traffic.
>
> You should turn on query logging. That would
> log each query that is being made and show
> you who is generating the high load.
Already done. I have pics of legitimate traffic.

> If you are looking for a monitoring tool, there are two interesting tools that can hook into DNS logs:

> DNStop:
> http://dns.measurement-factory.com/tools/dnstop/
Everyone knows this site ;-)

> There are others too.
>
> Finally, there is nothing wrong with using
> tcpdump or other packet sniffer, getting a
> trac of the packets to see how many are
> incoming, from where, and for what are they
> querying.
>
> By the way, Is this an Internet- or
> Intranet-facing DNS server?
Intranet only of course !

> b) Another possibility is that some of the
> servers cannot reach internet (maybe another DNS server or mail server) and they are
>[...]
This is another possibility. However, we are directly connected to our ISP ...

> Your "rndc status" seems incomplete.
> I expect to see more lines in the report, like:
>
> recursive clients: 50/35/1000
> tcp clients: 48/100
No, my "rndc status" is unfortunately correct! It's a rndc status of a Bind 9.2.0.

This is another rndc status from test server :
version: 9.5.1-P3
number of zones: XX
debug level: 0
xfers running: 0
xfers deferred: 0
soa queries in progress: 0
query logging is ON
recursive clients: 0/0/1000
tcp clients: 0/100
server is up and running

In this, it's easy to have in real time the number of recursive client. BUT, and ths problem is here for me, it's more difficult witrh Bind 9.2.0 because the "rndc status" return only :
number of zones: 27
debug level: 0
xfers running: 0
xfers deferred: 0
soa queries in progress: 0
query logging is ON
server is up and running

Source code from bind-9.2.0\bin\named\server.c:2863
n = snprintf((char *)isc_buffer_used(text),
isc_buffer_availablelength(text),
"number of zones: %d\n"
"debug level: %d\n"
"xfers running: %d\n"
"xfers deferred: %d\n"
"soa queries in progress: %d\n"
"query logging is %s\n"
"server is up and running",
zonecount, ns_g_debuglevel, xferrunning, xferdeferred,
soaqueries, server->log_queries ? "ON" : "OFF");

Now I'm not trying to find a solution in the logs after the number of recursive clients that there was a BUT I want to see a solution in real time the number of recursive clients.

> Cheers,
Thank you again for your reply

Re: DNS monitoring

2xyo — Thu, 07 Jan 2010 10:08:09 GMT

> Forgot to add...
>
> Another global option in named.conf:
>
> recursing-file "bind.recurse";
>
> This statement defines the file name to which
> data will be written when the command
> "rndc recursing" is issued. May be an absolute
> or relative (to directory) path.
It's not implemented in Bind 9.2.0 :
Usage: rndc [-c config] [-s server] [-p port] [-y key] [-V] command

command is one of the following:

reload Reload configuration file and zones.
reload zone [class [view]]
Reload a single zone.
refresh zone [class [view]]
Schedule immediate maintenance for a zone.
reconfig Reload configuration file and new zones only.
stats Write server statistics to the statistics file.
querylog Toggle query logging.
dumpdb Dump cache(s) to the dump file (named_dump.db).
stop Save pending updates to master files and stop the server.
halt Stop the server without saving pending updates.
trace Increment debugging level by one.
trace level Change the debugging level.
notrace Set debugging level to 0.
flush Flushes all of the server's caches.
flush [view] Flushes the server's cache for a view.
status Display status of the server.
*restart Restart the server.

* == not yet implemented
Version: named 9.2.0

But I agree with you that this option exists in recent versions of bind.
Thank you anyway