Boosting inetd performance

[Ralph Grothe]

Hello,

we have complaints from clients that they sometimes experience temporal interrupts of either their DB client sessions or telnet sessions (as far as DBAs are concerned, whom I couldn't desuade from using insecure r*-services despite the SSH servers that listen on our DB servers).

Though the network load is considerable I haven't seen any real LAN bottlenecks that could be blamed for.

My suspicion is rather that the poor inetd superserver who is burdened with firing up all those DB connection servers simply is brought to its knees sometimes when there are too many simultanuous connect requests from clients.
I'm quite convinced that inetd was never meant to supervise so many services, and that the exploitive use of inetd by the DB application developers is a sheer maldesign.

To show you what I mean let me just count the services they (the DB application developers) registered for servicing by inetd.
I know that their entries start from line 190, and those also appear in inetd.conf:

# awk 'NR>190&&$1!~/^#/&&!/^[:space:]*$/' /etc/services|wc -l
162

Boy, 162 services, and they all are being used.

If I had to develop a listening server for DB connections/queries etc. I would implement it as a standalone server (just like sshd or others), and probably bless it with either preforking or multiplexing capabilities which is self-adapting to the load.

But I'm surely not in the position to have the faintest influence on design suggestions (it probably would be too late anyway).
Despite the blame is sought on the sysadmin part.

Is there anything I could do to increase performance of inetd?

Or if it isn't inetd, where else could I seek for improvement?

Regards

Ralph

[Steven E. Protter]

You need to collect some data so you know what the problem actually is.

You might need to tune some kernel parameters. You might have temporary network or even disk overloads.

Try the scripts I'm attaching. I just fixed an annoying error in them.

For more information, see this performance tuning document.

http://www2.itrc.hp.com/service/cki/search.do?category=c0&docType=Security&docType=Patch&docType=EngineerNotes&docType=BugReports&docType=Hardware&docType=ReferenceMaterials&docType=ThirdParty&searchString=UPERFKBAN00000726&search.y=8&search.x=28&mode=id&admit=-1335382922+1058272787313+28353475&searchCrit=allwords

SEP

[Michael Steele_2]

With all of those telnet sessions some are going to loose their device. This happens when the user terminates incorrectly. So make sure you're looking for spinning and defunct processes. Here's an outline for killing 'spinning' processes:

ps -ef | while read USER PID PPID CPU STIME TTY TIME
do

if [[ $TTY = '?' ]]
then
if [[ $PPID = 1 ]]
then
/usr/sbin/kill -9 $PID
fi
fi

done

##########################################

Here's an outline for killing defunct processes:

ps -ef | grep -i defunct | while read USER PID PPID CPU STIME TTY TIME
do
/usr/sbin/kill -9 $PID
done

##########################################

Also use the idle timeout (* 'TMOUT=3600' *) environment variable for each login. Include this in either /etc/profile, or $HOME/.profile.

export TMOUT=3600

###########################################

Finally there's always logging.

inetd -l

But this will fill up your syslog.log really fast. But it can be invoked from the command line and toggled off when desired.

[Ralph Grothe]

Steven,

thanks for your kind scripting help.
But there is no longer any need to rely on tools such as sar, vmstat etc. because we have the mwa suite of agents running on the db servers, and we have a regular eye on what I believe to be the crucial system metrics.
Actually I found the by HP predefined alarm definitions in /var/opt/perf/alarmdefs quite suggestive, since whenever these threshholds were passed and a bottleneck discovered in the past the servers really were buckling down.
So I stick with the metrics from those definitions and added a few application or disk specific graph definitions.

Currently the servers have reached their final stage of upgradability since everything what will fit in those N-class boxes as far as RAM, CPUs etc. is concerned is stuffed in.

Unfortunately I cannot follow your link, and get a "could not login" response (probably because I have't come over the European proxy?)

[Ralph Grothe]

Michael,

you aren't taking the mickey?

1st how can you kill a proc that is already defunct?

It will stay in the process table (though consuming no resources) until you reboot.

2nd if I used your kill script I would shoot all process that aren't associated to a tty (i.e. daemons) and whose parent is init (i.e. who became session leaders), which ends aup to killing almost every vital daemon.

[Dagmar Boelen]

Hi,

Try to adjust the nice-value of inetd. It might boost performance if it has a higher priority then other processes.

[Dagmar Boelen]

Hi,

Dont forget to disable all the logging of the inetd.

[Chris Wilshaw]

Ralph,

If telnet performance seems to be suffering, you can try the following;

set the telnetd line in /etc/inetd.conf to

telnet stream tcp nowait root /usr/lbin/telnetd telnetd -s
400 -z5

This sets the timeout and buffer size values of telnetd (under HP-UX 11.*) to more reasonable values. This works for IP and serial connections.

Chris

[Ralph Grothe]

Michael,

forgot to mention, the inetd is already running with the -l switch on special demand of the DBAs, which annoys me because they fill up the syslog.log with their connection entries.
I in vain asked them several times if I couldn't make up an extra logfile for such.

# grep INETD_ARGS /etc/rc.config.d/netdaemons
# INETD_ARGS: The command line arguments to be used when
export INETD_ARGS=-l

[Ralph Grothe]

Dagmar,

thanks for the hint about renicing inetd.
I will consider this.

But of course the demanded logging (see my last response to Michael) is really defeating this.
This is kind of idiotic.

Maybe one could influence the buffering of inetd messages to syslogd?

[Dagmar Boelen]

Hi,

try writing to a striped logical volume. Really boosts you disk I/O and in the case the performance of inetd.

[Ralph Grothe]

Dagmar,

the cluster shared LVs (whereon the TSpaces reside) are already striped.
Apart, disk i/O isn't any more such an issue
(n.b. caches of the involved disk subsystems also subsequentially have been increased)

[Sridhar Bhaskarla]

Hi Ralph,

Every service that is registered in /etc/services does not mean it is used by inetd. What you see in /etc/inetd.conf is what it matters.

I do not believe that inetd is the culprit here. Once a connection is established with the port managed by inetd, it will fork a seperate server process for the client and it will be out of the picture for that process. To prove it, telnet to the box and keep the session open. Logon to the box through console and kill inetd. New connections will be refused but still the old connection would be in tact.

I would say to pay more attention to the pattern. Is it happening to everyone or only certain users in one subnet?. When you try to telnet to the box at that time, are you experiencing that issue?. Check for any ARPA patches that are missing on the box.

-Sri

[rick jones · Accepted solution]

a few random things:

iirc there is code in inetd to limit the rate at which it does new services - to dela with runaway client requests and such. iirc, when that is triggered, there will be a syslog or dmesg entry somewhere.

inetd calls listen() with a backlog of 128 - however the default tcp_conn_request_max "connections awaiting call to accept()" setting is 20. You could check netstat -p tcp and see if there are connections dropped due to full queue (second to last line of the output) and if there are, use ndd to increase tcp_conn_request_max

ftp://ftp.cup.hp.com/dist/networking/briefs/annotated_ndd.txt

[Michael Steele_2]

Hah. Copy, paste and haste.

Doh!

Check the tty = ? though + PPID = 1 is definition of spinning process. 'ksh' if not telnet.

Sorry for the misinfo.

[Ralph Grothe]

Sridar,

I know that the /etc/services isn't the decisive file for inetd, but rather the the inetd.conf (I thought to have stressed it already).
It was just that the parsing of /etc/services was quicker for me
(i.e. I didn't have to have another look at the file in advance since I knew the db-special services started from line 190)

But with your 2nd statement you are absolutely right.
I've already come myself to the conclusion that my suspicion of inetd was bullshit.
As you said, inetd only forks off the new service.
Afterwards the service is up to its own responsibility.

You're right, I should pay more attention to LAN branch from where interrupts were reported.

[Ralph Grothe]

Sridar,

I know that the /etc/services isn't the decisive file for inetd, but rather the the inetd.conf (I thought to have stressed it already).
It was just that the parsing of /etc/services was quicker for me
(i.e. I didn't have to have another look at the file in advance since I knew the db-special services started from line 190)

But with your 2nd statement you are absolutely right.
I've already come myself to the conclusion that my suspicion of inetd was bullshit.
As you said, inetd only forks off the new service.
Afterwards the service is up to its own responsibility.

You're right, I should pay more attention to the LAN branch from where interrupts were reported.

[Ralph Grothe]

Oops, sorry for the last double posting

[Ralph Grothe]

Rick,

10p for your annotations and the URL you submitted.

This is most welcome documentation of ndd tunables on HP-UX.

I still have another problem lingering where a Solaris box is affected, and which I spread in this thread:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x2773227a6ab4d711900a0090279cd0f9,00.html

As a reaction on this posting Stefan Farelly posted the URL of an excellent description of Solaris ndd tunables that I needed so much.

Also thanks for your insight into inetd.
(if I only had the time and guts I could have looked up the Source Code of inetd implementations of the free Unices like the *BSDs or Linux or GNU programs to find out how things really work behind the scenes, and learn more about Unix)
But as I replied to Sridar's posting I don't believe anymore that inetd is the culprit.

[Ralph Grothe]

Sridhar,

forgive me for continously mispelling your name.