Connections keep dying...

[David M Campbell]

I work at a University where our department has an aging HP box running HP-UX 11.11 that I am in charge of managing, but had no part in setting up.

Recently, we've started having a problem where after about 5-10 minutes, the server closes all connections from the outside world (internet) and any requests just time out with at network timeout error in the browser.

However, the IT guy who is helping me with this issue is still able to access the server from the switch in the server room that sits in front of this server.

When the problem happens, he can go in and simply perform any type of network function (pinging, for example) from the server, and the connection seems to spring to life again.

He did some research and found mention of a dead gateway problem on some of these machines, but his changes don't seem to have fixed it so far. It is still dying after a pretty short amount of time.

There hasn't been anything particularly helpful in the Apache error log...

a "caught SIGTERM, shutting down" here or there...

a couple of segmentation faults, which led me to believe that somehow PHP and MySQL were not getting along, but today, even after not starting MySQL for a while, the problem persists.

He is getting ready to set up a cron job that will ping a server every five minutes and see if that keeps the connection from dying, but we are at our wits end with this...

Can ANYONE help out??

[Tim Nelson]

Can you ping your default gateway ?

If you CANOT ping your default gateway then it probably is the dead_gw_probe issue.

If you CAN ping your gateway then the problem lies somewhere else ( but typically a routing or NIC issue, two nics same subnet ?).


This issue typically occurs with servers on DMZs where it is best practice not to have the firewall(which is also the default gateway) to not answer ICMP requests. By default HPUX will disable the default gateway if it does not get an answer from the gateway_probe request.

This feature can be turned off with the ndd command.

ndd -get /dev/ip ip_ire_gw_probe

use -set to change it, then modify /etc/rc.config.d/nddconf to make permanent.

[Steven E. Protter]

Shalom,

/var/adm/syslog/syslog.log

There should be adequate evidence in there that will lead to a cause, be it external or internal.

SEP

[Ivan Krastev]

Examine nettl logs as well - /var/adm/syslog/nettl.000*

regards,
ivan

[Michael Steele_2]

Hi:

I've seen this problem twice of several years. Both times an excessive number of messages were being written into the /var/adm/syslog/syslog.log file.

In one instance, the debug flag had been enable in /etc/syslog.conf.

What is the size of your /var/adm/syslog/syslog.d file? It clears to zero with reboot.

What is written at the bottom of /var/adm/syslog/OLDsyslog.log? What is its size?

[David M Campbell]

I've checked out the syslog.log in var/adm/syslog, as well as the OLDsyslog.log

I don't see much information in the syslog that seems helpful, but in the OLDsyslog there were a couple of interesting entries...here are the last 5 entries:

krsd[2264]: Delay time is 300 seconds

/usr/sbin/envd[1826]: terminated by signal 15

inetd[1068]: Going down on signal 15

rpcbind: terminate: rpcbind terminating on signal. Restart with "rpcbind -w"

syslogd: going down on signal 15


So far, the cron job he set up last night is keeping the connections alive. I am starting to wonder if this is HARDWARE related, with the NIC. I have not tried pinging the default gateway yet. Will do that next...

[David M Campbell]

Tim Nelson...

Which area of the nddconf file needs to be modified to keep the feature turned off?

I see this entry in the nddconf file...

TRANSPORT_NAME[1]=ip
NDD_NAME[1]=ip_ire_gw_probe
NDD_VALUE[1]=0

Is this the one that needs to be modified, and if so, what should it be changed to?

[Michael Steele_2]

Can you check the GSP error logs?

Can you check /etc/shutdownlog? Any strings?

Can you check the /var/tombstones/ts99 log?

Can you check /var/crash for crashdumps?

Have you made any hardware replacements recently?

Please copy and paste in /etc/fstab.


Please provide all of the above for quickest diagnosis.