Telnet connect but no login prompt

[BEN WRIGHT]

Hello there,
This problem is a strange one, I know it relates to the inetd, but how can I stop it happening again?
This morning our web site went down, we went to log onto the machine and found that we could not telnet or ftp on to either of the network cards.
Let me make this clearer, we could telnet, as in we got to:
Trying xxx.xxx.xxx.xxx...
Connected to xxx.xxx.xxx.xxx
Escape character is '^]'.

But that was it, when I used the status command in telnet it told me that:
Connected to xxx.xxx.xxx.xxx
Operating in single character mode
Catching signals locally
Remote character echo
Local flow control
Escape character is '^]'.

But we never got our /etc/issues banner, nor our login prompt.
This was the same on the Web Console access we had, once I typed in the login name and pressed enter it seemed to hang.
We switched the machine off then on again and it came up with no error messages, but again nobody could telnet or ftp onto the machine.
The web console was working now so I issued a "kill -1" on the inetd daemon, and suddenly it all worked again.
Please if anyone has any ideas as to why this happened or how to stop it happening again I would be much appriciated.
We had 1 error the whole time in the syslog:
May 16 01:07:56 uatlax3 telnetd[20457]: Error checking child termination status:
error 4: Interrupted system call

Looking at the IRTC there is very little on this. Your help is appriciated.
With regards
Ben

[S.K. Chan]

Well not much to go with except for the error message "..child termination status". I can think of a couple ..

1) The npty/nstrpty parameter too low hence ran out of pseudo-terminal.
2) /etc/inittab corrupted (since console is respawned from inittab).

I would try to check on 1) first, bump up the kernel values. 2) is not likely since you're able to get your login prompt at console. If you have to deal with 2) (replace /etc/inittab) make sure you run ..
# init q
afterwards.

One final note , make sure you got the latest and greatest "cumulative" patches installed. For instance ..
- telnet
- ARPA transport
- rlogin
- LAN products

[Steve Steel]

Hi

The pty is a good direction.

The system cannot allocate or recycle processes fast enough for the
application. Check the following kernal parameters have at least the
following values:

npty 300
nstrpty 300
nflocks 8192
nfile 8192
nproc 512

If you increase npty or nstrpty then use insf to create the device files.

Steve Steel

[BEN WRIGHT]

Hello there thanks for the response, I would also ask if there is any way that we can test this theory that the machine cannot recyle these processes fast enough?
I will push for the Kernel Parameters to be increased, but we have had an Oracle IAS Expert in the office viewing the Params and he thinks they are sufficient.
I would add that this is a working machine, there are very few actual telnet or ftp connections going on the machine.
With regards
Ben

[Wilfred Chau_1]

Other than the pty devices, I would check
name resolution as well. Make sure the other host can reverse lookup your IP.

[Sandip Ghosh]

I think it is better to enable the inetd logging by giving "inetd -l". It will log all the logging message in the syslog.log file. You can enable the inetd debugging also by giving "inetd -b". But be careful, in this option you should have enough space in the /var directory.

Sandip

[U.SivaKumar_2]

Hi,
Try this.
Without any telnet login to system through console access (getty) see how many telnetd instances are running.
#ps -ef | grep telnetd
if you happen to see any telnetd kill that process and try telnet again.

if still it does not work , you can try this
Remove the entry from /etc/inetd.conf and run
telnetd through command.


regards,
U.SivaKumar

[T G Manikandan]

check this link out.

http://forums.itrc.hp.com/cm/QuestionAnswer/0,,0x30adba808b46d611abda0090277a778c,00.html

check for the telnet kernel parameters.
I suggest to rebuild the kernel and reboot the machine.

check for these patches for telnet and ARPA patches
PHNE_26096
PHNE_25116
PHNE_26771
PHNE_26250 PHCO_21596
PHNE_14820
PHNE_25385

[Tom Gore]

I had the exact thing happen to me this morning. Our network guys brought down the web server and users that were logged on experienced slow response time. Anybody trying to connect would wait a really long time. The minute the web server was brought bacj online, everything returned to normal. This leaves me a little confused with the responses. How would any of the solutions posted fix this problem?

Thanks,
Tom

[Tom Gore]

Oh, I forgot to mention that we don't even go through the web server. We use Reflections or Rumba to telnet to the HP.

Tom

[Robert Gamble]

The suggestion Wilfred Chau is correct about reverse DNS lookup.

I had the exact same symptoms as you described, and found out that a DNS change on the PC network didn't allow the HP host to get a reverse lookup. One the change was backed out, the HP host repsonded fine again to non-UNIX systems. (it always responded to UNIX systems, not through reflections or hummingbird).

Good Luck!

[sven verhaegen]

HI

telnetd: Error checking child termination status: error 4: Interrupted system
call

This problem is most common on new system installs that have not been performance tuned or patched. It is most commonly seen when running a new application on the system.

RESOLUTION
The system is unable to allocate or recycle processes fast enough to meet the application's demands. Check the following kernal parameters have at least the following values:
npty 300
nstrpty 300
nflocks 8192
nfile 8192
nproc 512
If you increase npty or nstrpty then use insf to create the device files. Always check if the application vendor has specific requirements for kernel tuning. If the kernal parameters look good, check the following patches are at a good level (these are 10.20 patch titles, check for equivalent at 11.0):
STREAMS PTY cumulative
ARPA Transport cumulative
LAN Products cumulative
telnet cumulative
telnet kernel and throughput
rlogin cumulative
libc
inetd

(creation of device files)

insf -d ptys -n 300
512 being the number you set NPTY in SAM (10.20 and 11.0)

insf -d ptym -n 300 (10.20 and 11.0)

insf -d pts -s 300 -e -v (for 11.0 only )


if you have added several users lately this could also be an issue with the maxusers being to low , you could change this upwards also

[Sebastian Galeski_1]

Hi
Check entry in /etc/inetd.conf, should be:
telnet stream tcp nowait root /usr/lbin/telnetd telnetd

do inetd -c
check entry in /var/adm/inetd.sec

[BEN WRIGHT]

Thank you all so much for your responses. We finally tracked the problem down do the Oracle Portal where it's priorities setting were too high and it was grabbing to much of the CPU (we only have one, thus it was making the machine hang). I feel I need to give you an over view of what went wrong so that if it happens to somebody else:

Thu, 16 May - uatlax3 - Hanging PRoblem appear. cannot log in . Reset MAchine

Fri, 17 May - uatlax3 - Problem appear again -Comment wwwin.go2uti.com out of DNS (external and internal) - done by LAX

Mon, 20 May - git5 - Make following config changes:
kernel : maxfiles 2048 -> 4096
maxfiles_lim 2048 -> 4096

tcp config changes:
tcp_conn_request_max 20 -> 1024
tcp_ip_abort_interval 600,000 -> 60,000
tcp_keepalive_interval 7,200,000 -> 900,000

JServer config changes

Tue, 21 May - git5 - Manage to repeat problem. Get crash dump

Wed, 22 May - git5 - Deliver crash dump to HP

- Get Oracle involved on hi-level - confirmed that they are not waiting for us.

- Load HP latest patchset - Mar2002 (was previously June 2001) - onto git5
Need to do another test after this. (Crashed again)

- Did the load test after installing JDK 1.3.1.02 - SAME SYMPTOMS!
- Run HPjconfig tool - it recommended the following Kernel Changes:
max_thread_prod 256 -> 3000
maxdsiz 1048576000 -> 2063835136
maxfiles 4098 -> 2048 (previously recommended by ORacle to up to 4096)
maxfiles_lim 4098 -> 2048 (same as above)
maxusers 400 -> 512
ncallout 3220 -> 6000
nfile 6209 -> 30,000
nkthread 2048 -> 6000
nproc 3220 -> 2068
It also recommends the following tcp configuration change:
tcp_conn_request_max 1024 -> 2048 (previously upped from 20 -> 1024)

It also showed some missing patches (will investigate later - for now, only made kernel and tcp changes)

HANG AGAIN!

Wed, 22 May - 23h15 - Response from ORacle after sending log and config files to them.
Edit the jserv.properties file and find the following entry:

port=8007
Replace 8007 with 8007-8020 like below:
port=8007-8020
Wed, 23 May - 02h15 - MAde the following change in the jserv.properties file for the java binary from JDK 1.3.1.02:
"wrapper.bin.parameters= -client" (for quick gui apps using Hotspot)
(was previously changed from the default (-server) to "wrapper.bin.parameters= -classic")
Test, but could not get the server to hang - needs more people to do a proper test.

So it seems that it was a simple entry in the J-Server properties that was causing the problems with the network, how many times have we heard that before!!!!
Thanks for all your input, and Tom I hope you fix your problem.
With regards
Ben