Telnet not working??

[Tim Medford]

We have a weird problem with one of our L1000 servers. Running HPUX 11i dec2002 patch bundle.

Every couple months we lose the ability to telnet into the server. We can ping it fine, the Oracle listener still works and databases are up. We just can't telnet into it.

I can get into the GSP through the serial port, but if I issue the "co" command to get to a console prompt, it just hangs.

The only way I've been able to fix this is a reset through the GSP. Not exactly a graceful recovery to say the least.

Any ideas what is going on here?

Thanks,
Tim

[Sridhar Bhaskarla]

Hi Tim,

What error do you get when you try telnetting to the server?. Does it timeout or hangs before prompting you with login?.

Did you look at OLDsyslog.log under /var/adm/syslog after you rebooted the box to see if there are any errors?.

HP can easily find the problem if you dump the image of the system by TOC'ing the box.

-Sri

[Steven E. Protter]

Turn on enhanced logging on inetd

manually inetd -l

From the console when this happens:

inetd -l

tail -f /var/adm/syslog/syslog.log

Try telnet.

If you see the log update you are connecting and should act on the error message.

If you don't see anything then you are being blocked before you get to the box.

Check into IP conflicts, firewalls and other issues that can cause this on an inermittant issue.

SEP

[Jeff Schussele]

Hi Tim,

My bet would be on running out of streams telnet connections.

What are your kernel parameters

npty
nstrpty
nstrtel

set to?

If they aren't large enough to handle all the streams connections (and these can be non-telnet streams connections as well) then you won't be able to connect via telnet.

So next time it happens check the connection count & compare it to those parms. Or better yet if those parms are low, go ahead & bump them up now - there's no penalty memory-wise in doing so. We set all ours from 256 - 1024 depending on anticipated loads.

Rgds,
Jeff

[Tim Medford]

Thanks for the replies.

Telnet simply says "Connecting to voyager..." and then never comes back.

Steven - I can't get to a console prompt to start anything manually. The only thing I can get to is a GSP prompt.

I don't think I'm hitting any resource limits. This is a sandbox server that never has more than 2 people connected to it.

It's in it's hung state right now if there's anything you think I might be able to try, short of an RS from the GSP.

Thanks

[Sridhar Bhaskarla]

Tim,

Try "tc" instead of "rs" so that it will dump it's image. After it came up, look at /var/adm/syslog/OLDsyslog.log and see if there is any interesting activity. Once the crashdump is saved, you can run "Q4" analysis with "whatHappened" and generate whatHappened.out file.

You can find Q4 documentation from the site

ftp://contrib:9unsupp8@192.170.19.51/crash/using_q4

Do you have patches uptodate on the server?.

-Sri

[Jeff Schussele]

Hi (again) Tim,

Are there any active connections now?
Are they OK & responding normally?
Can you get to one of them & su to root?
Then try to remove another user by killing their shell & see if you can now open a new session. If so - you're hitting your connection limit.

The *actual* user count could be low, but if you had a lot of Oracle connections they'll each count as a streams connection & you could be hitting one of those limits. I think they're 60 by default & that's easily reached on an Oracle box.

Rgds,
Jeff

[Tim Medford]

Thanks again guys.

I just issued a tc command, it is booting now. I'll check the logs and the kernel parameter and let you know what I find.

Tim

[Tim Medford]

I reset the system and it appears the problem was related to the root disk. I don't know if the reset hosed the disk or if it was bad to begin with. I've rebooted a couple more times now and it will not use vg00, it's coming up on the alternate boot disk.

I have a good igniteUX tape luckily. I'll log a hardware call tomorrow.

Thanks everyone.



Trying Primary Boot Path
------------------------
Booting...
Failed to initialize.

ENTRY_INIT
Status = -7

00000000 FFFFFFF9 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000012 00000000 00000000 00000000 00000000
00000000 454E4841 00000000 00000000 00000000 00000000 00000000 454E4841
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000200 00000000 00000000 00000000 00000000 00000000 00000000

************* SYSTEM ALERT **************
SYSTEM NAME: voyager
DATE: 01/13/2004 TIME: 22:03:41
ALERT LEVEL: 6 = Boot possible, pending failure - action required

REASON FOR ALERT
SOURCE: 8 = I/O
SOURCE DETAIL: 6 = disk SOURCE ID: 0
PROBLEM DETAIL: 0 = no problem detail

LEDs: RUN ATTENTION FAULT REMOTE POWER
FLASH FLASH OFF ON ON

0x0000006086002370 00000000 00000000 - type 0 = Data Field Unused
0x5800086086002370 00006800 0D160329 - type 11 = Timestamp 01/13/2004 22:03:41
A: ack read of this entry - X: Disable all future alert messages
Anything else skip redisplay the log entry
->Choice: