telnet errors

[Steve Lewis]

We have an odd issue here. As of Saturday morning people at one specific site can no longer telnet into one of our machines. They get no login prompt.

However they can ftp into the machine.
They can also telnet into other machines on the same lan as the problem machine.
Users at 2 other sites (including me), however, can still telnet into the box with no issues.

The issue is complicated by the fact that the troubled users have their PC's IP addresses NAT'd, en route through 10 routers and 2 firewalls, from an external organisation into ours.
Taking the firewalls down (temporarily) didn't fix it.

Previously the system was up for 6 months with no problems. There have been no changes on the server before this event. The network people are also saying that there were no changes made to their equipment either.

The routing table looks OK, since the clients IPs are NAT'd they all go through the default route, just like the rest of us.

the machine is x.x.141.14/24
the default route is via x.x.141.1
FTP -l in syslog shows their IP is x.x.230.22

I have searched the forums and found a couple of documents, one pointed to telnetd: getpid: peer died, error 0. These messages have started appearing in my syslog this morning.

Nettl tracing has been started but we cannot figure it out.

The network staff tell me that they have traced packets all the way into the box, but not out again, so they are blaming us, when we haven't changed anything.

Any ideas?

[Steven E. Protter]

diagnostic steps:

1) Check ping connectivity as root user.
2) Run traceroute commands to see if a router is holding it up.
3) check and see if there is a /var/adm/inet.sec file. If so check contents. telnet deny????? Try renaming it and re-testing
4) Check the /etc/inetd.conf file and make sure the telnet daemon is running
5) inetd -l will do logging. Try telnet then check /var/adm/syslog/syslog.log

Take appropriate action based on what you find.

SEP

[Steve Lewis]

Ping and traceroute are blocked by the firewalls. I cannot use these.

inetd -l will be tried, since we have nothing to lose. The man page appears to say that I can just type it in directly and it will recycle the daemon, is that so?

[Chris Vail]

This could easily be the inetd.sec file. Check to see if it exists. If it does, rename it temporarily, recycle init.d and have them try again.

BTW: You should prolly encourage the use of secure shell instead of telnet.


Chris

[Ron Kinner]

Your message: getpid: peer died, error 0

just means the other end disconnected improperly. This can be caused simply by the user hitting the X to close telnet instead of typing exit like a good user. Network troubles such as a flapping route can cause this too. There is even one report of another host squatting on the gateway's IP address.

When your locked out users ftp to the site are you sure they are reaching the correct box?

Ron

[Volker Borowski]

Hi,

if you ftp is proxied, it could be several things, but if ftp is not proxied, I'd first assume a proper TCP/IP configuration, because otherwise you would not get a connection at all.

If you can afford to disable FTP for a while, i'd suspend it, start a seperate telnetd which is listening on the FTP port and try to get a telnet connection on the ftp port. If this works, I'd suspect the firewall rules for telnet are defective. May be filtering does not take place only on the firewalls, but on one of the routers in between.

Good hunting
Volker

[Steven E. Protter]

Have the firewall administrator do the ping and tracerotue from the firewall to the source and from the firewall to the destination. It is critical that you be sure your packets are passing back and forth.

SEP

[Steve Lewis]

Thanks for all your suggestions.

inetd.sec isn't used.

I try to encourage ssh, but everybody else in the organisation is against me, for reasons of short-term management and practicality.

The users did not disconnect at all since they never got as far as the login screen. Their telnet may have dropped an initial connection, but not having the source to telnet I cannot tell. I assume its just a simple matter of socket/bind listen/connect send/recv etc?

Your suggestions of flapping routes etc are interesting. I am sure that nobody is squatting on the gateway IP since I also use it and I have no issues at all.

The people on the firewall claim they can ping both ways. They are firmly denying all responsibility. I have learned too often in this game this it doesn't pay to accuse people too quickly (never say it defininity isn't you) but options are limited.

I like the idea of putting telnetd on the ftp port. I would try sshd as well, only they dont have it.

Our nettl trace shows packets are being sent back out of our rp8400, but they never get back to the destination. Traceroutes get as far as our switch, then stop (* * *s) which means they don't get back past one or other firewall.

We have a short-term workaround, which involves the users connecting to a different machine, which then automatically connects to the main one.

[Massimo Bianchi]

hi,
you said:

"Traceroutes get as far as our switch, then stop (* * *s) which means they don't get back past one or other firewall. "

Do you manage to ask (politely and at low priority) to the network people if the natting and routing table has changed, only for that firewall ?

Another thing: sometimes a reboot of the firewall/reouter may help, a system may have a half corrupted routing table.

Can you put a sniffer on both sides of the net, in your hpux boxes and in your pc ?


What is the difference between that hosts, those that are telnet-able and those not ? I'mt thinking at a route change.. For example, they changed the subnet mask. maybe one host is in the /24 (ex: 63) and now the mask is /26 (and one on the ip is 172). Double Check all the subnet mask in your hpux.

HTH,
Massimo

[Steven E. Protter]

I am pushing the envelope

There is a free ssh client called PuTTY.

It is 300K runs like a rabbit and can give your users Secure Shell Clients.

More important is to move away from the muddle of my earlier posts.

If you have log entries then you don't have firewall problems.

10 lashes with a two by four for me.

The connections are dying, before the login prompt.

What can cause that is the accidental deletion of some of the devices that telnet and term sessions need to live. With the help of HP, I did that once.

Possible solution, though a royal pain in the tushie.

1. Start at http://www.itrc.hp.com
2. In the "maintenance and support" section, click on the "search technical
knowledge base" link
3. Login with your username and password
4. In the "technical knowledge base" screen, you can "Search by Keyword" w
ith the "Search
Criteria" marked as "All Words". You will want to search on the
phrase: KBRC00004879

Follow that procedure.

It will however nuke your telnet devices which is probably what already happened. The fix is ....


I'm guessing you or someone ran a procedure like this.

(Steve: I edited URL out to avoid page expansion...Dan)

The solution is on a notepad at my office. I'll update you with it in the morning. HP never updated the document as I asked, and the fix isn't even in my old support call.

SEP

[Steve Lewis]

Sorry, nobody has done anything with device files, but wouldn't that effect everybody who tried to log in?

I like that PuTTY thing and will give it a go. Maybe the cost will persuade people to give up on Rumba. However ssh will not solve my problem immediately.

Currenly the network people are doing more tests. I assume they have sniffers etc, but staff rationalisation has now put them in a different part of the country to me and we dont have the level of communication I used to have.

Massimo said that a reboot may be the cause and it may need another one. That is possible, since our firewall was rebooted on Friday night. I am guessing that somebody made a change to it some days previously which never got put into place until it was rebooted.
They are still claiming that no change was made.

The ftp test definitely connected to the right machine, since I see the evidence in the syslog.

I will check the subnet masks on the rp8400, also ask people to check the router subnets (I know its a very complicated set-up).

[Brian Hackley]

Steve,

I will venture two guesses.
The first and least likely is that the login prompt is delayed while the rp8400 tries to resolve the IP of the client to a hostname. I frequently tell folks to shorten the resolver timeouts in /etc/resolv.conf:
domain mydomaim.com
nameserver 10.1.1.1
retrans 1000
retry 2
more into in 11.11 resolver() man page
also a big tip:
Use nsquery hosts 192.1.1.1 to check the reverse lookup and nsquery hosts hostname to check the forward lookup. These should be used to validate gethostbyname() and gethostbyadddr() operations with DNS.

The second and more likely cause is that one of more of the /etc/utmp* files e.g. utmp or utmpx has been corrupted. See the "Special Installation Instructions" in patch PHNE_24829 (file PHNE_24829.text go to the end of the file to find these). This contains the explaination of this problem and the exact procedure as to how to clear corrupted utmp or utmpx files at boot time.

Hope that helps,

-> Brian Hackley

[Steve Lewis]

Thanks for that Brian, DNS hadn't come into it though, since people tried telnetting to the IP address directly.

What we have been able to establish now, is that by reducing the length of /etc/issue file so that the network packet form the server is under 300 bytes, the login prompt appears.

The user is then able to log in and after a couple of short messages displayed to their screen, the application starts...which causes a screen re-draw and a BIG PACKET to be sent - which the user never sees.

It seems as if large packets (or maybe fragmented packets) no longer get through one of the network devices (switch/router/firewall).

I will keep you all informed...

[Massimo Bianchi]

Hi,
what is the size of the /etc/issue ?

I'm thinking of some router that do not forward packets of more than 1200 bytes.

Another thing: is there any special control char in the file ?

On some customer site i saw very nice issue, with flashing and bumping words, but that created some problem at login, as in your case.

Massimo

[Steve Lewis]

Sorry brian I didnt fully read what you said.

I tried the nsquery hosts their-ip and it came back straight away with a not-found.

[Steve Lewis]

Massimo, the old issue file (dated March 24th and working fine for those users for 3 months) was 445 bytes long.
The new one is just 24 bytes long and it works.

[Brian Hackley]

Hi again,

With your latest update, it sounds possible somewhere there is a duplex mismatch, bad-marginal {cable | NIC | switch port }
or some kind of mystery in the 10 routers and 2 firewalls.

Brian

[Ron Kinner]

Steve,

ndd has an MTU discovery setting which has three options. (do ndd -h ip_pmtu_strategy to read about the three options.)
ndd -get /dev/ip ip_pmtu_strategy
to see what it is set to. You might try one of the other options to see if that helps.
ndd -set /dev/ip ip_pmtu_strategy x
where x is the option you want to change it. Also edit /etc/rc.config.d/nddconf or it will go away after a reboot:

TRANSPORT_NAME[0]=ip
NDD_NAME[0]=ip_pmtu_strategy
NDD_VALUE[0]=x

Use the next higher integer in the bracket if you already have entries.


You can also send a ping with a gradually increasing packet size
ping host 100
ping host 200

until you find the size that cause the problem. Then you can put in a route to the problem users which uses a smaller pmtu.

Ask your network guys about the MTU across the network. They should be able to correct the problem. Could also be that the firewall has something against the DNF (Do not fragment) bit being set.

Ron

[Steve Lewis]

Just to let you all know that the problem was caused by our old friends British Telecom, who added encryption between two routers last Friday night, without telling anybody they had done so.

Once they removed it, users was able to log in again without issue. I wonder if they had also set the DNF bit for security...

Thank you all again for your help. This thread is going to be a useful source of things to check should anything like that happen again.