connection refused on port 1022

[Helen French]

We have an admin that uses cosmos, an admin tool that uses rexec to perform remote commands and spans across different OS's. I am having trouble getting it to connect on one server (K460/B.11.00 E). It connects via port 1022 which is setup correctly in /etc/services. I am leaning to a port problem on that server because telnet localhost 1022 results in:
Trying...
telnet: Unable to connect to remote host: Connection refused
Where on a working server it opens a connection and disconnects appropriately.
I have a similarly built server in production that is working fine with this utility.
Any suggestions on what else to check would be appreciated.

[MARTINACHE]

If you have lsof (if not, install it) :

lsof -i:1022

Then you if something is lestening to port 1022.

Regards,

Patrice.

[Jeff Schussele]

Hi Christopher,

Don't know if this is your problem, but all ports < 1024 are reserved ports & as such can only be used by root level users. So if the application that listens is not owned by root you'll have problems.

Rgds,
Jeff

[MANOJ SRIVASTAVA]

It looks that the port is still occupied wiht some process which has not been killed , what you can do is either kill the proces ie , use lsof | grep 1022 and then find out with proceess is using it and kill it or , if the same is not easy then may be define another port in the /etc/services , but you need to cahnge the same in application too.

lsof can be had form :

ftp://vic.cc.purdue.edu/pub/tools/unix/lsof/



Manoj Srivastava

[Helen French]

In response to these posts so far I do not have lsof loaded. I could and will have to get it from the porting center I suspect, which may help as a longer term option (probably should have it anyway).
This is loaded as root, loaded identically the same as 100's of other servers.
There is nothing in /etc/services defining this port and a ps -ef doesn't find anything on that port, so I am not sure that there is something already using that port either?
Again, lsof -- good idea also.

[Helen French]

Correction -- there is nothing in /etc/services other than the entry I put in there to define that port this to tool.

[Ron Kinner]

netstat -an |grep 1022

should show you if anything is listening at all.

netstat -a |grep rexec

or whatever you called it in /etc/services will show it too.


Ron

[Helen French]

Both working/non-working server show:
# netstat -a | grep 1022
udp 0 0 *.1022 *.*

and

# grep -i cos /etc/services
COS_shell 1022/tcp

[Jeff Schussele]

Hi (again) Christopher,

It's interesting to note that your netstat -a output shows the system listening on 1022/UDP, whereas your services file defines it to be 1022/TCP.
Which is correct - TCP or UDP?

Also how is this port "enabled". Is it started as a daemon or is it listed in /etc/inetd.conf & started on demand by the inetd daemon on a connection request?

Rgds,
Jeff

[Helen French]

It is being started through /etc/inetd.conf with the following:
COS_shell stream tcp nowait root /usr/cosmos/COS_3.2/local/bin/COSrshd COSrshd -
l

It should be tcp, I can't answer why netstat comes up and says udp? Maybe that is part of the problem? But my other box in which it is working says that also (not necessarily correct but...) The only difference I know between the two boxes is that the broken box is using NIS (although should not be impactive).

Thanks

[Jeff Schussele]

Well, since this uses a streams socket type, one thing I would insure is that this system is at the same patch level streams-wise as the other systems.

Rgds,
Jeff

[Patrick Wallek]

Since your broken box is using NIS, you might check your /etc/nsswitch.conf and see what the config line is for services.

I'd try doing:

services: files nis

[Helen French]

I thought we might we going somewhere with the /etc/nsswitch.conf because I didn't have a services line. But i put that in to no avail. Still: Connection refused.
They both are patched the same for streams
# swlist | grep streams
PHNE_17051 1.0 Cumulative streams patch
Have had some good things to check here but no change. Any other ideas?

[Ron Kinner]

If both working and nonworking servers both say they are listening on UDP then I would expect telnet to 1022 to fail on both. Telnet want to establish a TCP/IP connection and that is not available so you should always get the Connection Refused message. Your stream nowait stuff is all tcp so why is it listening on udp?

Something does not make sense. Go back and retry your telnets (use the ip address and not the hostnames and try it from a different box - not one of the ones which is involved.) and verify that one works and the other fails. When the one connects try a
netstat -an |grep 1022
and see if you see ESTABLISHED. (from a separate telnet session or the console).

Then verify that you are really logged on to the correct box when you do your netstat over again. Finally when you do
netstat -a|grep COS_shell
do you see it in both cases?

Ron

[Jeff Schussele]

What happens if you manually start the cosmos daemon from the command line - if that's possible?

Can you then connect?

If so it has to related to the way inetd.conf is starting it.

Rgds,
Jeff

[Helen French]

Ron,

I think you may have found something here.
I do still get the same symptoms using ip addresses from an uninvolved server.
However, netstat -an | grep 1022 reveals

working server:
# netstat -an | grep 1022
tcp 0 0 164.121.12.34.1022 204.53.90.100.61970 TIME_WAIT
tcp 0 0 *.1022 LISTEN
udp 0 0 *.1022 *.*

server not working:
# netstat -an | grep 1022
udp 0 0 *.1022 *.*

Hopefully one final question. Maybe I know how but can't think of it - how do I add that tcp line to listen like it should be?

[Ron Kinner]

I think I would rename the bad box's inetd.conf file to inetd.conf.old and /etc/services to /etc/services/old and then copy both to the bad box from the working box and then run
inetd -c
to get it to reread the conf file.

Also check to see if you have an inetd.sec file. If you do you need to have the service listed in there.

A bit of a stretch but check /etc/protocols to make sure it says:

ip 0 IP # internet protocol, pseudo protocol number
icmp 1 ICMP # internet control message protocol
igmp 2 IGMP # internet group management protocol
ggp 3 GGP # gateway-gateway protocol
tcp 6 TCP # transmission control protocol
egp 8 EGP # exterior gateway protocol
pup 12 PUP # PARC universal packet protocol
udp 17 UDP # user datagram protocol
hmp 20 HMP # host monitoring protocol
xns-idp 22 XNS-IDP # Xerox NS IDP
rdp 27 RDP # "reliable datagram" protocol

We only really care about tcp and udp but I just copied the file from our server.

TCP is supposed to be used when you tell inetd.conf "stream tcp nowait". udp is usually "dgram udp wait" so I don't understand why the udp is starting at all unless you've done something in inetd.conf or services that you didn't tell us about. You should only have one line in your services for 1022/tcp and none for 1022/udp.

Ron

[Ron Kinner]

OOPS typo: meant /etc/services.old and not /old.

Also I meant to say to be sure that the service name in services and inetd.conf are exactly the same.

Ron

[rick jones]

assuming that you are running this stuff as a child of inetd, after doing an inetd -c check syslog and look for errors.

also, iirc, /etc/nsswitch.conf is only read once - at the first resolution call made by the process. change it after the process is up (eg inetd) and it will not see the change.

so, on the NIS using system that does not work, if you have entered a list in /etc/nsswitch.conf to resolve services locally before/after trying NIS, you might try bouncing the inetd - inetd -c will not bounce the inetd (iric) only re-read the inetd.conf config file.

[Helen French]

Some late night answers, that is great.
Here is what I have done so far this morning...
I copied over the /etc/inetd.conf and /etc/services file from the working system, with a inetd -c to follow. Here is the syslog that follows that. My problem is around the middle of this, being COS_shell/tcp:Unknown service -- what can I do about that is anything?

Aug 23 07:02:40 sdmance6 inetd[2603]: Rereading configuration
Aug 23 07:02:40 sdmance6 inetd[2603]: protocol = udp
Aug 23 07:02:40 sdmance6 inetd[2603]: ntalk/udp: Unknown service
Aug 23 07:02:40 sdmance6 inetd[2603]: ident/tcp: Unknown service
Aug 23 07:02:40 sdmance6 inetd[2603]: kshell/tcp: Unknown service
Aug 23 07:02:40 sdmance6 inetd[2603]: klogin/tcp: Unknown service
Aug 23 07:02:40 sdmance6 inetd[2603]: recserv/tcp: Unknown service
Aug 23 07:02:40 sdmance6 inetd[2603]: instl_boots/udp: Unknown service
Aug 23 07:02:40 sdmance6 inetd[2603]: vopied/tcp: Unknown service
Aug 23 07:02:40 sdmance6 inetd[2603]: bpjava-msvc/tcp: Unknown service
Aug 23 07:02:40 sdmance6 inetd[2603]: COS_shell/tcp: Unknown service
Aug 23 07:02:40 sdmance6 inetd[2603]: protocol = udp
Aug 23 07:02:40 sdmance6 above message repeats 7 times
Aug 23 07:02:40 sdmance6 inetd[2603]: bgssd/tcp: Unknown service
Aug 23 07:02:40 sdmance6 inetd[2603]: registrar/tcp: Unknown service
Aug 23 07:02:40 sdmance6 inetd[2603]: Configuration complete

/etc/protocols are the same on both with identical dates from install. And I haven't made any changes to /etc/nsswitch.conf because the only difference broke server: files nis dns and in the working server: files dns
Next step? COS_shell is in /etc/services I must be missing something?

Thanks again to all involved.

[Ron Kinner]

Christopher,

do

ls -l /etc/services
and
ls -l /etc/inetd*

on both boxes and check permissions and ownership and sizes. Looks like its having problems reading /etc/services.

Maybe you should move the old files somewhere else so there is no chance of them getting read by accident.

Ron

[Helen French]

No luck there there, the permissions are identical accross many machines I have. For fun I also took the old copies out and moved them then re-read inetd.conf with the following result.

# tail /var/adm/syslog/syslog.log
Aug 23 09:49:20 sdmance6 inetd[2603]: recserv/tcp: Unknown service
Aug 23 09:49:20 sdmance6 inetd[2603]: instl_boots/udp: Unknown service
Aug 23 09:49:20 sdmance6 inetd[2603]: vopied/tcp: Unknown service
Aug 23 09:49:20 sdmance6 inetd[2603]: bpjava-msvc/tcp: Unknown service
Aug 23 09:49:20 sdmance6 inetd[2603]: COS_shell/tcp: Unknown service
Aug 23 09:49:20 sdmance6 inetd[2603]: bgssd/tcp: Unknown service
Aug 23 09:49:20 sdmance6 inetd[2603]: protocol = udp
Aug 23 09:49:20 sdmance6 above message repeats 7 times
Aug 23 09:49:20 sdmance6 inetd[2603]: registrar/tcp: Unknown service
Aug 23 09:49:20 sdmance6 inetd[2603]: Configuration complete

[Ron Kinner]

Have you looked at the syslog for your good box recently? Does it have the same unknown service errors for all of these services that you are not concerned about?

If not then perhaps it only reads the /etc/services file at boot and we need to reboot?

Ron

[Helen French]

Thank you to everyone for their help on this. I had to break down and call in a problem. It was discovered that inetd -c didn't actually restart the daemon. So after a kill -9 PID on inetd and a restart re-read the configuration and I am back in business.
inetd -c is supposed to work but there are some cases in which it doesn't apparently.

Thanks again.