Test Port 1691 does not appear to open

[Acxiom Unix Team]

I am having big problems in trying to install CA Softwares SysEdge agent on a HP 11.v3 server. The agent will not start but i want to be able to test that the port it is trying to use (1691) will open when requested. The log files produced by the agent are next to uselss and dont provide any information as to why it wont start.

Networking is not my strongest skill so could do with a little advice please to help check out some things.

Looking at output on netstat -an | grep 1691 it returns nothing which i think indicates that the port is NOT open in anyway? Correct? So i am trying to figure out how i can prove that this socket/port will open when required by a service or is there a way of forcing 1691 to be open? Is there any way to do this?

Any suggestions would be greatfully received.

[Matti_Kurkela]

Firewall administrators often talk about ports being "open", meaning that a firewall is configured to allow traffic to pass to those ports, or "closed", meaning that the firewall will block connections to that port before they reach their destination. Whether there is a program listening for incoming connections or not is unimportant for the firewall.

But within the operating system, the terminology is a bit different, and this difference seems to be causing some confusion: quite a few people seem to understand ports in the firewall sense, and will try to apply this information to the OS-level networking ports. This is incorrect and will cause much confusion.

At the operating system level, a TCP or UDP port can only be "open" if there is something using it. Usually, the "something" is a regular userspace process, but in certain very special cases (mainly NFS) a kernel driver may use network port(s) directly.

A TCP or UDP port is opened either when a process uses the listen() system call to set it up for receiving incoming connections, or when it uses the connect() system call to set up an outgoing connection. From the viewpoint of the TCP/IP networking drivers, opening a port means allocating packet buffers for it; in case of TCP, the driver must also track the connection state, process any necessary acknowledgement packets and resend already-sent data if the recipient does not acknowledge it in a timely manner. All this work is pointless if there is no process (or in-kernel equivalent) to produce outgoing data and/or receive incoming data. Therefore, having a port open in the OS sense with no application using it would just be a waste of system resources, and would make the system incredibly vulnerable to denial-of-service attacks over the network (like the Ping-of-Death attack that used to be a big issue for the Windows 95 systems before it was fixed, only worse).

So, what exacly you want to do? If you specifically want to test the OS behavior regarding port 1691, setup some other program to listen on that port and see what "netstat" says. (Google can find you a lot of small Perl scripts which can be used to listen on a particular port.)

But the networking code treats all ports exactly the same (okay, ports 0-1023 are restricted for root user only, but that's not relevant here), so I don't see what could stop that particular port (only) from opening. Software firewall products like ipfilter don't generally prevent the ports from opening in the OS sense, because that would often cause applications to die; rather, they stop the traffic before it actually reaches the part of the OS that "runs" the port. From the point-of-view of the application listening on a firewalled/ipfiltered port, the port opens just fine, but no traffic ever comes in.

(This is how a firewall/ipfilter can protect a legacy application that insists on listening on all the network interfaces the system has and crashes if the port cannot be opened. This situation used to be very common in the early days of firewalls.)

If you want to test connectivity while the SysEdge agent still refuses to run, one possible solution would be to dump the incoming network traffic on this system while attempting to connect to port 1691 from another host. The expected result would depend on the protocol; if the SysEdge agent uses TCP (which is usually the most likely choice) you would see an incoming TCP SYN packet from the remote host to port 1691 of this host, and then you would see this host sending a TCP RST (ReSeT) in response, since there is nothing listening on that port. If the agent uses UDP, the response would be an ICMP error packet instead, since the UDP protocol does not define any universal "no service available" respones: after all, its purpose is to allow the protocol developer a complete freedom if TCP is not suitable for his/her requirements.

Another way to test connectivity would be to temporarily configure some other application or service (one that you understand very well) to port 1691 and then test it from another host. That would be an "end-to-end" test: it would simultaneously test both the network connectivity and the port behaviour.

[Akio Kabutogi]

If the application's socket is binding to other port than expected, you don't see the entry in netstat -an.

lsof will show you the owner of the port if my memory is correct. If you find other program owning the

port, the other application can not use that port. But from what you said over netstat -an, it looks

unlikely...

It's unclear about ". The agent will not start but i want to be able to test that the port it is trying to use (1691) will open when requested. ".  Does this mean the server side of the program does not work well generating some errors that indicate

bind() failing ? Or the client programs are simply failing with some footprint indicating the server side port is not open ?

Please make this point clearer so that I can have proper thought over this problem.

[Acxiom Unix Team]

 Hi Akio,

What this agent is meant to do is start a process like this one on the server.

root 10161     1  0  Oct 12  ?        15:48 /opt/EMPsysedge/bin/sysedge -b -p 1691 -l /etc/sysedge.lic -f /etc/sysedge.cf -m /etc/sysedge.mon

It uses port 1691 for SNMP traffic as the configuration file changes the defaults (161/162) to this for some reason. On the server i am having the problems with this process is just not being created. We see nothing in the sysedge log (which is useless) or anything else. On the server i got this example from above if i do an netstat -an i can see this port detailed;

root@uk201:/> netstat -an | grep 1691
udp        0      0  *.1691                 *.*
udp        0      0  *.1691                 *.*

On the server where the process does not start, we get no response to the same command.

So i was hoping to prove somehow that this is not an OS issue. The servers it is trying to send snmp traps to/from is in a different subnet so i have looked at firewalls to see if anything is being blocked and it isnt.

[Matti_Kurkela]

So you have another server that works OK. That is good: you can run comparisions.

Most veterans of this forum would prefer to see the logs for themselves to judge their uselessness: what is useless to you might contain essential clues to someone who knows the application well. But I assume you mean the log only contains copyright messages, and a comparision of the logs between the working server and the failing server does not yield any clues either. (For example, what is the next message logged in the working server = the first log message that is missing in the failing server?)

But if the sysedge logs and error messages are not helpful, and there are no obvious cries for help in the syslog or kernel message output (run "dmesg"), it's time to take a bigger hammer.

You'll need the "tusc" tracing utility:

http://hpux.connect.org.uk/hppd/hpux/Sysadmin/tusc-8.1/

You should look at the start-up script of the SysEdge agent to find all the environment settings it may need, and set your shell environment to match that. Open another shell session, just in case something goes wrong and you'll have to kill the traced process.

Then start the agent through the tusc tool: (In this example, the tusc output will be directed to /tmp/sysedge-trace.txt. Choose a location with plenty of disk space, just in case.)

tusc -f -k -o /tmp/sysedge-trace.txt /opt/EMPsysedge/bin/sysedge -b -p 1691 -l /etc/sysedge.lic -f /etc/sysedge.cf -m /etc/sysedge.mon

 Do this on the server where the agent fails, and again on the server where the agent works. You should stop the real agent before running this test, to prevent it from conflicting with the agent you're starting for this test. On the server with the working agent, stop the agent as soon as you can confirm it has completed starting, as the trace output will grow without limit as long as the traced copy of the agent is running.

The trace is likely to slow down the agent a lot, so be patient.

The tusc tool will provide a record of each system call the agent makes, and its results. It will include each file the agent accesses as it starts up, and all the network operations the agent does. You can then compare the results of the successful and the failing runs: although the exact numbers in the outputs will be different, both runs should start essentially the same way. They should behave the same until the failing one encounters an error that prevents it from continuing. Try and find the point where the failing run becomes different from the successful one, and then see what happens just before the derail. If it opens, reads or writes files before the error, you might want to check the existence, permissions and contents of those files.

The trace outputs are rather technical and may be very long: it might be very helpful if you could get someone with Unix programming experience to look at the traces with you. If you can't analyze them on your own, please attach both traces to this thread.