Oracle listener rerouting to wrong port

[Rita C Workman]

I have never seen this, but this morning it sent me & a few others folks here scrambling.

This morning folks could not get logged into their application. Unable to connect. At first I figured network (chk'd speed and duplex on all ends - all ok). Everything else up and networking fine..

Everything was up and seemed started ok. Some folks could get connected while others hung and finally died. On checking, one DBA found when he ran a trace, that it seems as though Oracle (which listens on port 1521) was re-routing to port 1537. And to the best of everyones knowledge their is nothing config'd for 1537. It doesn't exist in /etc/services....everyone claims they do not reference with a getservice() or anything to such a port.....but there it was.
We checked the client tnsnames.ora file and all was ok. Nothing had changed with the app. No strange messages in listener.log. Listener log showed it started (listening on 1521)......but almost nobody could get connected. A couple made it through, but most...nothing.
We got it going by simply shutting down all Oracle instances and restarting everything......bang....all ok now.

My question, Any ideas what happened? Honest there is nothing in any logfile indicating a problem or anything out of the norm. Or any ideas what else I might look at?

HPUX 10.2 2-Oracle instances 8.0.5 and 8.0.6

I'd feel better if we could figure out what happened.....but so far we keep drawing a blank.
.

Thanks,
Rita

[harry d brown jr]

Rita,

Too bad you didn't run an "lsof", it would have shown exactly what ports were opened and by whom.

live free or die
harry

[Jeff Schussele]

Rita,

If the listener.ora file on the server has no reference to 1537, I would suspect that a DBA inadvertently started a listener on 1537, probably manually. Most likely they stopped or killed the 1521 listener accidently & restarted on 1537 manually.

My 2 cents,
Jeff

[Ian Dennison_1]

Rita,

This is a lateral thought, that might have some bearing on this situation.

Try telneting to the server name from the network segments that are having problems. If even the basic telnet is having problems connecting, then it is network level rather than Listener level. Try a 'traceroute [IP Address]' or a 'tracert' in DOS, have a look at the path taken.

If telnet gets rejected, check who else is using the same IP Address. The NT administrators should be your first port of call for investigating this ('scuse the pun).

Share and Enjoy! Ian

[Rita C Workman]

Harry:
Well actually I did. But nothing appeared out of the ordinary. I couldn't differentiate enough to determine if it was the one regular Oracle running or the custom app. And the clock was ticking.
The only thing we could find is that trace showing that the listener was re-routing to port 1537.....which doesn't exist and isn't used-so nothing every connects to it.
So the question became .. why was the listener trying to re-route itself????
And since this put hundreds of users down...we had little time to research. We had to get things back up (fast). So what I'm hoping for is any ideas on what to look for first in the future. Of course I hope I don't have this happen to me again...but any ideas what you'd check for first?

Jeff:
Good thought, but on checking the listener.log it showed listening on 1521. And when this started, the DBA's weren't even in yet. Unfortunately, just me that early.

Rit

[Jeff Schussele]

Rita,

Are you saying the server was BOTH listening & sending on 1537.
OR listening on 1521 & sending on 1537?
Not sure I'm clear on this.
Also you mentioned "custom app" is there an Oracle app that's listening on 1537 at some point?
Is there a possibility that a "power-user" may have fired it up & it "killed" the std Oracle listener thus leaving only the "custom" to listen?
Also any unexplained time lapses in the listener log?

Jeff

[Rita C Workman]

No, according to logfile it was listening on 1521. But according to trace it 'heard' on 1521 then routed to 1537. Why it did this nobody seems to know (or isn't admitting) yet?
The only 'power user' would be me (root) or any DBA. If there is a developer that has managed to get certain rights, I am not aware of this.
Lapses in logfile is something I hadn't looked at...I will.
Thanks,

Rita

[Rita C Workman]

Ian,

Networking was the first thing I did look at, as it gave all the indications of this at first glance. The NT Administrator was on top of it and was able to rule that out quickly.
Everything else was working fine on all the other systems. No problems with those Oracle databases or connecting to them. Just on this server and just with these Oracle instances.

To me, it looks more like something with an application call. But proving it...

Rita

points to follow soon..

[Sridhar Bhaskarla]

Rita,

I would use ethereal to capture the traffic for few minutes before I fix the problem. It will clearly give the tcp stream from which we can determine what was happening. I simulated this kind of situation before in my old office where I tried to configure a secure channel using ssh's port forwarding feature.

-Sri

[Rita C Workman]

Sridhar,

I have ethereal set up on some other boxes..I'll make sure it's set up on this one for the future.
This is the first time I have ever run into something like this....

Guess I'll have to wait till it happens again (hopefully never)..

Thanks all - Closing this Post
Rita

[MANOJ SRIVASTAVA]

Hi Rita

Here is a rookie way of viewing it .

Oracle has the following two config files which determine the port nos. and they are hard coded in these files .
Listner.ora ( Server side connection )has :

Port #
protocol ( TCP/IP)
instance (id)
execuablte directory ( /opt/oracle/product)

tnsname.ora ( Client Side connections to the Server)

host < IP AD>
port #
protocol
sid

so essentially I am asking you to look for 1537 in tnsname files for instances ( if both gave this problem)


Manoj Srivastava

[Tom Danzig]

Check the /etc/listener.ora file for a misconfigured port number.

[harry d brown jr]

Rita,

have a look at this, but more importanly the next paragraph:


http://www.dbatoolbox.com/WP2001/ccmgrs/Walking_the_CCMgr_Tightrope.pdf

And here is why your DBA found or thought they found it listening on a different port:

You make a request for a connection to port 1521, the
listener receives the connections, negotiates with the database to listen on a
new port greater than 1024, and then replies back to the client with the new
port number. The client then connects directly to the database over the new
port.

-Allowing port redirection through your firewall would require yo u to open up
all the ports above 1521

http://www.appsecinc.com/presentations/oracle_security.pdf

I search google with this string: "+tnslsnr +port +listens +firewall"


live free or die
harry

[Tom Geudens · Accepted solution]

Hi Rita,
I know you closed the post ... but I still want to add a thought. You said you restarted all the Oracle instances. Now that seems - to me - a bit drastic. I would have tried restarting the listener (ok, the "hanging" connections would have to wait for a timeout, but the downtime for the new connections would have been less).

Just a thought,
Tom Geudens

[Rita C Workman]

BELOW HOLDS THE FINAL SUMMARY AND SOLUTION.....

Manoj,
The tnsnames.ora was/is set up correctly.

Harry,
That is correct. The listener was not 'on' right..so it went to a higher port (1537), but this port was not enabled so things began to hang. The firewall is set to only turn on exact ports used...not open the gates. So the client was left hanging and unable to re-route through a different port.

Tom,
I agree real drastic. Fact is that wanted me to spin the whole box and we got them to just 'spin' Oracle. Had they looked correctly at the listener.log they would have found the oracle error that I found later looking through it myself. (Guess it's true-always check it yourself !) Turns out the listener never started right and the oracle error was right there....
The only needed action to be done was as you say...just restart the listener correctly !

Rita

..mystery solved..

Moral-never believe when somebody tells you the log is clean-look for yourself

[Jeff Schussele]

"Experts" corollary:

Especially when it comes from DBAs or developers.

Cheers,
Jeff