Oracle listener rerouting to wrong port

[Rita C Workman]

I have never seen this, but this morning it sent me & a few others folks here scrambling.

This morning folks could not get logged into their application. Unable to connect. At first I figured network (chk'd speed and duplex on all ends - all ok). Everything else up and networking fine..

Everything was up and seemed started ok. Some folks could get connected while others hung and finally died. On checking, one DBA found when he ran a trace, that it seems as though Oracle (which listens on port 1521) was re-routing to port 1537. And to the best of everyones knowledge their is nothing config'd for 1537. It doesn't exist in /etc/services....everyone claims they do not reference with a getservice() or anything to such a port.....but there it was.
We checked the client tnsnames.ora file and all was ok. Nothing had changed with the app. No strange messages in listener.log. Listener log showed it started (listening on 1521)......but almost nobody could get connected. A couple made it through, but most...nothing.
We got it going by simply shutting down all Oracle instances and restarting everything......bang....all ok now.

My question, Any ideas what happened? Honest there is nothing in any logfile indicating a problem or anything out of the norm. Or any ideas what else I might look at?

HPUX 10.2 2-Oracle instances 8.0.5 and 8.0.6

I'd feel better if we could figure out what happened.....but so far we keep drawing a blank.
.

Thanks,
Rita

[harry d brown jr]

Rita,

Too bad you didn't run an "lsof", it would have shown exactly what ports were opened and by whom.

live free or die
harry

[Jeff Schussele]

Rita,

If the listener.ora file on the server has no reference to 1537, I would suspect that a DBA inadvertently started a listener on 1537, probably manually. Most likely they stopped or killed the 1521 listener accidently & restarted on 1537 manually.

My 2 cents,
Jeff

[Ian Dennison_1]

Rita,

This is a lateral thought, that might have some bearing on this situation.

Try telneting to the server name from the network segments that are having problems. If even the basic telnet is having problems connecting, then it is network level rather than Listener level. Try a 'traceroute [IP Address]' or a 'tracert' in DOS, have a look at the path taken.

If telnet gets rejected, check who else is using the same IP Address. The NT administrators should be your first port of call for investigating this ('scuse the pun).

Share and Enjoy! Ian

[Rita C Workman]

Harry:
Well actually I did. But nothing appeared out of the ordinary. I couldn't differentiate enough to determine if it was the one regular Oracle running or the custom app. And the clock was ticking.
The only thing we could find is that trace showing that the listener was re-routing to port 1537.....which doesn't exist and isn't used-so nothing every connects to it.
So the question became .. why was the listener trying to re-route itself????
And since this put hundreds of users down...we had little time to research. We had to get things back up (fast). So what I'm hoping for is any ideas on what to look for first in the future. Of course I hope I don't have this happen to me again...but any ideas what you'd check for first?

Jeff:
Good thought, but on checking the listener.log it showed listening on 1521. And when this started, the DBA's weren't even in yet. Unfortunately, just me that early.

Rit

[Jeff Schussele]

Rita,

Are you saying the server was BOTH listening & sending on 1537.
OR listening on 1521 & sending on 1537?
Not sure I'm clear on this.
Also you mentioned "custom app" is there an Oracle app that's listening on 1537 at some point?
Is there a possibility that a "power-user" may have fired it up & it "killed" the std Oracle listener thus leaving only the "custom" to listen?
Also any unexplained time lapses in the listener log?

Jeff

[Rita C Workman]

No, according to logfile it was listening on 1521. But according to trace it 'heard' on 1521 then routed to 1537. Why it did this nobody seems to know (or isn't admitting) yet?
The only 'power user' would be me (root) or any DBA. If there is a developer that has managed to get certain rights, I am not aware of this.
Lapses in logfile is something I hadn't looked at...I will.
Thanks,

Rita

[Rita C Workman]

Ian,

Networking was the first thing I did look at, as it gave all the indications of this at first glance. The NT Administrator was on top of it and was able to rule that out quickly.
Everything else was working fine on all the other systems. No problems with those Oracle databases or connecting to them. Just on this server and just with these Oracle instances.

To me, it looks more like something with an application call. But proving it...

Rita

points to follow soon..

[Sridhar Bhaskarla]

Rita,

I would use ethereal to capture the traffic for few minutes before I fix the problem. It will clearly give the tcp stream from which we can determine what was happening. I simulated this kind of situation before in my old office where I tried to configure a secure channel using ssh's port forwarding feature.

-Sri

[Rita C Workman]

Sridhar,

I have ethereal set up on some other boxes..I'll make sure it's set up on this one for the future.
This is the first time I have ever run into something like this....

Guess I'll have to wait till it happens again (hopefully never)..

Thanks all - Closing this Post
Rita