Network disconnect: ORA-03114 not connected to ORACLE

[Aziz Zouagui]

Greetings,

Recently, we performed the following upgrades without any problems.

-Upgrade Oracle from 9.2.0.5 to 9.2.0.7
-Upgrade HP-UX from release 11 to 11i (11.11)
-Replaced the rp7400 with the rp7410.

All these upgrades were done individually within 3 to 4 weeks apart, everything was running smoothly after.

Except that now we get ORA-03114 from our batch servers, which are constantly connected to Oracle running on the rp7410.

This happens randomly, it could take a day or 2 before we get this error. The other puzzling thing is that we have 4 batch servers on 4 different NT boxes that connect to Oracle and sometimes they all disconnect and sometimes only one of them disconnects with the error showing on the screen.

The vendor is blaming the application but the application people are blaming the vendor, point out that something definitely wrong with the network some place.

We also have HP APA (auto aggregation software) running on 2 1000 base sx cards.

Recently, we applied this patch PHNE_33116 thinking that somehow this is related to the APA.

My network admin has Solar Wind analyzing software product running on the LAN to check for anomalies. Every once in a while, he would see a small blip of packet losses coming from the HP rp7410 and they disappear. Sometime, the error on the batch server occur during the blip of packet losses and sometimes they appear with showing anything.

Anyway, we are puzzled because we can't put a finger on this problem.

Does anyone have any ideas on where to look next.

Thank you in advance.

Aziz.

[Victor Fridyev]

Hi,
It's very hard to find real reason of the case, but try to run on the server 4 listeners, which listen on different ports and redirect each NT box to its own listener.
I'm not 100% sure that this will help, but this may decrease number of such failures.
HTH

[Delrish]

Take a look at metalink Note:17613.1
It is full of information in regards ORA-03114

[Aziz Zouagui]

Thank you for your replies.

We haven't had the chance to test setting up different listeners for each process server because nothing has changed on that end.

Also, I have read the Oracle Metalink Note, it mentions that this error is broad and that more information is needed to determine why it is happening.

At this point I am almost ruling out the Application side and the Oracle side.

Any other ideas ?

Thanks again in advance.

[Sameer_Nirmal]

Hi,

Are you using "DBLINKS" ?
What sort of "application tier" you have?

It is hard to predict the possible cause in your case just knowing "ORA-03114" error.

This oracle error is always preceeded by another oracle oracle errors which makes the scenario much clear. You need to check oracle alert log trace log to know those other errors. Moreover you can check listener or SQL net logs.
F.E. somtimes this oracle error is preceeded by "ORA-03113: end-of-file on communication channel" error. In this case the oracle instance itself is down.

I guess the strategy could be drill down from the errors you get and put something in place to debug at host,DB,application level whichever is possible.

As fas as the network debuggins is concerned, I guess "tcp" protocol ( used by Oracle TNS ) debugging could be done besides others to assessment at network level. At HP-UX side,you can enable network debugging using nettl and netfmt at various intervals for a specified time( depending on the previous occurances).
Looking at DB and application logs also help to know the cause evenif they doesn't seems to be the cause of the problem.

Since the APA is being used, you can check the errors in syslog , demsg regarding the connection binding (depending on how APA is configured.) If possible , I would disable APA after going through above/other factors and see if error occures.

[Yogeeraj_1]

hi aziz,

in case you want to trace your network connetions, you can also use nettl utility.

attached a document containing more information

kind regards
yogeeraj

[Aziz Zouagui]

Hi Everyone,

We are going to disable the HP APA to eliminate this last variable. The packet losses are only showing on the 2 HP systems that are running HP APA.

Will let you know what we find out.

Again, many thanks for all the suggestions.

Aziz.