Change in Kernel, Now Oracle 7 No Longer Works

[Scott Buckingham]

We have both Oracle 7.3.4.5 and 8.1.7 running on an HP-UX 11.00 OS, HP-9000 K360 machine. Due to increased system load, I upped the nfile and nproc kernel parameters to 5000 and 1000 respectively. After a reboot, I started the 8.1.7 instances (there are 3) and the 7.3.4.5 one. When I came in this morning, I had a couple file systems that filled up (/software and /home) due to non-stop core files occurring and they continue to occur! I'm also getting lots of .trc files in my ORACLE_HOME/rdbms/log directory. I can connect just fine to Oracle 8.1.7, but not 7.3.4.5 (I'm getting ora-12500 errors). It looks like the changing of kernel settings affected this somehow, or is it just coincidence? I've attached one of the trace files, if they will help you diagnose my problem.
Thanks in advance for your help!

[RAC_1]

First what progrm/application is creating core files? What is filling up /software and /home? Are these the directories where oracle starts from?

Check core file with -- file core command.

What is orac-12500 error? You can check with oerr "errorcode"
Post the results.

Changing the kernel parameters should not cause this.

[A. Clay Stephenson]

Building the new kernel should not have caused this. I suspect it is a result of the order in which the instances were brought up. There was probably not a big enough chunk of unfragmented heapspace to grant a dynamic memory allocation request.

This will be especially true if you are running 32-bit applications. You may need to retune some the the Oracle parameters to make the children play better in the schoolyard.

[Steven E. Protter]

The kernel change should not have triggered the problem.

12500 errors are tnsnames type issues right? Maybe something changed in the Oracle configuration or there are other problems with the Oracle database.

Surely you could boot the box off of vmunix.prev and see if the database comes up. If it does, I'm wrong. I suspect it won't and its not a kernel problem its an oracle problem.

We are running a mixed 8.1.7.4/9.0.2.0.4 Oracle environment here, both are 64 bits.

SEP

[Scott Buckingham]

Thanks for the responses, but now I'm really stumped! First, I stopped and restarted the Oracle instances in the order that I have successfully been doing it the past couple years (Oracle 7, then 8.1.7). The results were the same. Then I reverted the kernel back to where it was last week and rebooted (I know, it shouldn't make a difference, and it didn't). I keep getting the ora-12500 error whenever I try to connect to a Oracle 7 database. This is also what gives me my core dump (in /home/oracle). Did my Oracle executable get corrupted? I've looked the error up on Metalink and it suggested that I should use the adapters command to check for Protocol Adapters. This is what I get:

Net Protocol Adapters linked with tnslsnr are:

BEQ
IPC
TCP/IP
RAW

Net Naming Adapters linked with tnslsnr are:

Oracle TNS Names (tnsnames.ora)
Oracle Names


Oracle Advanced Security products linked with tnslsnr are:

Any more suggestions?

[Sridhar Bhaskarla]

Scott,

Well. If it looks like the executables are corrupted, then I would suggest you relink the binaries. It doesn't hurt anything.

Did you install any patches that wouldnt' reboot the boxes?.

-Sri

[Steven E. Protter]

At this point I'd consider the following:

relink the oracle binaries. I've seen them get corrupt for no reason at all. Only the ones that don't work.

Test network connectivity with tnsping

That will check that tnsnames.ora and sqlnet.ora are valid.

Its possible that if you use hostnames in those configuration files(we use raw ip addresses) a change on your network has triggered the problem.

That seems likely.

Let us know how it turns out okay?

SEP

[Scott Buckingham]

No, no patches have been installed for quite some time. Can you outline the process of relinking the binaries? This is new to me as I inherited the DBA duties here (as well as the Oracle 7 environment).

Thank you!

[Volker Borowski]

Hi,

to re-link, backup $ORACLE_HOME/bin first.

cd $ORACLE_HOME/rdbms/lib

check if "ins_rdbms.mk" is present.

make -f ins_rdbms.mk install.

Search Google for "ins_rdbms.mk" and you'll get more detailed instructions.

Be sure not to mix up you ORACLE_HOMES Environment !

Hope this helps
Volker

[Scott Buckingham]

Here's the latest. I went ahead and tried to relink based on Volker's response. This is what I got:

/software/oracle/product/7.3.4.5/rdbms/lib 27 > make -f ins_rdbms.mk
`/software/oracle/product/7.3.4.5/rdbms/lib/psoliblist' is up to date.

What to do now?

[Volker Borowski]

Additional idea !

Do you have seperate Listeners for each database, or a single Listener that serves all of them ?
In this case, you might simply have the wrong listener started up first.
Since each Listener searches his config in
$TNS_ADMIN or if not set
$ORACLE_HOME/network/admin
you might simply run the wrong listener, and the one runnning has not all database configured ?

Check your scripts. DB-Startup might not mean listener startup so is the correct one running.

Check

lsnrctl status
--> This one lists you a log file, which might hold additional information.

lsnrctl services

Second: Actually your cores are not caused by the 12500eds, your tracefile clearly shows an ORA-7445, which relates the core to an OS error. Check the corresponding alert.log file for additional information around these Error-Messages.

Good hunting
Volker

[Scott Buckingham]

Hi Volker.

I checked out what you suggested and this is what I know. I'm using the listener that's associated to my 8.1.7 version. The sequence of events that I've been doing is to start the 7.3.4.5 instance (TST0), then 8.1.7 production, (PRD817), 8.1.7 training (oratrain), and then 8.1.7 test (TST817). After these are up and running, I then start the listener using 8.1.7 settings.

The alert logs are not revealing anything error-related. However, the listener.log has this:

02-FEB-2004 15:16:20 * (CONNECT_DATA=(SID=TST0)(GLOBAL_NAME=aquila_TST0.world)(C
ID=(PROGRAM=OraPgm)(HOST=Windows PC)(USER=robo))) * (ADDRESS=(PROTOCOL=tcp)(HOST
=172.16.1.4)(PORT=1356)) * establish * TST0 * 12500
TNS-12500: TNS:listener failed to start a dedicated server process
TNS-12547: TNS:lost contact
TNS-12560: TNS:protocol adapter error
TNS-00517: Lost contact
HPUX Error: 32: Broken pipe

What is a "broken pipe"?

[Steven E. Protter]

This thread has hope concerning broken pipes.

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=124174

Perhaps some data column or something is too long.

This one has real promise:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=111639

Shridar says had the same problem, relinked, it went away.

He's pretty smart, so I'd say relink the database.

SEP

[Scott Buckingham]

Well, it seems like relinking is the answer, but I can't get it to work. Am I suppose to be an Oracle 7 user (at least, set up with those env vars) or root? I have tried the 'make' commands that are highlighted in this post but I keep getting errors. The errors are basically 'ld' related, as in not using the right one? Which version is the right one?

[Steven E. Protter]

There needs to be an oracle7 user.

That user's environment must have an ORACLE_HOME and ORACLE_SID and other environment variables that are specific to the ORACLE 7 environment. That user should not have anything to do with ORACLE8 on its PATH either.

SEP

[Scott Buckingham]

Thanks, Steven, for hanging in there with me!

I have an Oracle 7 user and here are it's environment settings:

_=/usr/bin/csh
MANPATH=/usr/share/man/%L:/usr/share/man:/usr/contrib/man/%L:/usr/contrib/man:/u
sr/local/man/%L:/usr/local/man:/opt/pd/share/man/%L:/opt/pd/share/man:/opt/ignit
e/share/man/%L:/opt/ignite/share/man:/opt/perf/man/%L:/opt/perf/man:/opt/hparray
/share/man:/opt/pred/share/man/%L:/opt/pred/share/man:/opt/hpnp//man:/opt/omni/l
ib/man:/usr/dt/share/man:/opt/pd/share/man/%L:/opt/pd/share/man:/opt/audio/share
/man:/opt/ansic/share/man/%L:/opt/ansic/share/man:/opt/langtools/share/man/%L:/o
pt/langtools/share/man:/opt/OV/man/itose/%L:/opt/OV/man/itose:/opt/imake/man:/op
t/aCC/share/man/%L:/opt/aCC/share/man:/opt/blinklink/share/man
PATH=/usr/bin:/opt/ansic/bin:/usr/ccs/bin:/usr/contrib/bin:/opt/nettladm/bin:/op
t/pd/bin:/opt/upgrade/bin:/usr/bin/X11:/usr/contrib/bin/X11:/opt/perf/bin:/opt/h
parray/bin:/opt/pred/bin:/opt/hpnp//bin:/opt/omni/bin:/opt/ignite/bin:/opt/langt
ools/bin:/opt/OV/bin/OpC:/opt/imake/bin:/opt/aCC/bin:/software/oracle/product/7.
3.4.5/bin:/bin:/usr/local/bin
ORACLE_BASE=/software/oracle/
EDITOR=vi
LOGNAME=oracle0
ORACLE_SID=TST0
ERASE=^H
EPC_DISABLED=TRUE
SHELL=/usr/bin/sh
ORACLE_TERM=vt100
ORA_NLS32=/software/oracle/product/7.3.4.5/ocommon/nls/admin/data
HOME=/home/oracle0
TERM=vt100
ORACLE_HOME=/software/oracle/product/7.3.4.5
PWD=/home/oracle0
TZ=CST6CDT

When I try to relink rdbms, I get this:

/software/oracle/product/7.3.4.5/rdbms/lib 40 > make -f ins_rdbms.mk install
chmod 755 /software/oracle/product/7.3.4.5/bin
rm -f ksms.s ksms.o
rm -f oracle imp exp sqlldr
cc +O2 +Ofastaccess +Oprocelim -Wl,-PF,linkorderfile -Wl,-aarchive -o oracle -L/
software/oracle/product/7.3.4.5/lib -L/software/oracle/product/7.3.4.5/rdbms/lib
opimai.o /software/oracle/product/7.3.4.5/rdbms/lib/config.o -lserver -lclient
-lcommon -lgeneric -lknlopt -lslax -lapps -lcog -lcox -lidl -lknlde -lpkg -lpls
-lsem -lsyn -licx -lndw1 -lserver -lclient -lcommon -lgeneric -lknlopt -lslax -l
apps -lcog -lcox -lidl -lknlde -lpkg -lpls -lsem -lsyn -lserver -lclient -lcommo
n -lgeneric -lslax -lapps -lcog -lcox -lidl -lknlde -lpkg -lpls -lsem -lsyn -lse
rver -lsqlnet -lncr -lsqlnet -lclient -lcommon -lgeneric -lsqlnet -lncr -lsqlnet
-lclient -lcommon -lgeneric -lepc -lepcpt -lnlsrtl3 -lc3v6 -lcore3 -lnlsrtl3 -l
core3 -lnlsrtl3 -l:libcma.sl -lcl -lm -l:libcl.a -L/usr/lib -lm -lstublm -lm -l:
libcl.a -l:libdld.sl /usr/lib/libnsl.sl -l:libnss_dns.1
/usr/ccs/bin/ld: Can't create oracle
/usr/ccs/bin/ld: Permission denied
(Warning) Optimizing ADDILs in the presence of the debug information. Debug info
rmation may be corrupted (1594)*** Error exit code 1

Stop.

What's my next move?

[Volker Borowski]

Hmm,

failed to start dedicated server process can mean no OS-resources or to low value of "processes" in init.ora.
Re-check /etc/system. You changed nproc (in a way that seems not dangerous!) May be a formuala-depended parameter ran over a 32Bit-limit and is now very low for a nonobvious reason ?!?! I.E if it results in a value of 65535+2000 it may in fact only be 2000 !
If possible attach /stand/system.

LogIn with full login-scripts executed for your ora817 user. Goto
cd $ORACLE_HOME/network/admin

check the listener.ora
This file say how the listener distributes incoming requests to the ORACLE_HOMES attached to the corresponding databases.
So the TST0 database must refer to the ORACLE7 Oracle-Home, so that for an incoming request the oracle-process for the vers7 DB is launched.
Please attach this file.

LogIn with full login-scripts executed for your oracle7-application user. Goto
cd $ORACLE_HOME/network/admin.
If ORACLE_HOME is not set, goto $TNS_ADMIN.
Check tnsnames.ora in this location (which CAN be diffrent from the directory above).
attach this file as well, esp. if the connect data for the same SIDs look diffent.

Check the tnsnames.ora on the faulting pc 172.16.1.4. In this config the GLOBAL_NAME is set as "aquila_TST0.world". Check this against the file sqlnet.ora in the config directory of your listener.

Volker

[Volker Borowski]

Hi,
This re-link fault looks strange.
You did the backup of ORACLE_HOME/bin I'd mentioned before did you ?

I just re-read the thread and it comes to my mind, that I did not mention the point that you need to shut down the database before you re-link it. That is the problem with the things you do all the time. You miss the things that are obvious to yourself :-(

The relink fails, because it can not acces the oracle-file (see message). This may be a permission problem. But it may be the problem, that the file can not be accessed because it is still active.

Did you shut down the database before the re-link ?

Volker

[Steven E. Protter]

I owe you an apology.

I assumed that you would know that to relink the database it must be down.

Whenever I assume I become the first three letters of the word.

Though, if you can't bring the database up, its not the most unreasonable assumption.

Stop the database
kill all processes such as smon associated with the database.
relink

report back.

SEP

[Scott Buckingham]

Thanks Volker and Steven. I thought I shut down the database, but just the Oracle 7 instance. Do all others (there are two 8.1.7 instances running) need to be shutdown as well?

[Volker Borowski]

SEP, never mind, I was in the same boat :-(

[Volker Borowski]

Scott,

if you go to $ORACLE_HOME/rdbms/lib

ALL databases running from this specific ORACLE_HOME need to be down !

So if you have been the oracle 7 user, ORACLE_HOME pointing to the oracle 7 SW and you have been in the Oracle 7 ORACLE_HOME, all is ok if the oracle 7 db was down.

beside the error :-)

Volker

[Scott Buckingham]

I believe we are OK as there is only the one instance of Oracle 7 (user is oracle0) and I'm positive that I shut that one down prior to trying to relink. I also do not see any oracleTST0 processes in the process list. So there must be something else not quite set right for my Oracle0 user?

Boy, when I screw things up, I go all out!

[Volker Borowski]

Scott,

check your environment

Oracle 7.3.X, 8.0.X, 8.1.X or 9.X.X:
- Set SHLIB_PATH to include
$ORACLE_HOME/lib
- If using 64bit 8i Oracle,
SHLIB_PATH should also include
$ORACLE_HOME/lib64.
- ensure LPATH is unset

Check the permissions on the oracle-executable.

% cd $ORACLE_HOME/rdbms/lib
% make -f ins_rdbms.mk install

If it fails again, it might be a patchable problem with the HP-linker. If it does not work at all, restore the backup from before the re-link-tries and work on the SQL-NET config first.

New day for me now.
I hope SEP can still give you a helping hand.
I'll check this thread tomorrow morning.
Good night
Volker

[Steven E. Protter]

Victor's last post was a good one.

It is what I would say. I'm plum out of ideas right now, but if something pops up, I'll post it.

Perhaps a search of http://metalink.oracle.com or
http://technet.oracle.com

SEP