Oracle upgrade in SAP with in MCSG-environment

[Peter te Focht]

We upgraded our Oracle-database from 8.0.5 to 8.1.7.
It is the base of a SAP/R3 environment in a MCSG-cluster. We are not able anymore to bring the package up, database starts and workprocesses of SAP are started up, but then a signal comes from ???? and stops the package again. What should we change in /etc/cmcluster/.... ? It looks like an time-out.
Kind regards Jan Poortman

[melvyn burnard]

So did this all work before?

What other changes have you made?
What version of SG and what patch level?
What signal are you referring to?
Has any of your environment variables, or paths changed after upgrading the Oracle version?

Try bringing the package up in DEBUG mode.

[Peter te Focht]

Hi melvyn, here are the answers:
1-It worked fine before, also switching on or off the package.
2-No other things were changed.
3-SG-version=A11.09 and the ux-patch-level=B11.00.48
4-I changed the environment variables for the oracle-user. Oracle-home is extended to /oracle/E13/817_64 instead of /oracle/E13.
5-I am able to mount the filesystem without starting SAP. Then, when I start the listener, I can start SAP. The startup however is very, very slow and in a log-file the system complains about a network-error.
I hope you recognize something.

[melvyn burnard]

I would now strongly suggest you log a call with your local HP Response Centre, as this could get very involved.

regards
Melvyn

[Peter te Focht]

OK Melvyn,

I already had a bad feeling about this all.
Thanks for your help, maybe there are some notes from SAP about this subject.
When everything is solved, I will forward it to this place to avoid victims of oracle-upgrades in future.
regards, Jan

[Volker Borowski]

Hi Jan,

had the same problem and was OK after reboot (sounds like NT does it ?).

Did you upgrade the SAP Kernel in addition ?
If you switched from lower than 4.6D to 4.6D you SAP loads will be invalid now, this is why everything has to be recompiled and appears to be slow.
You might need to schedule the generation in transaction SGEN.

Hope this helps
Volker

[Peter te Focht]

Hallo Volker,

Thanks for your help. I just took the machine out of the cluster and started a reboot.
When it is successful, I will send a reply about the results. I don't hope ux is going to behave as NT.
cheers Jan.

[Peter te Focht]

Hello Volker,

I hoped it would work, but it didn't. It is the same situation as before. I will start the package now in debug-mode and start SAP by hand. Then I can investigate all log-files without angry clients behind me. The story will be continued.
Regards, Jan

[Volker Borowski]

Ah, another thing that I remember,...

If you have distributed database and central instance, you should keep in mind, that on the CI, ORACLE_HOME might still be wrong (or may be better "correct" now, but the directory is not there).

May be you adapted the environment to the new oracle home, but the CI now looks for tnsnames.ora in a diffrent directory /oracle/SID/817_64/network/admin, which might not exist on the CI host ? You might need to adjust the CI and the application servers as well.

Volker

[Peter te Focht]

Well, I can run the package in the following way. I edit /etc/cmcluster/E13/dbci.cntl, where I commented out the lines that start the sapdbci.cntl. I execute the cmrunpkg-command and the filesystem will be mounted. Then I start as ora the listener and then as adm I start SAP.
Everything runs without complaining. What is the difference in starting with sapdbci.cntl and a hand-start?I don't use the MCSG-monitor-function, so I cannot understand why the package stops again. I also see that the database is running for a wile. During the startup of the DW-processes something is crashing, for during that period the package gets a signal to stop it.

[Volker Borowski]

A reason fro the deaths of the dw-processe you will fin in the developer traces.

/usr/sap/SID/DVEBMGSnn/work/dev_*

check

dev_disp (dispatcher)
dev_w0, dev_w1 ( the first two work processes )
dev_ms (message server)

Another reason might be, that aftre the upgrade you might need to recreate the OPS$ connect. There should be a script oradbusr.sql on the RDMS-CD and you should set the password for SAPR3 with "sapdba -sapr3
Volker

[Peter te Focht]

Good morning,
Volker, thanks for your hints. I already checked the processes you mentioned. The typical thing is that every thing works fine when I start the package, without startup of SAP. When I start listener and SAP by hand, everything works fine. So I don't know whats the reason of shutting down SAP. I will try to start SAP with R3trans -d and look what happens then.
Have a nice day, Jan

[Peter te Focht]

Here is attathed the logfile of the startscript during a startup-faile

[Volker Borowski · Accepted solution]

Hi Jan,

ok, this gives us a closer insight.
The sapcpe program is responsible to upgrade the sap-kernel on application servers, if they have a newer version. Any information in this sapcpe.log file, which is mentioned in your clip ?

I would suspect a permission problem with some programs here. Most likely this is related to either a permission problem due to the NFS-mounts of service guard, or the suid-programs like br*, sapdba or saposcol.

May be a way out is to upgrade the ctrun-executables manually and adjust the permissions afterwards.

Hope this helps
Volker

[Peter te Focht]

Hi Volker,
Well here are these two log-files. I never looked at them, but they seem to be quiet important. Especially in this case.
I think your answer will help me, I'm going to check some things and let you know how it works.
Attached are the two logfiles
Cheers, Jan

[Peter te Focht]

Volker,

Tears in my eyes. It still doesn't work. I changed the ownership of dbadaslib.sl, I don't know what this file will do in an oracle-environment. One error message disappeared, but the package still shuts down.
I will finish now for this week. Maybe during the weekend I get some fresh ideas. Monday we will see again. Whishing you a nice weekend,
regards, Jan

[Volker Borowski]

Hi Jan,

sorry for being late, I was at a customers site with no internet access...

Well this looks strange, because dbadaslib.sl belongs to the ADABAS D or SAPDB database. Since you run oracle, there should be no need to have this thing at all. In your case a dboraslib.sl should be needed.

As you see in the beginning of the logfile all the other databases are excluded:
inf=informix
db2=db2
db6=AS400 DB

Could this be an old installation fault ?

I think you should be able to either exclude this file in one of the lists, or delete it at all (keep a backup copy :-)

Volker

[Peter te Focht]

Hallo Volker,

I'm back again for just 2 weeks and in the meantime my collegue updated the SAP-kernel of two other instances.
The result was the same as in the above mentioned one. So we think that the problem is not the oracle-upgrade, but the new SAP-kernel. This is SAP-patvh-level 1225. Today we will change de the dispatch&work-process file to an older version.
If the problems disappear we are sure about the cause of the problems. I will keep you in touch.
Best regards Jan

[Peter te Focht]

Hi Volker,

I just downgraded the disp&work executable to patch-level 944. Everything is working well again. Patch-level 1166 have the same problem as 1225, we used now.
The cause is probably the fact that after starting up the database, four mainprocesses are coming and the start a lot of dw-processes. With the old kernel that happens very fast and with the new kernel, the startup-time between the four main-processes and the child-processes is about one minute. So I assume that in MC/ServiceGuard there is a time-out and that might be the reason of killing the whole process and stopping the package.
I'm happy that it was not the oracle-upgrade.
But what now ?????
Best regards, Jan

[Peter te Focht]

Hello Volker,

We solved the problem. After patch-release 1078, SAP implemented a network-check in the disp&work-process. In our system it found 2 network failures. The time-out was 32 sec's. So together 64 sec's, that's our one minute. It was caused by a /etc/resolv.conf file, that has been placed a long time ago, when we tried to setup DNS. However we disabled DNS again and left the file where it was. After removing it everything goes in the right way again.
Although, thanks for your help.
Greetings, Jan