SAP not starting up

[AL_3001]

Hi Friends,
We have a wekly offline backup, where the sap is brought down by the script and than the backup start.After the backup, the script starts sap. However, recently there has been problems with starting up sap after the offline backup. The scripts in itself is fine. The error that is received is as follows:
------------------------------
The system failed to restart due to the error below:
**************************************************
PXA_NO_SHARED_MEMORY -- 08.03.2008 at 22:33:23
205748 kilo bytes of shared memory for PXA not available.
**************************************************

The system starts up but in a Zombie state and is not usable.
------------------------------

It seems, it cannot find enough memory to start up. Application team is blaming the unix team, saying there is problem with memory allocation. Kindly advise how shud we try to diagonse this unusual error at unis end.

Inputs would be really appreciated.

Thanks.

[Dennis Handly]

>it cannot find enough memory to start up

Is this a 32 or a 64 bit application?
What does "ipcs -ma" show?

>The system starts up but in a Zombie state

Do you mean a Unix zombie?

[AL_3001]

Attached is the output of ipcs.

[YAQUB_1]

Hi Ashish,

>>Application team is blaming the unix team, saying there is problem with memory allocation.

I strongly protest Ur SAP team, because with any evidence why they are blame U.

Please cross check with Ur SAP team, login a SAP administrator & run the command st22 and analysis the output & U find out what is the actual problem. After the analyses if they (SAP team) want change the abap/buffersize parameter using the st02.

[Note:- If possible pls send me st22 O/P]

Thanksâ Yaqub
HP Support!!!

[Eric Antunes]

Hi Ashish,

The solution here is to restart the server because your Oracle instance wasn't shutdown correctly (you should check why).

Best Regards,

Eric Antunes

[Dennis Handly]

You have a bunch of large segments:
519 0x91246e28 --rw-rw---- orarsp dba orarsp dba 54 1064181760
525 0x00002713 --rw-r----- rspadm sapsys rspadm sapsys 20 55059200
20503 0x00002716 --rw-r----- rspadm sapsys rspadm sapsys 18 210686772

[Hein van den Heuvel]

Ashish,

Check out:

http://forums12.itrc.hp.com/service/forums/questionanswer.do?threadId=1197626

Try /usr/sap/RSP/SYS/exe/run/memlimits | more

Have your sap team explain to you how much memory that are requesting (things like
em/max_size_MB em/initial_size_MB ...)

Eric>> The solution here is to restart the server because your Oracle instance wasn't

How can you tell? Judging by 'ctime' the Oracle sintance was brough up seconds before SAP was started, which is normal no?

Dennis>> You have a bunch of large segments:

Well, yeah, which seems normal for an (partial) start no?

Hein.

[Bart Paulusse · Accepted solution]

Ashis,

log on as sidadm user:
issue "stopsap r3" command
verify that both sap and oracle have stopped.
If there are any sap processes still lingering, "kill" them (and if that doesn't help "kill -HUP or kill -9"

logon as orasid user:
"shutdown immediate" oracle from sqlplus, and check oracle alert log.

log on as sidadm user again:
then "cleanipc " command
if you have any entries returned, remove them with "cleanipc remove"

Now you've removed any sap processes hogging memory.
Have you (or the application team) changed the sap buffersettings recently? You can verify that by looking at the timestamp (or contents) of profile files in directory /sapmnt//profile
If so look for large buffersettings. The message you get means sap tries to reserve more memory for the abap buffer than there is available.

Or have there been changes in the oracle memory settings? Verify that by checking the init.ora or spfile.ora in the /oracle//_64/dbs directory.
If oracle hogs all memory, then sap will never be able to start...

I don't like blame games, but it wouldn't be the first time the "offender" is the first one to start pointing fingers...

regards and good luck,

Bart

[Bart Paulusse]

Ashish,

another question, why don't you use brbackup with the offline_force option? This does an oracle offline backup without stopping sap.
This keeps your sap buffers intact.

regards,

Bart

[AL_3001]

I will put forward your inputs to the SAP team and update you all...

[AL_3001]

Hi Friends,

Attaching o/p of Memlimits as mentioned by Hein van den Heuvel....

I see a few failures at the end of the logs...

Kindly advise....

[Hein van den Heuvel]

From output....

>> Maximum protectable size (mprotect)..: 176 MB

So that would explain the reported:

>> 205748 kilo bytes of shared memory for PXA not available.

Now as to WHY?

Back to an earlier question:
Are you using 32-bit products?
What HPUX version?

You may be limited to 1.75 GB total shmmax
Give too much to Oracle, and none is left fro SAP.

Or it could possibly be the system tuning setting shmmax itself.

Check out the 'readme' in ftp://hprc.external.hp.com/sysadmin/programs/shminfo/

For yet an other tool check: Dave Olker's reply in:
http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1048211

Also:
http://unix.derkeiler.com/Mailing-Lists/HP-UX-Admin/2005-05/0020.html

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[Eric Antunes]

Hi Hein,

I know it because after Ashish said SAP script couldn't start the instance and posted the "ipcs -ma" output, I saw this line:

m 519 0x91246e28 --rw-rw---- orarsp dba orarsp dba 54 1064181760 8514 23662 6:09:52 6:10:00
2:23:07

This is the Oracle instance still in memory, so it wasn't shut down properly and, thus, Oracle RDBMS won't be able to start it.

About the serve restart, in the 2 ou 3 similar situations I had, I found it to be faster than trying to remove those active segments from memory...

Best Regards,

Eric Antunes

[Hein van den Heuvel]

Eric, I saw the orcacle segment CTIME being 2:23:07, followed by some sapsys owned segments with CTIME 2:23:21.
So I read that as Oracle having been started as a step in the failing sapstart, not as a leftover from a previous run.


Ashish, Has that SAP instance been down for all this time? I kinda hate to suggest this, but you might as well hit 'the big red button'... try a reboot?
"re-boot macht immer gut"

Hein.

[AL_3001]

Hi Hein Van Den,

It is a 64 bit application and HP-UX version is 11.00.


Cheers,
Ashish

[AL_3001]

Hello,

Some more information:

Recently the application team had shutdown the application as unix had to carry out disk replacement activity.

When they tried to start SAP/Oracle through script it started in unstable state with PXA errors.

They tried to start sap/oracle again but the same PXA errors occured.

Tried to start sap/oracle through standard SAO commands, SAP started with the same pxa errors. The cleanipc command failed to resolve the issue (as told to us)

Application team asked us to reboot the server, and than the application team started the sap/oracle through script and the system started fine.

Is it that the memory is not getting released when the application is stopped?

How to overcome this issue, so as to the memory is deallocated.

Thanks.

Regards,
Ashish

[Bart Paulusse]

Hi Ashis,

if Oracle and SAP startup fine after a system reboot, but not after a stop of Oracle and SAP, then it looks as if there is a problem with the stop script.
Does the script wait for the stop commands to complete?
Does the start script wait for the stopscript to complete?

could you perhaps post both scripts?

[Eric Antunes]

Hi Ashish,

" Is it that the memory is not getting released when the application is stopped? "

Yes, and the active shared memory segment is not deactivated because the shutdown command wasn't successful (script's return code not equal to 0). That can happen for several reason that you or your DBA must investigate.

Best Regards,

Eric Antunes

[Yogeeraj_1]

hi,

Allow me to also add that the database alert.log can also shed some lights on the exact problem. (in case it is related to the database itself)

hope this helps too!
kind regards
yogeeraj

[AL_3001]

Friends,

We will be carrying out the activity to stop/start sap/oracle this weekend to find the root cause without the need to reboot the server.

I am quite confident that the shared memory is not getting released when the DB is stopped. The reason behind this is, when we tried to start it, we got PXA shared memory error, but when the server was rebooted and oracle/sap was started, it started normally without any error. This tells me, the problem is wih the memory not releasing.

During the activity, the basis team will be stopping the application. Once it is done, we (unix team) will check the shared memory (ipcs -m) to see if the shared memory is released or not. If it has not, we will remove it using ipcrm -m and than ask basis team to start the application.

My question is, if it is problem with releasing of shared memory, how can we overcome it as it will not be possible to manually release the memory everytime.

Thank You.

Regards,
Ashish

[Dennis Handly]

>will check the shared memory (ipcs -m) to see if the shared memory is released or not.

There are 3 states for shared memory:
1) memory being used, NATTACH != 0
2) memory not used, NATTACH == 0
3) memory segment removed.

If you have NATTACH == 0, you might want to know when the last use was. You can use my program attached to this thread to give you the date:
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1136341

>if it is problem with releasing of shared memory, how can we overcome it

Well, you just don't use kill -9. You must shut things down normally.
If you are doing it correctly, this is an application bug that must be fixed.

[Yogeeraj_1]

hi again,

> We will be carrying out the activity to stop/start sap/oracle this weekend to find the root cause without the need to reboot the server.

I would also suggest that you review your startup/shutdown script for both SAP and Oracle. It seems that the flaw lies there. As Dennis mentioned above, it may be that the application/database software are not being properly shutdown.

just some thoughts....

kind regards
yogeeraj

[YAQUB_1]

Hi Ashish,

Can U send me Ur system "kmtune" O/P, I will cross check Ur kernel value. If needed I will send U new value which one U change and why?

Thanks--Yaqub
HP Support!!!

[AL_3001]

Thanks.