Alleged swap space shortage of SAP

[Ralph Grothe]

Hello,

our SAP admins came to me complaining that some of their jobs abort with inexplicable malloc errors.
Unfortunately the printouts of their SAP admin GUI I was supplied with by them are all in German (so probably of little value citing here).
But one error string that shows up read "no memory of class perm".
After they fed their SAP knowledge base with the phrase it spit out the following assumed cause of error (I was given the German text but found by googling the obviously English original under this URL (sorry, that it's not clickable but I checked "Retain format" when posting this):

http://help.sap.com/saphelp_erp2004/helpdata/en/fb/cbb96cd9ee11d188be0000e83539c3/content.htm

In it the culprit is claimed to be insufficient swap space and as remedy either reduction of SAP memory footprint for the job or extension of system swap space is named.

I don't believe this because one of the first things I checked was the swapinfo,
and it says that since last swapon (viz. reboot which was on 5 Sep. when I applied the Support Plus June 2006 GOLD* patch bundle) none of the swap space has ever been used for swapouts.

# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 16 4080 0% 0 - 1 /dev/vg00/lvol2
dev 16000 16 15984 0% 0 - 1 /dev/vg00/lvol12
reserve - 20064 -20064
memory 22323 10800 11523 48%
total 42419 30896 11523 73% - 0 -

What the SAP admins see from their GUI is the total pseudo swap of abt. 42 GB
(the box has abt. 28 GB RAM so that those 3/4 RAM make the added 22 GB pseudo swap confusingly named as "memory")

I asked them to rerun their aborting job while I was collecting glance gbl_mem_util, gbl_mem_swapout_rate and gbl_mem_pageout_rate.
During the test gbl_mem_util was below 45% and the latter two rates always stayed at 0.0

Because I applied the GOLD* patch bundle during last maintenance downtime I suspected some of the contained patches to maybe have introduced some SAP idiosyncracy?
At least these seem to be swap related,
but I haven't read their READMEs yet, for waht they patched or any side effects

# swlist -l fileset GOLD\*|grep PH|grep -i swap
# GOLDBASE11i.PHKL_24026 1.0 JFS Filesystem swap corruption
# GOLDBASE11i.PHCO_27010 1.0 swapon(1M) patch
# GOLDBASE11i.PHCO_27007 1.0 swapinfo(1M) cumulative patch


Has any of you who also runs SAP systems on their HP-UX 11.11 boxes experienced this,
and knows a remedy?

Rgds
Ralph

[Ralph Grothe]

Maybe it also helps if you know
what the swap related tunables are set to

# kmtune|grep sw
allocate_fs_swapmap 0 - 0
dmp_swapdev_is_vol 0 - 0
maxswapchunks 11000 - 11000
nswapdev 10 - 10
nswapfs 10 - 10
remote_nfs_swap 0 - 0
swapmem_on 1 - 1
swchunk 2048 - 2048
vol_swapdev_is_vol 0 - 0


I must admid that giving both swap devices the same priority was pretty daft since both lvols use extents on the same disk.
This was a misconfig corpse I had forgotten about when I applied the patch bundle.
But this should theoratically merely degrade swapping/paging performance but not render it useless.

[Ralph Grothe]

Maybe it also helps if you know
what the swap related tunables are set to

# kmtune|grep sw
allocate_fs_swapmap 0 - 0
dmp_swapdev_is_vol 0 - 0
maxswapchunks 11000 - 11000
nswapdev 10 - 10
nswapfs 10 - 10
remote_nfs_swap 0 - 0
swapmem_on 1 - 1
swchunk 2048 - 2048
vol_swapdev_is_vol 0 - 0


I must admid that giving both swap devices the same priority was pretty daft since both lvols use extents on the same disk.
This was a misconfig corpse I had forgotten about when I applied the patch bundle.
But this should theoretically merely degrade swapping/paging performance but not render it useless.

[DCE]

Ralph,

I seem to remember from a SAP installation I did several years ago, that SAP would not recognize psuedo swap. I actually had to configure 2x memory physical swap for the install to work.

Don't know if this helps,
Dave

[Volker Borowski]

Hi Ralph,

what type of DUMP headlines did they get in transaction ST22 ?

Run it again and continiously look at the lower part of transaction ST02 to monitor growth of allocation of HEAP and extended memory.

This way you'll find out at least, if they are bouncing against a buffer-limit (100%) or really strikes physical limits.

It is normal that HEAP is filling up to 100% and the process then starts to eat extended memory.

Check in addition, in transaction SM50, if the process in charges is running private after a while (PRIV-mode).

Some older ABAP Statement still make use of SAPs own SWAP and ROLL-files. So may be you are hitting a border on their parameters rdisp/PG_MAXFS & rdisp/ROLL_MAXFS or the filesystem where these files are located are full ??
If the ABAP Statement you are hitting is i.E. EXTRACT, it is likely, one of these is your problem.

Good hunting
Volker

[Ralph Grothe]

Hi Dave,

it wasn't swap space requirements.
The SAP installation requirements demand
20 GB of swap space (that was what I got from the SAP admins when I prepared/adapted the system for them back then).
As you can see I added a second 16 GB swap device (albeit with the silly high priority of 1)
because when the system was installed the default 4 GB first swap device on lvol2 was automatically taken (at that time nobody knew that the box would host SAP later).
Because I loath such a waste of disk space (although disk storage has become much cheaper) for every new installation where I know in advance that it will host SAP I make lvol2 10 GB wide and configure the remaining 10 GB as very low priority filesystem swap directed at /var/adm/crash/paging with a maximum limit for 10 GB pages in total.
Thus, one can at least use non allocated swap space partly as filesystem space.

Today I was informed by our SAP admin that a solution for the problem was found after contacting SAP support.
As it turned out the error message was misleading because it wasn't lack of swap space but rather that a too large file or data structure was attempted to read in one go (something I know from Perl as the so called "slurp mode" by localizing $/, and which is considered as extremely wasteful memory management and thus deprecated for large files).
So I take that SAP have told them which switch to toggle in their admin suite to prevent this in the future.

Hi Volker,

I have to confess that what you suggest sound s all Greek to me because I lack SAP training (even haven't got access to SAP admin tools or a test installation to play with).
Therefore, I will have to forward your suggestions to our SAP admin,
and I hope she can make something out of it
(as a self-respecting SAP admin she should, shouldn't she?).