Lots of semaphore activity on a SAP app server

[Olivier Masse]

Hi,

The Basis are complaining that some of their SAP transactions are slow on an app server. I checked and there they have configured a lot of "worker" dw.sapSID processes, I'd say a ratio of 7-8 for each CPU.

OVPM and glance reports that most processes are stopped because they are blocked on a semaphore.

I *think* that there might be too much work processes trying to access the same resources, thus the high semaphore usage, and overall this wastes CPU time.

Am I on the right track with this assumption? Could having less workers be more efficient?

Thanks for your input. I'll come back and give points. :)

Olivier

[Mel Burslan]

I am not a basis admin but as far as my understanding of SAP landscape goes, app servers can be run in parallel and if you are having problems with performance, you scale your landscape by addind one or more appservers into the landscape. And yes, after some bending point in the users vs. work done curve, gets past, your performance will tremendously suffer due to the fact of racing for the resources.

Again as I said, I am not a basis admin and take my words with a grain of salt.

[Olivier Masse]

According to the basis, there's enough CPU. They notice an abnormal slowdown during a specific transaction.

[Mel Burslan]

Check your disk activity during the slowdown. Your transactions may be waiting for a disk write at the same physical device. A common source of contention and slowness if the data is not evenly distributed over the physical devices.

Also, have an SAP consultant take a look at the activity and this transaction slowing you down. They may have a solution to this symptom and your organization may not be only one suffering from it.

[Olivier Masse]

Thanks for your advice. There is no contention on the disks, and no VM activity either. Only CPU that goes way up. I asked The SAP consultant to look in OSS about this.

The only thing that goes in my mind is the hight number of processes but this might be far from the bull's eye.

[Alan Meyer_4]

If the CPU is going up but not disk activity, then they could be hitting the app server buffers extra hard. What do the buffer stats look like?

[Mel Burslan]

Well, very high number of proceesses determination may not have hit the target at the bull's eye but you are not grossly off-target either in my humble opinion.

As you know, every process launched/forked, grabs a chunk of resources when starting. Semaphores are one of these resources. And their global availability is capped by kernerl parameters starting with "sem" letters. I am no system programer and I do not exactly know what each of these parameters do exactly, but if you can run this command and let us know the output along with your system model, physical memory, swap and OS version, maybe people can compare it to their instances and have some suggestions.

kmtune | grem -i ^sem

[Mel Burslan]

Today I can NOT type...

command should have read grep NOT "grem"... sorry...

sleep deprivation is not good for me :) must cut down coffee consumption...

[Olivier Masse]

Thanks for the help, guys.

Alan: I don't think the buffer cache is a bottleneck, as glance shows that the most of the CPU is used by user processes and not the system. We also have minimal buffer cache on our app servers, around 1%, as they have lots of RAM and don't do much disk access and we wasted too much using the default values.

Mel: These values are those suggested by SAP. The system tables are not filled with this configuration.

# kmtune | grep -i sem
sema 1 - 1
semaem 16384 - 16384
semmap 4098 - 4098
semmni 4096 - 4096
semmns 4096 - 4096
semmnu 8208 - 8208
semmsl 2048 Y 2048
semume 100 - 100
semvmx 32767 - 32767

[Mel Burslan]

Your kernel parameters are btter than my app servers running on N-4000 5-CPU machines with 10 GB of memory. So, I am stumped as this sound like an application related resource contention. Especially if you are not experiencing a semaphore tables coming close to exhaustion. Your call to SAP OSS is the best way to proceed in my opinion.

[Hein van den Heuvel · Accepted solution]

7-8 Sap disp+work tasks for each CPU sound like way too much. This also suggests a great many per sap instance.

I run SAP benchmarks for a living, so I admittedly have a distored view of the world.
In benchmarks we get the best results to close we can get to 1 worker/cpu. Of course that is not realistic in real life, but a low number of workers should be the goal.

SAP has a nice queuing and dispatch mechanism built in. Use it! Let SAP only schedule a new worker of there is a chance for progress for the task. Otherwise you are just creating more clutter, more contention notably on semaphores.

Also, each active task within an instance will require semaphores (message, paging, presentation buffer, extended memory) and if you have too many (10+? 20+? 30+?) they will get in each others way. If you can manage that, you could/shoudl consider multiple instances. If you were to check out details for recently published SAP benchmark then you will see some vendors going extreme using an instance per cpu core!.

For further investigation / understanding it is of course critical to monitor the SAP semafor usage for example through SAP tools:
ST02 (tune summary)
--> Detail analysis
[additional functions]
--> Semaphores

If this is the first time going there, then be sure to read the legend carefully:
first line = "Wait time entering the critical path"
second = "Residence period in the critical path"

Let us know if you find something interesting.

hth,
Hein.

[Olivier Masse]

Thanks Hein, I'll forward your message to the Basis so that they can look it out. Looks like they configured too much workers indeed.