Interesting discussion. You want your system to be busy, providing optimal use of an expensive resource. But, at some point, it is too busy, and other expensive resources (staff, customer loyalty, competitive edge) is jeapordized.
If your SAP system is like others I've seen (any computing environment, for that matter), useage is a very peaky thing. I like to see some headroom during the average times, to allow for the "1PM syndrom", where everybody comes back from lunch at once, and all indicators go to 100% for an hour.
If you are never paging, all day, ever, it is a good indicator that you have sufficient headroom on memory size. Paging provides an interesting performance curve. Performance stays pretty much constant/acceptable until you start paging. At that point, it can suddenly drop dramatically for everyone. If you avoid paging, or just see a tiny bit at the worst peaks, memory is not your issue.
Having busy CPUs, processor usage around 90% as you mention, tells me that if you watch it very long, you will see peaks hitting 100% from time to time (and drops to 60% from time to time). Again, the key is how long you are at 100%, and how often. If your CPU utilization rarely maxes out, and only for short times, and only during the worst times (8AM, 1PM, or unspecified other for your particular work environment), then that is probably on the healthy side, and you are getting your money's worth out of that system.
I hate to see it that close, though, because the slightest growth, and all of a sudden you are at 100% lots of the time, and very frequently. Like the paging, this generally means a sudden dramatic impact to users, which directly correlates to phone calls and emails and boss visits that we'd all prefer to avoid.
Then there is disk I/O. The good news for you is that it is very hard to have all three elements of the system under equal stress, CPU, memory, and I/O. If you have high CPU usage, you are probably not I/O bound. Or, you won't know that you are until you add a CPU, and see your CPU usage drop to 55% average, but see no apparent performance increase.
One bottleneck generally hides others, because they can't be seen until the most prominent one is removed. That's why I generally recommend to add memory when adding a CPU. "Why, we don't have memory issues?" Well, not that you can see, anyway. Time and again, I've helped customers with CPU upgrades, only to help them with a (generally unbudgeted) memory upgrade a few weeks or months later, since the CPU addition didn't do much for them except reveal the "other" bottleneck.
Sorry this is so general, but the question has general applicability. If I were you, looking at 90% CPU utilization (presuming this is average, not peak), I'd be uncomfortable with that, and would be pinging management for more (or faster) CPUs in the upcoming budget cycle.
Lacking particulars on the memory situation (although you look to be OK right now), I'd look at adding memory along with the processor(s), to keep from hitting the snag that experience tells me you may be ready to hit.
When it comes to memory, there are some interesting gotchas, depending on what type of server you have, how the memory is distributed across the memory buses, and how well it is interleaved. Lots of memory boards are better than few, and DIMM counts should be in powers of two (like 4 or 8 or 16, never 3 or 7). The HP 9000 config guides call for adding memory in large chunks for best performance, which irks a lot of people. It seems very self-serving on HP's part, to sell lots of memory.
That turns out not to be the case, there really are valid, technical reasons for the guidelines. Unfortunately, the magic of memory interleaving only comes into play for applications that do lots of sequential memory access. Of the scores of processes running in your system, some do this much of the time, and some are too random for interleaving to be a factor, and some do it well some of the time, but change their usage pattern under heavy load or other circumstances. So, how much will a well designed, properly interleaved memory scheme help with performance? Nobody can tell, without exhaustive testing of YOUR app, in YOUR environment, with various memory configurations being stressed by YOUR users.
It's enough to make a system architect weep. But real geeks don't cry...
If you post back with your memory configuration, and your server class & some other details (CPU counts, system board counts in RP7410/8410, I/O card counts and types), I'll try to post some more concrete suggestions. Plus, others may point out things I haven't touched on yet (LVM striping for RAID array data, alternating PV Links, etc).
Best Regards,
--bmr
We must indeed all hang together, or, most assuredly, we shall all hang separately. (Benjamin Franklin)
