CPU's are pegged on RP8410

[Kishore Gowda]

We just upgraded the Oracle Financials server from V-2500 (13 way 440Mhz with 14Gb Memory) to RP8410 with two partitions. The production partition is 6 way 875Mhz with 12Gb Memory. The second partition is 2 way 875Mhz with 8Gb Memory, and is being used for ad-hoc reports pointing BCV disks on Symmetrix. Since the migration, the nightly reports (Oracle Financial FSG's) that used to run in 6-8 hours are taking twice as long on RP8410 production domain (6 CPU partition). The CPU's are pegged during the enitre report processing period!

While sizing the server, we were told by HP & HP reseller that 6-way 875Mhz partition has 81,000 TPM compared to 40,000 TPM for 13-way 440Mhz V-2500. Seeing these TPM numbers, we thought 6 CPU's were adequate to run production application. But since the migartion, the results seem to be mixed. Some Oralce Financials reports (small reports) are running lot faster than V2500 system, but large reports seem to be running longer than before! I am not sure if I have tuned kernel parameters appropriately for RP8410. Any thoughts on CPU bottlenecks or tunable kernel parameters are greatly appreciated.

BTW, the dbc_max_pct is 5% and min is 2%; total memory is 12Gb. I am not sure if this is set too low

Thanks,
Kishore Gowda

[Steven E. Protter · Accepted solution]

I set min to 5% and max to 7%. The fact that not too much movement is allowed is important.

This sounds like a case for some good old fashioned performance analysis.

I'm putting in a link to a doc, and attaching a script for gathering data.

http://www2.itrc.hp.com/service/cki/search.do?category=c0&docType=Security&docType=Patch&docType=EngineerNotes&docType=BugReports&docType=Hardware&docType=ReferenceMaterials&docType=ThirdParty&searchString=UPERFKBAN00000726&search.y=8&search.x=28&mode=id&admit=-1335382922+1081434804537+28353475&searchCrit=allwords

The document is entitled and you need a service contract to access it(i think). Because I think its entitled, i won't past in a copy because thats a violation of my service contract.

SEP

[John Poff]

Hi Kishore,

Having survived a similar upgrade for Oracle Financials from a V2600 to an rp8400, I can tell you that your rp8410 should be much faster that the V2500. I would take a look at your kernel parameters first. Can you post them please? Specifically I would like to see what your 'timeslice' is set to on the rp8410. There used to be a monolithic database kernel tuning template that would set the 'timeslice' value to 1, when it should normally be 10. That would sure pound your CPUs.

Also, when your CPUs are hammered with the large reports, are they spending most of their time in User or Sys?

JP

[Kishore Gowda]

Steve, Please attach the script for gathering info. I will update dbx max & min parameters over the weekend.

John, I have attached the kernel configuration. When CPU's are pegged; over 95% is in %usr mode. Timeslice kernel parameter is set to 10.

Thanks for your valuable time.

[John Poff]

What mount options are you using for your Oracle filesystems, and what mount options were you using on the V2500?

JP

[Hein van den Heuvel]

[Don't shoot the messenger!]

I hate to be a spoilsport, and do encourage you to do all reasonable performance tuning, but please do not set your hopes too high based on system tuning/dbc_max_pct/mount direct or any of the good stuff to have a significant effect on your application based on it running 95% user mode.

Just about all of that is aimed at reducing overhead in SYSTEM code. But even if all system time were to go away, you would just get back 5% which is not the kind of improvement you are hping to achive is it now?

Reducing context switching (timeslice) could help y making the cache more effective. Can you experiemtn with PINning processes to processors? MPSCHED and friends?
Is the time spend in Oracle slaves, or forground oracle apps processes. A nice ps report can highlight that.

If most time is spend in the slaves, the Oracle image itself, then statspack should be your next step. Is time being wasted waiting for latches? re-parsnig SQL that coudl be remembered? Cursor_space_for_time? parsing similar or exact only? enough Oracle pool?

Check your high-cpu-consumption images with chatr. Are they using a large page size?

Is the memory config optimal? Caches working? Best possible interleaving (if that is applicable to your box).... because remember, 100% CPU could mean 80% waiting for memory, 20% actually doing some instruction... If you can change that balance you will make a serious impact!

Good luck!
Hein.

[Chris Vail]

First of all, running the CPU's at 95+% is a good thing, not a bad one. It means that you're getting good, honest work out of them. You may have a well tuned system, but one that is just running at maximum capacity. We'll need more information to help you tune your system.

Don't disregard the Oracle parameters as well. Did you change the SGA when you reduced the amount of memory? If not, you need either to reduce the SGA or increase the memory in the system. Remember that Oracle is the memory hog to end all memory hogs. The more you give it, the more efficiently it will run--but it'll always be a pig.

Is this ONLY a database server, or are you also running the applications on the same host? Are the filesystems laid out the same now as before? Are you using PowerPath? Are you running F/C or straight SCSI? What does Glance or iostat say about disk activity?

Your short reports are running faster probably because the 8410 IS significantly faster than the 2500, and all the data can be read into memory, and dealt with it there. Larger reports need a lot of disk I/O, this is probably whats slowing you down. EMC drives are not known for their blinding speed (unless you paid for it). So there may be some things you can do on that end to speed things up.

Anyway: collect some more data (especially I/O, before and after) and post it here.


Chris