Experts...Can Use Your Tips On Tuning Large Memory Server

[Rita C Workman]

Tuning is something I enjoy doing. I've worked on these kinds of boxes plenty of times, but generally with less memory.

This particular box is an rx8640 running 11.31 Itanium. It has 95.95Gb of memory, plus swap:
lvol2 = 10Gb, plus 6 more 8Gb disks for additional swap. Lots of CPU.

Running Oracle apps.

So, are there any caveats or tips anyone could share for tuning so much power. I don't want to increase a parm, just because I have so much, and create a waste of resources.
I want to be sure that this much memory, doesn't call for different methodology in tuning.

Many thanks,
Rita

[Zygmunt Krawczyk · Accepted solution]

You can consider tune the memory for Cell Local Memory (CLM). Read the docs:

The Oracle Database on HP Integrity servers
http://h20195.www2.hp.com/v2/GetPDF.aspx/4AA2-0547ENW.pdf

Running Oracle Database 10g or 11g on an HP-UX ccNUMA-based server
http://h20195.www2.hp.com/V2/GetPDF.aspx/4AA2-4194ENW.pdf

[Zinky]

Is it running as a single nPar?
By "Oracle Apps" - what do you mean?
- Database?
- Fusion?
Have you any idea of what the overall memory requirements are for your "Apps"?

Do you see any "surplus" on memory?

[Torsten.]

I think you need to tell some more details to get some good hints ...




Regarding "...so much power", what does "machinfo" show?


Examples:

# machinfo
CPU info:
8 s (1.6 GHz, 20 MB)
4794 MT/s bus, CPU version 4
64 logical processors (4 per socket)

Memory: 981492 MB (958.5 GB)

Firmware info:
Firmware revision: 002.074.000
FP SWA driver revision: 1.18
IPMI is supported on this system.
BMC firmware revision: 1.02

Platform info:
Model: "ia64 hp Superdome2 8s"
...

and


CPU info:
64 Intel(R) Itanium 2 9000 series processors (1.6 GHz, 18 MB)
533 MT/s bus, CPU version C2
128 logical processors (2 per socket)

Memory: 981492 MB (958.5 GB)

Firmware info:
Firmware revision: 7.44
FP SWA driver revision: 1.18
IPMI is supported on this system.
BMC firmware revision: 24.04

Platform info:
Model: "ia64 hp superdome server SD64B"
...

[Innate Ideas]

Kernel parameter base_pagesize = 16 will reduce the CPU overhead for managing all that memory.

We've had good experiences setting base_pagesize = 64 on our HPVM hosts (not HPVM guests) with 96GB of memory. base_pagesize = 16 seems to be optimal for our Oracle application and database environments.

As always your milage may vary.

[Zinky]

Rita,

FWIW - if it is going to be a single nPar, single Instance, Lots of Surplus mem kind of environment -- and IF the env needs superfast I/O for some routine or processing -- I'd allocate a part of Memory as "RAMDISK" -- to me that is the BEST use of memory that would otherwise just be "wasted" in cache.

Exmples of RAM DISK use would be sort areas and TEMP TABLESPACES...

[Rita C Workman]

Sorry folks, been out of town for a couple days, but was able to do some reading on this while on the plane(s)..

Here is the machinfo on the box requested, and the box is I believe running or going to be running Oracle 11gR2.

CPU info:
7 Intel(R) Itanium 2 9000 series processors (1.6 GHz, 12 MB)
533 MT/s bus, CPU version C2
12 logical processors

Memory: 98254 MB (95.95 GB)

Firmware info:
Firmware revision: 9.022
FP SWA driver revision: 1.18
IPMI is supported on this system.
BMC firmware revision: 3.01

Platform info:
Model: "ia64 hp server rx8640"
Machine ID number: n/a
Machine serial number: n/a

OS info:
Nodename: no-name-server
Release: HP-UX B.11.31
Version: U (unlimited-user license)
Machine: ia64
ID Number: 0769464687
vmunix _release_version:
@(#) $Revision: vmunix: B.11.31_LR FLAVOR=perf

Now...what I don't know yet is if the DBA's plan to, or even know about, turn on for Oracle Optimizing using ccNUMA or NUMA.
I'm waiting to hear about that.

Zygmunt's docs were very helpful. But now I'm looking for a white paper called "Dynamic server resource allocation with Oracle Database 10g or 11g on an HP-UX ccNUMA-based server" - and so far I can't find it under the new docs webpage. So if anyone knows where to find this...let me know.

Points will be generously given later,

Many thanks!
Rita

[Zinky]

Oh so a Database. Purely DB hosting?

AFAIK, 10GR2 and 11G are both NUMA aware and have lots of tools to auto-this or auto-that BUT nothing beats a superfast I/O for parts of Oracle that NEED superfast I/O. Your DBA should be able to tell you which -- but TEMP Tablespace and other sort areas or other transient TableSpaces that require super high speed mosdt certinly would be very good use of that excess memory -- that is IF you forecast your 96GB of RAM will have some surplus.

[Rita C Workman]

Thanks Alzhy,

What's I'm seeing is that the server itself is not set up for NUMA (i.e. policy & LORA_MODE show default).
I'm "hoping" to get some info from DBA's or mgmt to clarify what the DBA's plans are, because if they 'think' that all they have to do is flip the switch on their side and it's a done deal, then everyone is in for a surprise.
I know if they turn on Oracle Optimization that I will need to change the settings to ensure it's mostly CLM and the rest ILM. BUT, what else do I need to do to ensure performance- what other parms might I need to tweek.
Cause once they turn it on, you can not easily change parms with CLM - not even dynamic ones with things up and running.

ILM tuning is one thing, but CLM is something I'm such a newbie on.

And I'm having trouble trying to find the white papers I want to review....wish HP would have left well enough alone. Now it's harder than ever to find things.

Thanks,
Rita

[James R. Ferguson]

Hi RIta:

I believe that this is the whitepaper you want:

http://h20195.www2.hp.com/v2/GetPDF.aspx/4AA3-1350ENW.pdf

Regards!

...JRF..

[Rita C Workman]

Thanks Jim! Got it.

Now I have a bit of info on NUMA, and I know how to turn it on. Unfortunately, no where in anything do I recall it give any insights as to what to set some static parms, like filecache_max/min; maxdsiz; maxssiz; shmmax for few. In fact, it doesn't give any parm recommendations that I can recall from my reading the last few days. But it does say that you either go with dynamic configuration or NUMA optimization, but not both.

So...my question would now say:

If they don't go with grid technology and NUMA and they opt to stay with ILM (Interleaved memory).....are there any tips for parm setting you folks might recommend I watch out for with this box?

Thanks,
Rita

[Steven E. Protter]

Shalom Rita,

Back online after a long break.

filecache_max/min;
set these two as you would on older HP-UX. Note that Oracle does sometimes benefit from file cache so you might find increase does help.

Oracle actually dictates some of this in the installer, and violating those rules is done at the peril of the sysadmin. shmmax needs to be equal to physical memory even though it can not possibly be that large.
maxdsiz; maxssiz; shmmax

Regards,

SEP

[Zinky]

Since it appears you will not be partitioning your rx8640 - there's less of an issue sticking to the "rules" on CLM/ILM, processor/cell localisation,etc -- as set forth in the HP and Other Whitepapers.

I *would* focus "tuning" and turbo-charging your Oracle DB Server environment on two things:

1) Processor binding to certain Oracle Processes
2) FastIO! (aka RAM Disk) for key

I would also advice your DBAs to go with ASM Storage -- preferably 2 or 3 ASM Diskgroups instead of using traditional RAW or CookedIO -- no matter what the overhead projector says on improvements with VxFS/DirectIO.

[Rita C Workman]

Thank you all for your responses. Going to hold the thread open for a day or two longer and then will give points.

Rita

[Zinky]

Rita Ma'm,

Pls. also be aware of:

http://skrajend.blogspot.com/2008/09/numa-after-10204-upgrade-is.html

In case you end up your DB is not using all available CPUs.... That is IF you're on 10Gr2

[Zygmunt Krawczyk]

Alzhy wrote:
"I would also advice your DBAs to go with ASM Storage -- preferably 2 or 3 ASM Diskgroups instead of using traditional RAW or CookedIO -- no matter what the overhead projector says on improvements with VxFS/DirectIO."

From performance point of view the ASM is good, but not the only solution. Comparable performance can be achieved on filesystem with Oracle Disk Manager (ODM).

Read the doc:
"Improving the performance of single instance Oracle on file systems: Analyzing the impact of Oracle Disk Manager (ODM) â a feature of HP Serviceguard Storage Management Suite"
http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c01917294/c01917294.pdf

[Zinky]

Zyg...
I won't disagree with ya but it is really what the DBA and SysAds are comfy about and which offers the best manageability and less risks. WIth ASM, there is no filesystem/mount point to corrupt, no "volume manager layer" to deal with. Sure with ASM a DBA has to maintain a small DB Instance -- +ASM that acts as manager of provisioned disks - but compared to traditional storage solutions - this is a whole lot simpler and moves the responsibility of manaing and monitoring DB storage to our DBA Amigos...


Say -- is ODM an evolution from Veritas Database Edition sir? Does it use Veritas bits (i.e,. FIlesystem?)

We've used Veritas DBED in the past but moved most of our DBs to ASM on HPUX (and ASMLib flavoured ASM on Linux) -- practically zilch issues compared to when we were on sashimi (raw) and sushi (cooked filesystems with DirectIO).

Cheers!

[Rita C Workman]

Alzhy,

Thanks for the blog pointer. It mentions his problem with only seeing 3 CPU, but it doesn't go into detail though.
For example:
On HPUX NUMA is turned off by default, and he does not mention if HPUX was properly configured for NUMA.

On the Oracle side NUMA, depending on version is controlled through an initialization parmater and only checked at startup time! On 10gR2 and 11gR1 I think it's called "_enable_NUMA_optimization". I don't think even Oracle likes this parm. Now on the 11gR2 there is a new one "_enable_NUMA_support" that Oracle supports.
So on startup, even though Oracle may have the right startup/enable NUMA parm set as "on" for the 10g and later versions - with a caveat that folks miss. Oracle NUMA will first go out and check if the server is set up for NUMA. If it is, then it proceeds with it's next steps to confirm allowing optimization. If the HPUX is NOT set (i.e. LORA_MODE=0), then Oracle NUMA is disregarded and it's optimizations will be disabled.

There's a bunch of other parms and settings that Oracle must give attention to, that clearly in his blog they obviously didn't...but what all of this has shown is that doing a ccNUMA or NUMA configuration requires DBA's & SysAdmin's to work together.
Introducing a whole new concept b/t DBA's and SysAdmins, cause Grid Technology is not for the timid.

Regards,
Rita

[Rita C Workman]

Almost forgot...on the storage thing (ASM Storage), well each to his own on this.

For myself, I will try to be open to the notion of trying/testing Oracle's disk management utility....but at this point in time I can not say that I am in anyway moved to use it in any production environment.

FYI...I was actually a DBA for a very short time, and the one thing it taught me was that I knew nothing about Infrastructure. I say leave HPUX and Storage to the professionals...and unless the DBA is a certified HPUX and certfied Storage person-they shouldn't play with what they do not know.

Just my thoughts,
Rita

[Rita C Workman]

Very much appreciated the pointers to the info on NUMA.

Was hoping to get more parm advice in the event they go with ILM and not NUMA. Although according to one document, Oracle actually has no way of knowing if the box is set up to run with NUMA or not. Now, isn't that interesting!

Guess for tuning parms, if we go ILM (native) I'll have to go by the standards I'm used to and 'tweek' from there.

Kindest regards to all and Many Thanks!
Rita

This thread is now closed.