Cell Based Servers ccNuma and oracle

[Smucker]

After attending the tech forum and learning about ccNUMA and oracle on cell based servers we are concerned. The more I read the more concerned I am that we should follow HP's recommendation and oracle's new patch suggestion and to turn NUMA off for oracle.

Two questions keep nagging me and I have not been able to get a firm answer.

1). How crucial is it to have all memory and CPU spread evenly across cells in an nPar. What kind of performance hit/ risks are we talking if it is not balanced?

2). How many people have turned NUMA off in oracle. We found an oracle document that states NUMA is not supported in a vPar ... anyone have an opinion...

3). Are we overreacting to this NUMA thing or should be digging into this more?

[likid0]

Here you have a link to an HP webcast, I think you are going to find usefull:

Locality-optimized resource alignment (LORA) for NUMA servers:
http://h20341.www2.hp.com/hpux11i/us/en/solutions/kod-locality.html

[Smucker]

Yes, I have read about the the 11.31 options and LORA ... But I should have mentioned that we are running 11.23 and 11.11.

We are not in a position to run 11.31 at this moment and cannot migrate.

[Duncan Edmonstone]

What makes you think you have NUMA optimizationa enabled?? Oracle would only ever enable NUMA if you have more than one locality domain in your OS image - you can check this out by using "mpsched -s". If you see only 1 locality domain, then Oracle NUMA optimizations would be turned OFF. (and see below for details on a patch that changes this behaviour)

I think the question you have to ask yourself is "how much more performance do I need to squeeze out of my system?" If you are comfortable with what you have, don't use it - if you really need to get that extra degree of performance, then take adavtange of it AFTER YOU HAVE TESTED IT.

For defintive opinion on ccNUMA, I would look at Oracle Metalink note 759565.1 - this also describes a patch which changes the behaviour I described above so that Oracle does not enable ccNUMA even when it detects a ccNUMA system.

For the HP definitive word on this, see the following technical paper:

http://h20195.www2.hp.com/V2/GetPDF.aspx/4AA2-4194ENW.pdf

You also mention 11.11 - that version of the OS is not ccNUMA capable at all, so you won't have any ccNUMA optimisations on 11.11 partitions. To be honest I wouldn't try and use any in 11iv2 either... leave it until you get to 11iv3.

Can you give a reference for the Oracle document that suggests no Oracle NUMA support in a vPar?

HTH

Duncan

[Smucker]

The servers I was concerned about are cell based server (nPars) consisting of more than one Cell ....

/root>mpsched -s
System Configuration
=====================

Locality Domain Count: 4
Processor Count : 24

Domain Processors
------ ----------
0 0 1 2 4 6 7
1 8 9 10 12 14 15
2 16 17 18 20 22 23
3 24 25 26 28 30 31

I also know tht oracle sets the NUMA optimizations on by default....

I am inquiring to see if we should make this a high priority to disable NUMA in oracle on these databases...

I am also deciding how crucial it is to have the the memory and CPU's evenly spread accross the cells. what is your opinion.

I have read and understand the oracle patch but it only covers 10g + not 9i or 8i

[Duncan Edmonstone]

>> I have read and understand the oracle patch but it only covers 10g + not 9i or 8i

That's because only 10g has NUMA optimizations - there are none in 9i or 8i, so if thats what your databases are, you have nothing to concern yourslef about.

HTH

Duncan

[Steven E. Protter]

Shalom,

1) Even distribution works if there are equal work loads. if one of the oracle instances needs to run with more processing power then uneven distribution of cpu is called for. It depends.

2) No opinion here, other than Oracle testing on HP-UX is not thorough and make decisions based on what people with experience are doing.

3) No you are not over-reacting. If you are using NUMA and not experiencing problems, then I would be very careful about changing things. This post is exactly what you should be doing. You are looking around and trying to make an informed decision.

SEP

[Duncan Edmonstone]

>> I am inquiring to see if we should make this a high priority to disable NUMA in oracle on these databases...

If you systems have been OK up til now, then leave them as they are...

>> I am also deciding how crucial it is to have the the memory and CPU's evenly spread accross the cells. what is your opinion.

For what degree of crucial? Honestly you aren't going to see HUGE performance differences unless you have a SERIOUSLY imbalanced system (like a cell board with 1 CPU but most of the memory and another with most of the CPUs and hardly any memory). As long as you've got things "as balanced as you can", you're unlikely to have a major problem. Ideally have the same memory and same CPU count on each cell board, but don't sweat it too much if there are some minor differences...

There are plenty of other places to address performance before gettin g worked up about this stuff...

HTH

Duncan

[Smucker]

This may not be the case there are several sources that claim that these are hidden option within oracle 8i and 9i

Oracle Essentials: Oracle9i, Oracle8i & Oracle8, 2nd Edition from safari books

I think it may be as old as cell based servers themselves. However, it may not work the same under 11.11

[Duncan Edmonstone]

SEP,

>> 1) Even distribution works if there are equal work loads. if one of the oracle instances needs to run with more processing power then uneven distribution of cpu is called for. It depends.


I'm not sure what you mean there... within a nPar, you ideally have equal memory and equal CPU on each cell board - I can't think of a situation where this wouldn't be "the ideal".

HTH

Duncan