Memory management on Integrity blades (CLM/ILM)

[tibey]

Hi

We are recent owner of a bl870c i2 server. We intend to use it as a single server (no n/v partitions) to run a single instance or oracle database version 10.2.0.4

After having read a lot of HP documentation, it seems that :

- By default, the server is in NUMA mode, which means most memory is in CLM (or SLM) mode

- On oracle 10.2.0.4 numa optimizations are broken and thus must be disabled

- If oracle numa optimization are disabled, it is very recommended to allow at least enough memory for Oracle SGA in ILM

(source : HP whitepaper "Running Oracle Database 10g or 11g on an HP-UX ccNUMA-based server")

So i am trying to allow interleaved memory on my blade. I did :

-  change the kernel parameters numa_policy to 2 and numa_mode to -1 with kctune

- change the memory configuration in UEFI manager and setting memconfig to MostlyUMA (default MostlyNUMA)

But after trying each of them one by one, having done several reboot/resets, it still seems that i don't have ANY ILM, either in checking with machinfo -m :

Memory: 32596 MB (31.83 GB)
0% Interleaved Memory

of with locinfo :

 --- System wide locality info: ---
 index  ldom physid   type  spus     total      free     float      used
     0     0      0 C{S}LM     4    15.98G    12.88G         0     3.10G
     1    -1      2    BLM     0         0         0         0         0
     2     1      3 C{S}LM     4    15.85G    13.94G         0     1.91G
     3    -1      5    BLM     0         0         0         0         0
                                     -----     -----     -----     -----
                                    31.83G    26.82G         0     5.01G

What am i doing wrong ? is it only possible to allocate interleaved memory with n/v parts ?

Thanks in advance for anyone who could help

[S_Logan]

Hi tibey,

>>>>> memory configuration in UEFI manager and setting memconfig to MostlyUMA (default MostlyNUMA)

Memory interleaving options for Integrity server blades:

• Option: Max UMA
  Description: Maximized Uniform Memory Access, 100% ILM
  Comments: Memory is interleaved across all processor modules installed in the system; has the potential to improve bandwidth by distributing memory regions across more DIMMs. When choosing this option, customer needs to consider the longer latencies associated with 1 or 2 QPI hops.

• Option: Mostly UMA
  Description: Mostly Uniform Memory Access, 6/8 ILM and 2/8 SLM
  Comments: 6/8 of the available system memory is interleaved across all processor modules installed in the system and 2/8 is interleaved as local memory.

• Option:Balanced
  Description: Equal allocation of uniform and non-uniform memory access, 4/8 ILM and 4/8 SLM
  Comments: N/A

• Option: Mostly NUMA
  Description: Mostly NUMA, 1/8 ILM and 7/8 SLM
  Comments: Default memory interleaving selection at boot, optimum for HP-UX.

• Option: Mostly NUMA_MBI
  Description: Mostly NUMA, minimum balanced interleaving; 1 GB ILM, and the rest of the memory SLM
  Comments: Optimum for Windows®. Allows for enough shared memory space for the kernel and any registers that need to be accessed by all processor modules while minimizing memory latency by configuring most of the memory space as SLM.

• Option: Max NUMA
  Description: Maximized NUMA, 100% SLM
  Comments: Lowest memory latency configuration.

NOTE: ILM is only supported for some configurations, Please verify your system configuration is  supported for MaxUMA before making any changes.

Shell> memconfig -mi MaxUMA

memconfig : Reset required for memory configuration change to take effect.

HTH,

Surendar

Was the post useful? Click on the WHITE KUDOS! Star.

For HP Guided Troubleshooting; http://www.hp.com/support/hpgt

[tibey]

Same result with Max UMA after a reset.

NOTE: ILM is only supported for some configurations, Please verify your system configuration is  supported for MaxUMA before making any changes.

-> How can i verify this ?

Is it even possible to use interleave memry on a standalone (no nparts) configuration ?

[S_Logan]

Hi Tibey,

Please have a look in to this document and check the link provided in the document for memory subsystem informtion for Blades:

http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/kb/docDisplay/?javax.portlet.tpst=ba847bafb2a2d782fcbb0710b053ce01&javax.portlet.prp_ba847bafb2a2d782fcbb0710b053ce01=wsrp-navigationalState%3DdocId%25253Demr_na-c03200935

HTH,

Surendar

Was the post useful? Click on the WHITE KUDOS! Star.

For HP Guided Troubleshooting; http://www.hp.com/support/hpgt

[tibey]

Thanks for your answer.

The support document stating that BL890c i2 Server Blade needs 8 CPU for ILM is interesting... but does it applies to BL870c i2 ? Because my configuration has only 2 cpu (out of 4) , if BL870c i2 needs 4cpu for ILM that would explains my issues...

I 've already checked the document "Memory subsystem information for HP Integrity server blades" and according to what it says my configuration seems to be ok : i got 4 slots of 4gb ram for each cpu, same type (p/n M393B5270CH0-CH9).

[S_Logan]

Hi Tibey,

>>>>The support document stating that BL890c i2 Server Blade needs 8 CPU for ILM is interesting... but does it applies to BL870c i2 ? Because my configuration has only 2 cpu (out of 4) , if BL870c i2 needs 4cpu for ILM that would explains my issues...

As previously updated " ILM is only supported for some configurations".

With reference of the support document "BL890c i2 Server Blade needs 8 CPU for ILM" , it explains all cpu needs to be populated for configuring ILM.

HTH,

Surendar

Was the post useful? Click on the WHITE KUDOS! Star.

For HP Guided Troubleshooting; http://www.hp.com/support/

[S_Logan]

Hi  Tibey,

For More Information:

Please refer the document "Memory subsystem information for HP Integrity server blades"  Pgno:5 and 6

HTH,

Surendar

Was the post useful? Click on the WHITE KUDOS! Star.

For HP Guided Troubleshooting; http://www.hp.com/support/