Oracle RDBMS performance drop on Integrity bl860c i2 servers

[Wim Rombauts]

Since about one month, we are migrating our rx3600 and rx6600 servers to new bl860c i2 servers with a lot more memory.

Since the number of cores in the system doesn't increase signifficantly, I didn't expect big performance jumps, I just expected them to be about the same.

Now, when I look at the response time metrics of our application, I see that for frequent (= more than 100 times per hour) queries, response time is twice what is was on the rx systems (50ms instead of 25ms - response times are JDBC times per java method, not per query)

Our database version is 10.2.0.4, with a hew patch bundles on it.
Already tried :
- set _enable_NUMA_optimization=FALSE : no effect
- pstatectl disable and kctune pwr_idle_ctl 0 : no real effect
- Traced the Oracle sessions : No clear indication of increased wait times.

I have alrady a support case open at HP, but performance issues are difficult to get investigated at support, so :

Has anyone of you done a similar move, noticed similar behaviour and fixed it ?
Or has anyone a good idea for what to look at ?

[Kenan Erdey]

Hi,

is i2 server is one blade or a combined blade of 2 or more servers ? perhaps, because processes runs on cores of two different blades, cache and links between cpu's may result the problem. You can restrict the cores which the processes run, with PRM, and see what happens.

[INH]

Hi
Is i2 blade is dual core or quad core? how many CPUs?

If you feel DB performance is low.. check swap and shmmax parameter if that can be tuned further.

Rgds,
INH

[Turgay Cavdar]

What are difference between two systems?

Any difference on OS version/patches?

Ant difference storage configuration? Do you use the same disks for performance tests?

[Wim Rombauts]

Well, I can compare two sets of systems :

A 2-socket rx3600 with 56GB RAM has been replaced by a 1-socket bl860c i2 with 72GB RAM
A 3-socket rx6600 with 64GB RAM has been replaced by a 2-socket bl860c i2 with 144GB RAM

There is no difference in OS between the old and the new systems. All necessary upgrades have been done on the old systems, and they have been running that way for a few months.
The new systems were installed using Ignite/UX with a recovery tape created on the old system.

[Wim Rombauts]

We also use the same storage : All storage is LVM based and we just had to import the volume groups to recover the data on it.

Database parameters are not changed, shmmax was high enough to contain the full SGA, and that is still the case now.

[Jean-Luc Oudart]

Wim

would you have basline stats for OS and Oracle on previous hardware and new hardware to compare. eg STATSPACK reports.
You indicate that the response time at the java method level is twice.
The hardware has changed (and especially the amount of memory available), do you know of any changes in the code (java methods) that would have happened around same time ?

Regards
Jean-Luc

[Torsten.]

The errata mentions a patch related to I/O in general, do you have this installed?

http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c02111059/c02111059.pdf

Since Integrity systems also require certain firmware and efi driver requirements for I/O cards, you should check them too.


A problem here could cause retries and timeouts, hence lower performance.

[Wim Rombauts]

There has been no code change at that time. The application has built-in response time metrics, and I see a sudden jump on the day that we swapped the hardware.

Unfortunately, I only looked at this after a few weeks.
Ans since I didn't expect this, I also don't have a baseline AWR or statspack report from before the upgrade.
Thus : hard to detect where the problem is. Maybe, it has nothing to do with the database server, but everything with less efficient handling of java code (for some unknown reason.

I know the major change (besides the processor) is the way the system interfaces with memory, so I more or less expect there is a memory management issue, but I have no hard evidence.

[Wim Rombauts]

I checked the doucment :

Generally all driver versions, WBEM providers, ... are at revision 11.31.1009.
That gives me a releas higher than the minimum required.
Only DiagProdCollection has not been updated since september 2010.

As for a few patches I found :
PHCO_40554 -> OK
PHKL_40880 -> Superseding patch PHKL_40994 installed
PHKL_40942 -> Superseding patch PHKL_41362 installed