General
cancel
Showing results for 
Search instead for 
Did you mean: 

Oracle RDBMS performance drop on Integrity bl860c i2 servers

Wim Rombauts
Honored Contributor

Oracle RDBMS performance drop on Integrity bl860c i2 servers

Since about one month, we are migrating our rx3600 and rx6600 servers to new bl860c i2 servers with a lot more memory.

Since the number of cores in the system doesn't increase signifficantly, I didn't expect big performance jumps, I just expected them to be about the same.

Now, when I look at the response time metrics of our application, I see that for frequent (= more than 100 times per hour) queries, response time is twice what is was on the rx systems (50ms instead of 25ms - response times are JDBC times per java method, not per query)

Our database version is 10.2.0.4, with a hew patch bundles on it.
Already tried :
- set _enable_NUMA_optimization=FALSE : no effect
- pstatectl disable and kctune pwr_idle_ctl 0 : no real effect
- Traced the Oracle sessions : No clear indication of increased wait times.

I have alrady a support case open at HP, but performance issues are difficult to get investigated at support, so :

Has anyone of you done a similar move, noticed similar behaviour and fixed it ?
Or has anyone a good idea for what to look at ?
10 REPLIES
Kenan Erdey
Honored Contributor

Re: Oracle RDBMS performance drop on Integrity bl860c i2 servers

Hi,

is i2 server is one blade or a combined blade of 2 or more servers ? perhaps, because processes runs on cores of two different blades, cache and links between cpu's may result the problem. You can restrict the cores which the processes run, with PRM, and see what happens.
Computers have lots of memory but no imagination
INH
Regular Advisor

Re: Oracle RDBMS performance drop on Integrity bl860c i2 servers

Hi
Is i2 blade is dual core or quad core? how many CPUs?

If you feel DB performance is low.. check swap and shmmax parameter if that can be tuned further.

Rgds,
INH
Knowledge is power
Turgay Cavdar
Honored Contributor

Re: Oracle RDBMS performance drop on Integrity bl860c i2 servers

What are difference between two systems?

Any difference on OS version/patches?

Ant difference storage configuration? Do you use the same disks for performance tests?
Wim Rombauts
Honored Contributor

Re: Oracle RDBMS performance drop on Integrity bl860c i2 servers

Well, I can compare two sets of systems :

A 2-socket rx3600 with 56GB RAM has been replaced by a 1-socket bl860c i2 with 72GB RAM
A 3-socket rx6600 with 64GB RAM has been replaced by a 2-socket bl860c i2 with 144GB RAM

There is no difference in OS between the old and the new systems. All necessary upgrades have been done on the old systems, and they have been running that way for a few months.
The new systems were installed using Ignite/UX with a recovery tape created on the old system.
Wim Rombauts
Honored Contributor

Re: Oracle RDBMS performance drop on Integrity bl860c i2 servers

We also use the same storage : All storage is LVM based and we just had to import the volume groups to recover the data on it.

Database parameters are not changed, shmmax was high enough to contain the full SGA, and that is still the case now.
Jean-Luc Oudart
Honored Contributor

Re: Oracle RDBMS performance drop on Integrity bl860c i2 servers

Wim

would you have basline stats for OS and Oracle on previous hardware and new hardware to compare. eg STATSPACK reports.
You indicate that the response time at the java method level is twice.
The hardware has changed (and especially the amount of memory available), do you know of any changes in the code (java methods) that would have happened around same time ?

Regards
Jean-Luc
fiat lux
Torsten.
Acclaimed Contributor

Re: Oracle RDBMS performance drop on Integrity bl860c i2 servers

The errata mentions a patch related to I/O in general, do you have this installed?

http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c02111059/c02111059.pdf

Since Integrity systems also require certain firmware and efi driver requirements for I/O cards, you should check them too.


A problem here could cause retries and timeouts, hence lower performance.

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Wim Rombauts
Honored Contributor

Re: Oracle RDBMS performance drop on Integrity bl860c i2 servers

There has been no code change at that time. The application has built-in response time metrics, and I see a sudden jump on the day that we swapped the hardware.

Unfortunately, I only looked at this after a few weeks.
Ans since I didn't expect this, I also don't have a baseline AWR or statspack report from before the upgrade.
Thus : hard to detect where the problem is. Maybe, it has nothing to do with the database server, but everything with less efficient handling of java code (for some unknown reason.

I know the major change (besides the processor) is the way the system interfaces with memory, so I more or less expect there is a memory management issue, but I have no hard evidence.
Wim Rombauts
Honored Contributor

Re: Oracle RDBMS performance drop on Integrity bl860c i2 servers

I checked the doucment :

Generally all driver versions, WBEM providers, ... are at revision 11.31.1009.
That gives me a releas higher than the minimum required.
Only DiagProdCollection has not been updated since september 2010.

As for a few patches I found :
PHCO_40554 -> OK
PHKL_40880 -> Superseding patch PHKL_40994 installed
PHKL_40942 -> Superseding patch PHKL_41362 installed
Torsten.
Acclaimed Contributor

Re: Oracle RDBMS performance drop on Integrity bl860c i2 servers

My idea is, if it is slower on different hardware, it could be related to the hardware.

Unfortunately the release notes of the FC cards EFI drivers and firmware are almost useless, you can only find something like "several problems fixed" only.

I would check the firmware of the FC mezzanine cards and the blade interconnects, since I assume you access your data via fibre channel.

Same for hardware dependent patches (drivers) at the OS level, because you definitely use different FC cards compared to your old server.

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!