Increase in Wait IO

[Sani]

We have 2 nos of hpux 11iv3 servers as oracle database server configured for 2 mode RAC .The serever configuration is SD64A superdom with 48 cpu and 64 gb memory.Recently we migrated oracle version from 9i RAC to 10g RAC .

Now from " sar -u " we have observed that %WIO is around 40 % , where as with 9i RAC this was well below 10 %


Any body observed this issue with 10g migration .??

[Duncan Edmonstone]

So do you actually have a performance issue or is this just "an observation"?

9i was a "straight port" to Integrity/IA64, with virtually no code changes to enhance performance for Itanium, and it also used a much earlier version of the HPUX compiler which didn't produce as efficient code.

10g has *always* performed much better on IA64 than 9i did, so you could simply be seeing that effect... the code is running more efficiently, is less cpu-bound, which almost inevitably means it is becoming io-bound.

So do you have actual performance concerns? Did you change anything else when you moved to 10g? (e.g. moving to ASM?)

HTH

Duncan

[Emil Velez]

Yes but with 10G you probably have faster performance of the same workload overall since 10G is more efficient. This means that the processes are running faster and the i/o queue is being backed up a bit waiting for i/o s to complete since the processes are running faster.

Now get faster disks and then you will have a CPU issue since processes are doing I/Os faster more processes can do more.

Its a ever chasing issue.

[Sani]

Hi Duncan Edmonstone/Emil Velez ,

Thanks for reply.Our servers are PA RISC not Itanium .We didnt change any H/W configuration for moving to 10g.The only one change is, oracle 10g is using its own clusterware .

We couldnot find any performance related problems now also.but over all CPU utilisation inreases because of high WIO

Emil Velez, have u faced the same issue at your end.?

[Duncan Edmonstone]

>> The only one change is, oracle 10g is using its own clusterware .


So are you also using ASM?

HTH

Duncan

[Hein van den Heuvel]

>> We couldnot find any performance related problems now also.

Good. So there is only a lack of understanding,

Wait IO is NOT a bad thing.
It just means the CPUs are IDLE waiting for an IO to be finished to take a next step.

This may have ZERO impact on the end users (if it is Oracle page writes, or archive writes), or it could define the user experience, if it is the redo log write IOs,
or a data page read.

>> but over all CPU utilisation inreases because of high WIO

Please explain the above?
WIO is IDLE time.
Why would that cause or explain overall CPU utilisation?
It does REDUCE the time reported as IDLE which some folks may simplistically see as an increase in CPU, but those folks would be incorrect.

hth,
Hein van den Heuvel
HvdH Performance Consulting.

[Sani]

HI Duncan

We are not using ASM .

[Jeeshan]

also check "sar -b" and "sar -v".

[Duncan Edmonstone]

So same database/application and same system, just newer version of Oracle? Did you upgrade to 11iv3 at the same time, or were you running 11iv3 with 9i (presumably not as I'm pretty sure thats not supported)

Presumably if no ASM, then you are using Serviceguard Extension for RAC, or are you using Clusterware with just raw disk ( /dev/rdsk/... )?

So what is your storage? How is it laid out? How were you/are you doing MPIO to disk?

HTH

Duncan

[Md. Farhan A Azam]

hello,

autopath is installed in the server, if yes which load balancing topolgy applied.

and also tune the database properly.

[Sani]

HI Duncan

There is no change in OS version also.We we are using v3 . All oracle datafiles and clusterware are using raw devices.

There is no secjre path , as it is V3 and we use "least_cmd_load " as load balancing policy.
thanks

[Md. Farhan A Azam]

Hi,

Check database properly, as we faced same problem...after database tuning, sar ouput is ok.

[Duncan Edmonstone]

As already indicated, if *all* you have changed is the version of Oracle, then logic says the fix is going to be in Oracle somewhere. Do you have DBAs looking at internal Oracle stats (AWR/statspack etc)?

> we use "least_cmd_load " as load balancing policy.

with what sort of disk array? and how are the LUNs configured??

HTH

Duncan

[skt_skt]

Sanil,

We do see same symptoms after 10g upgrade on 11.23 but with no peformance problem. Since the changes is only on Oracle the fix/tune is at Orcle level only.

What do you observe interms of GBL_PRI_QUEUE? (Perfview related)

[Sani]

Hi Santhosh,

Happy to see your reply.Did oracle provided any fix for this at ur end ?


GBL_PRI_QUEUE is around 10 .

Hi Duncan ,

We have XP24K san array.

[Sani]

Hi Santhosh ,

No reply from You ???

[Hein van den Heuvel]

Sani,

Much like you, Santosh indicates a higher wait IO , but NO PROBLEM.

As per my prior reply... Why do you think the high 'WaitIO' is a problem? It is just a flavor of IDLE !

Did your SYS and USER time increase or decrease since the change for comparable loads (days)?

Do you have end-user performance stats or Oracle statspack data to obtain somewhat normalized transaction costs before and after?

Regards,
Hein.

[skt_skt]

There was no UNIX realted recommendations from Oracle support. If you meant any DB realted tuning then yes . But i dont have those informations.

[Eric SAUBIGNAC]

"Why do you think the high 'WaitIO' is a problem? It is just a flavor of IDLE !"

--> So true, Hein. As far as I know, only HP-UX shows this metric in "sar -u". Many of my clients are confused with %wio :-(

Sani, you should really understand that %wio shows that you have a lot of IO, but not necessary an IO bottleneck. It also means that the CPU is not overloaded. Good news, no ?

If you want to check wether or not there is a problem at IO level, check with "sar -d" and follow metrics "avwait" and "avserv". Under 5ms, life is beautiful. Until 10 ms, not so bad. Above, begin to investigate.

Eric

[Sani]

HI Farhan ,

What tuning was done in database side ?
Can you explain.