High IO wait on HP Itanium srevers

[Manoj (SAP Technology)]

Hello,

I'm experiencing the following issue with some of the customers who are using HP integrity systems. All are Oracle/SAP application servers.

We are facing high CPU io wait for all these servers. Storage systems varies from local disk to EVAs in different systems. Customers are NOT facing any performance issues due to this. But just curious why this high IO wait.

Oracle release : 9.2.0.6

All these systems are using HP filesystems and NOT raw devices. So no async io possible by default.

Only thing we could find was the following trace file entry;

Ioctl ASYNC_CONFIG error, errno = 1

As per SAP and Oracle notes, we have done following things to avoid this.

1. dba user is provided with MLOCK privileges.
2. Changed the following oracle parameters;

disk_asynch_io = FALSE
filesystemio_options = DIRECTIO

But still sar output is showing high io wait during normal operation.

Anybody has faced similar issues?
Your help on this is much appreciated.

Regards, Manoj

[Sameer_Nirmal]

Hi,

As you are aware the "wio" is CPU wait state for physical I/Os (block/raw). Maybe in your case , you would see high service time shown by the EVA disks.
I presume you are not getting 'ASYNC_CONFIG" error anymore, right?

The most probable causes for high "wio" are non-optimised entities like database layout on the EVA, Oracle configuration , Host/ Storage I/O setup.

EVA is quite sensitive to I/O pumped into it. Oracle which has invariably different types of I/Os , the matter becomes to concern about. It is always good idea to ensure optimised DB layout on EVA to get max throughtput from it. Oracle indexes,data, redolog,archive logs should be on separate Vdisks.

Secondly knowing the capability of EVA , more SCSI I/O could be queued in from host by increasing "scsi_queue_depth" parameter.
Zoning is must. You should ensure to equally distribute the Luns across two HSV controllers .Although EVA tries to load balance the HSVs ( when preferred path = no in CV-EVA), but it always good idea to pre-define the load balancing by setting preferred path ( controller A or B )thus releaving the EVA from such excercise contributing good throughput.
This will give you good control over I/Os being put on controllers. But you should know the I/O patterns well.

From Oracle side, one has to make sure it is tuned correctly to get optimised I/O for its operation. The tunning should be done in sync with OS to get better result.
e.g. SGA sizing , oracle buffers , OS kernel parameters , FS options ( direct, mincache, convosync as per I/O )

The bottom line is one has to look into various aspects as mentioned above to minimize "wio".

[Manoj (SAP Technology)]

Hello,

We are facing this problem even for systems which are having local disks. In these systems Oracle files are properly distributed. MLCOK for dba group solved the ASYNC_CONFIG error.

Do you think this is anyway related to HP-UX 11.23 kernel problems?

Regards, Manoj

[Ted Buis]

Two possible ways to help. One is to eliminate double buffering (Oracle and HP-UX buffer cache) if you have OnLineJFS. There are mount options that allow to do this. See the attached.

The other could be not taking full advantage of the EVA (or other array) queue depth. The default for HP-UX is 8 per LUN, but it can as high as 256. The command depth on the EVA controller is much deeper, so, it may be possible to more efficiently handle more I/O commands, if only the OS would send them. See the man page on scsictl (-m queuedepth).

[Manoj (SAP Technology)]

Hi,

Thank you for attending this.
I think the attachment is missing from your post. Could you please post it again?

Regards, Manoj

[Ted Buis]

Oops, sorry.

[Ted Buis]

Another interesting item.