100% IO/s in Disk whole day....

[hboo]

Hello all...

I have IO/s problems on disk (100%) in a HPUX 11.11 server that it is the central instance of SAP's BD, the conditions are the next:

- The server is connected to a SAN of Data (XP12000 of HP)

- The server is the active node of SAP's cluster

- The server has 5 cards of network(NIC's) operating and in moments reports necks of bottle to level of network, nevertheless, the switches are OK, and the configuration of the cards also

- IO's problem is general, that is to say, disk are to 100 % the whole day

Somebody can help me please...?

[Steven E. Protter]

Shalom,

1) See if the disk array itself has i/o problems. Check the tools that came with it. I'm assuming the OS is reporting 100% I/O. Often in such circumstances, the array does not also report 100% I/O

2) Have the array checked for compliance with good practices and configuration. Make sure no parameters that can effect performance are out of kilter.

3) See that the system itself is up to date on scsi and i/o patches, using a bi-annual patch set that is recent. See if all SAP patches are installed.

4) Check the data layout and make sure that write intensive parts of the database are on raid 1, not raid 5. Raid 5 slows down write intenseive databases and causes thisproblem.

SEP

[Ninad_1]

Apart from disk showing 100% (which I believe you are referring to glance output), you also need to check the avque (the average queue), the avwait and avserve times in the sar -d output to check if despite the disk being busy are there any queues on the disks and is the disk able to respond/serve in reasonable time.

Regards,
Ninad

[hboo]

Steven:

1)The array disk has no problems until now.

2)The array was checked by vendor, and has no problems, configurations is ok, and works good by another servers by the sames ports

3) The server is up to date, is a new RP5440 with all patches installed.

4)The disks of database of SAP (in XP12000) are configured Raid 5


Ninad:

The output by sar -d command is next:
[sirvep00]/home/fongsjeh> sar 1 5 -d /dev/dsk/c10t0d0

HP-UX sirvep00 B.11.11 U 9000/800 07/18/07

10:03:21 %usr %sys %wio %idle
10:03:22 42 7 47 5
10:03:23 49 12 39 1
10:03:24 33 6 59 2
10:03:25 41 7 51 1
10:03:26 28 11 58 3

Average 39 9 50 2

I don't see avque, avwait and avserve

Thanks for your comments!!

[Ninad_1]

Hi,

Please provide the output of the command
sar -d [ without any other parameters ]
The results you have pasted show the CPU stats - which are also relevant as it shows very high WIO figures, which means surely the system is IO intensive and possible IO bottlenecks.
Pls check sar -d

Regards,
Ninad

[Bill Hassell]

100% I/O (as reported by Glance) is a rather poor metric. It means that during the measurement period, one or more disks (LUNs) were busy during that time. So a single disk may be very busy, or 50 disks could be busy. Since you have a SAN-based disk array, the percentage bar isn't very meaningful.

Instead, go to the "d" screen (Disk) in Glance and look at Logical reads and writes. If you have at least 4 links to the SAN switch, you may see 50,000 to 100,000 requests for a relatively busy system. Now look at the "u" screen (IO by Disk) -- it may be several screens in length. The important column for system problems is Qlen where a queue greater than 1.0 is of concern. If there is something wrong with the disk or SAN switch, then requests begin to get queued. Have your SAN administrator look over the switch logs and XP12000 logs for possible problems.

On the other hand, if the application requires an enormous amount of disk records, then it is perfectly normal. You can ask the database administrator to look at the Oracle stats-pack to determine if there are inefficient searches or other procedures that can be improved. The Oracle and SAP administrators can improve performance by using a LOT more RAM more efficiently and reduce the amount of disk read/write activity. Consider adding 8 to 16Gb of additional RAM to help reduce disk activity.

[hboo]

Hi Bill

This is output for glance d, I don't see logical reads adn writes

---------------------------------------------------------------------------------------------------
PROCESS LIST Users= 7
User CPU Util Cum Disk Thd
Process Name PID PPID Pri Name ( 400% max) CPU IO Rate RSS Cnt
--------------------------------------------------------------------------------
oracleP01 16179 1 241 orap01 97.2/44.4 2210.6 0.0/22.3 27.0mb 1
oracleP01 20473 1 241 orap01 95.0/47.6 1544.2 3.0/53.9 26.0mb 1
oracleP01 13791 1 149 orap01 13.7/ 1.7 1772.7 590/20.4 36.0mb 1
dw.sapP01_D 21257 21217 155 p01adm 10.5/13.0 38641.1 0.0/ 0.6 101.8mb 1
oracleP01 13714 1 149 orap01 5.7/11.3 678.9 76.6/30.6 27.0mb 1
oracleP01 14611 1 149 orap01 4.8/ 0.8 144.0 184/20.3 28.0mb 1
oracleP01 24625 1 149 orap01 4.0/ 3.8 59.4 176/ 253 26.0mb 1
oracleP01 22029 1 149 orap01 3.8/ 1.8 5443.5 137/26.4 36.0mb 1
oracleP01 27210 1 154 orap01 3.6/ 6.7 29.7 80.2/38.0 26.0mb 1
dw.sapP01_D 21258 21217 154 p01adm 3.4/ 1.6 4847.2 0.0/ 0.1 104.2mb 1
oracleP01 21799 1 149 orap01 2.8/ 0.0 42.6 173/ 0.8 27.0mb 1

This is output for glance u command:

PROCESS LIST Users= 7
User CPU Util Cum Disk Thd
Process Name PID PPID Pri Name ( 400% max) CPU IO Rate RSS Cnt
--------------------------------------------------------------------------------
oracleP01 16179 1 241 orap01 94.2/45.0 2274.0 0.9/22.0 27.0mb 1
oracleP01 20473 1 241 orap01 92.5/48.5 1605.9 2.1/52.9 26.0mb 1
dw.sapP01_D 21257 21217 155 p01adm 10.6/13.0 38647.7 0.0/ 0.6 101.8mb 1
oracleP01 22259 1 149 orap01 9.8/ 1.4 4241.3 305/22.0 36.0mb 1
oracleP01 28001 1 156 orap01 9.4/ 4.0 6.8 367/27.7 26.0mb 1
oracleP01 24625 1 149 orap01 3.5/ 3.8 62.4 138/ 251 26.0mb 1
oracleP01 27210 1 154 orap01 3.3/ 6.3 32.0 88.8/44.4 26.0mb 1
oracleP01 22029 1 149 orap01 2.7/ 1.8 5446.3 86.2/26.4 36.0mb 1
oracleP01 4065 1 149 orap01 1.5/ 3.7 6786.3 219/23.7 27.0mb 1
oracleP01 22061 1 149 orap01 1.3/ 0.8 2275.8 126/14.0 36.2mb 1
oracleP01 7169 1 154 orap01 1.0/ 4.0 840.0 20.9/68.2 32.0mb 1

I don't see Qlen column....

[Bill Hassell]

Sorry, I wasn't very clear. When you run Glance, by default you are in the "g" screen. What this means is that if you type just 1 character, you will go to this screen. To see all the 1-character commands, type ? while Glance is running. Then to see the Disk Report, type the letter: d

And to see the IO by disk, type the letter: u

There are programmable keys at the bottom of the Glance screen for smart terminals but because there are so many (sometimes very poor) terminal emulators in use, they are not always functional. The exception is for HP smart terminals such as the 'real' 700/9x series (not PCs, real terminals), or the WRQ Reflection for HP (not Reflection/X), or the free HP emulator QCterm from http://www.aics-research.com/qcterm/

With HP terminal emulators, you will see the graphic bars in different colors and the softkeys will have meaningful labels for easier navigation.