High I/O rates on disks connected to external SAN

[Jonathan Caplette_1]

Hi guys,

I got a box connected to an Hitachi SAN trough HP Tachyon XL2 Fibre Channel Mass Storage Adapter. My server is an HP rp8420.

The adapter is set to 2 GB speed. We are currently experiencing slow response (on the application side), when this slow time happend, I can see through Glance Plus that a lot of I/O rates hits the disks "SAN" and locally. The main application is Oracle and Peoplesoft.

I see no error in my OS logs, the cards are CLAIMED, no errors with EMS... My question is how can I diagnosys the I/O that I see in Glance, I wanna know within the card, is there an util that I can use to see what hit the card exactly?

Regards
Jonathan

[Murat SULUHAN]

Hi Jonathan

Why don't you check Oracle side,

You can measure how many sql and how much read/write on disk subsystem on specific period with statpack.

If your Oracle is heavy loaded disk I/O you will see high disk I/O rates.

Do you have single fiber? if not do you use SecurePath?.

Local disk I/O is not related with SAN

Best Regards
Murat

[Tim Nelson]

How about confirming how the array is performaing before going down all these other non-related paths.

sar -d 5 10

What is the service time ?
Are there any disk queues ?
( I would expect such, but, if the disk are responding and there are no queues, then your problem is somewhere else )

[Jonathan Caplette_1]

I'm not the DBA... I've guys working on it right now...

I've got two cards but i'm not using SecurePath should I? I've manage to split my LVM disk when I create a VG to use 50/50 the both path... I'm going to read on SecurePath I don't know this product! :)

I know that local disk I/O is not related with SAN, but I forgot to tell something, Oracle binarie are on the local disks while Oracle data are on our SAN disks, is moving Oracle binaries on SAN disk can offload I/O from local disk?

jonathan

[Jonathan Caplette_1]

# sar -d 5 10

HP-UX psa2h01 B.11.11 U 9000/800 11/12/07

11:06:35 device %busy avque r+w/s blks/s avwait avserv
11:06:40 c0t6d0 2.39 0.50 5 41 0.00 5.74
c2t6d0 0.40 0.50 3 33 0.00 1.29
c12t1d0 0.20 0.50 0 0 0.00 3.94
c13t1d1 66.93 0.69 174 10410 0.48 17.08
c12t1d2 48.21 0.53 286 18425 0.10 5.44
c13t1d3 99.80 2.13 500 21265 3.55 13.09
c12t1d4 26.49 0.90 148 3394 0.80 5.07
c13t1d5 21.51 0.50 15 1629 0.00 18.98
c12t1d6 17.33 0.80 129 3920 0.73 6.73
c12t2d0 2.39 0.50 16 629 0.00 2.63
c12t4d0 0.60 0.50 1 16 0.00 6.73
11:06:45 c0t6d0 5.81 0.50 29 248 0.00 2.52
c2t6d0 0.60 0.50 5 44 0.00 1.30
c12t4d1 0.20 0.50 0 2 0.00 5.37
c13t1d1 59.12 0.86 107 5473 2.23 21.03
c12t1d2 58.72 1.06 264 17453 1.49 9.33
c13t1d3 74.75 0.88 194 16622 1.28 16.13
c12t1d4 42.08 0.56 240 12300 0.14 6.19
c13t1d5 5.01 0.50 6 1840 0.00 16.39
c12t1d6 1.20 0.73 13 279 1.35 4.20
c12t2d0 2.00 0.50 7 151 0.02 5.73
c12t0d2 0.20 0.50 1 10 0.00 3.64
11:06:50 c0t6d0 3.19 0.50 7 68 0.00 5.62
c2t6d0 0.40 0.50 3 34 0.00 1.75
c13t1d1 86.23 1.13 190 6357 2.91 20.52
c12t1d2 81.44 0.83 1021 58935 0.44 3.76
c13t1d3 35.93 0.67 41 1022 0.95 21.97
c12t1d4 48.90 0.51 102 2772 0.05 7.04
c13t1d5 4.19 0.50 8 380 0.02 10.13
c12t1d6 0.20 0.50 1 10 0.00 4.01
c12t2d0 6.19 0.50 11 213 0.02 7.44
c12t4d0 10.98 0.50 21 2770 0.00 9.84
11:06:55 c0t6d0 6.20 0.50 14 149 0.00 6.82
c2t6d0 1.60 0.50 7 63 0.00 3.32
c13t1d1 46.20 0.58 49 1235 0.53 16.50
c12t1d2 18.40 0.50 76 2672 0.02 4.43
c13t1d3 86.40 0.84 162 4112 1.46 18.39
c12t1d4 88.00 1.51 239 5674 1.77 9.07
c13t1d5 12.60 1.29 21 598 1.53 16.00
c12t1d6 0.40 0.50 0 6 0.00 7.61
c13t1d7 0.60 0.50 0 0 0.00 26.19
c12t2d0 0.40 0.50 3 24 0.00 2.09
c13t2d1 0.20 0.50 0 0 0.00 11.43
c12t4d0 2.20 0.50 7 134 0.00 3.47
11:07:00 c0t6d0 2.59 0.50 6 41 0.00 5.37
c2t6d0 0.20 0.50 3 18 0.00 0.99
c13t1d1 34.33 0.50 27 498 0.03 14.44
c12t1d2 18.36 0.50 52 843 0.00 3.58
c13t1d3 98.20 0.63 197 4880 0.28 13.37
c12t1d4 70.66 0.50 146 2424 0.00 5.91
c13t1d5 5.59 0.50 9 137 0.00 6.37
c12t2d0 0.60 0.50 1 14 0.00 8.76
c12t4d0 7.39 0.50 24 2785 0.00 5.82
11:07:05 c0t6d0 8.42 0.50 23 164 0.00 5.14
c2t6d0 2.00 0.50 17 111 0.00 1.82
c12t4d1 0.20 0.50 0 1 0.00 4.07
c13t6d6 0.20 0.50 0 0 0.00 8.32
c13t6d7 0.20 0.50 0 0 0.00 8.32
c13t7d0 0.20 0.50 0 0 0.00 8.42
c13t1d1 21.04 0.50 22 467 0.00 11.28
c12t1d2 27.66 0.51 275 13865 0.03 4.26
c13t1d3 98.00 0.61 220 5329 0.43 11.42
c12t1d4 63.13 0.50 152 2430 0.00 4.85
c13t1d5 9.02 0.50 9 144 0.00 10.29
c12t1d6 0.40 0.50 0 6 0.00 10.68
c12t2d0 13.63 0.50 20 181 0.00 8.11
c12t4d0 1.80 0.50 5 96 0.00 3.83
c12t0d2 0.40 0.50 1 11 0.00 3.51
11:07:10 c0t6d0 2.20 0.50 6 36 0.00 4.08
c2t6d0 1.00 0.50 4 29 0.00 2.79
c13t1d1 24.55 0.50 41 1919 0.01 13.08
c12t1d2 99.00 1.76 2037 99127 0.79 3.45
c13t1d3 98.40 0.65 208 6438 0.61 13.96
c12t1d4 28.94 0.52 44 1239 0.10 10.74
c13t1d5 0.60 0.50 1 26 0.00 9.73
c12t2d0 1.40 0.50 3 40 0.00 6.37
c12t4d0 24.35 0.50 6 1143 0.00 47.88
c12t0d2 0.60 0.50 1 11 0.00 8.18
11:07:15 c0t6d0 13.20 0.50 31 236 0.00 5.40
c2t6d0 2.40 0.50 22 161 0.00 1.42
c12t4d1 0.40 0.50 1 2 0.00 8.17
c13t1d1 47.00 0.50 34 2179 0.00 21.49
c12t1d2 99.80 1.57 1479 76048 0.69 4.46
c13t1d3 99.20 0.82 247 12587 0.71 14.92
c12t1d4 58.60 0.99 531 21174 0.70 5.41
c13t1d5 4.20 0.50 8 509 0.00 13.25
c12t1d6 16.40 0.50 146 8250 0.00 4.83
c12t2d0 2.20 0.50 2 18 0.00 11.18
c12t4d0 8.00 0.50 13 1662 0.00 11.95
c12t0d2 2.60 0.50 3 28 0.00 9.96
11:07:20 c0t6d0 2.80 0.50 8 63 0.00 4.69
c2t6d0 1.00 0.50 6 52 0.00 1.84
c13t1d1 95.60 0.72 427 19747 0.66 11.40
c12t1d2 88.40 0.83 369 18445 0.45 6.96
c13t1d3 99.80 1.11 337 14547 1.24 13.95
c12t1d4 14.40 0.50 30 478 0.00 4.84
c13t1d5 1.00 0.50 1 19 0.00 8.90
c12t1d6 10.80 0.52 36 1776 0.75 9.52
c12t2d0 1.20 0.50 2 13 0.00 8.98
c12t4d0 0.60 0.50 0 5 0.00 14.78
c12t0d2 0.20 0.50 0 1 0.00 4.52
11:07:25 c0t6d0 2.81 0.50 5 46 0.00 6.23
c2t6d0 0.40 0.50 3 34 0.00 1.45
c13t1d1 61.52 1.01 222 12008 1.54 13.46
c12t1d2 64.93 0.80 267 14522 0.49 7.56
c13t1d3 99.80 2.59 438 20899 2.94 12.78
c12t1d4 34.27 0.52 108 4864 0.03 5.31
c13t1d5 33.47 0.50 27 2719 0.00 19.34
c12t1d6 2.40 0.60 18 641 0.30 3.91
c12t2d0 1.80 0.50 3 33 0.00 5.29
c12t2d1 0.20 0.50 0 0 0.00 1.29
c12t4d0 10.22 0.50 23 2791 0.00 8.59

Average c0t6d0 4.96 0.50 13 109 0.00 4.85
Average c2t6d0 1.00 0.50 7 58 0.00 1.78
Average c12t1d0 0.02 0.50 0 0 0.00 3.94
Average c13t1d1 54.26 0.81 129 6029 1.18 15.23
Average c12t1d2 60.50 1.29 613 32051 0.62 4.53
Average c13t1d3 89.02 1.38 254 10768 1.75 13.99
Average c12t1d4 47.54 0.83 174 5671 0.55 6.20
Average c13t1d5 9.72 0.66 11 800 0.31 15.28
Average c12t1d6 4.92 0.63 34 1490 0.42 5.98
Average c12t2d0 3.18 0.50 7 132 0.01 6.16
Average c12t4d0 6.62 0.50 10 1141 0.00 10.59
Average c12t4d1 0.08 0.50 0 0 0.00 6.55
Average c12t0d2 0.40 0.50 1 6 0.00 7.60
Average c13t1d7 0.06 0.50 0 0 0.00 26.19
Average c13t2d1 0.02 0.50 0 0 0.00 11.43
Average c13t6d6 0.02 0.50 0 0 0.00 8.32
Average c13t6d7 0.02 0.50 0 0 0.00 8.32
Average c13t7d0 0.02 0.50 0 0 0.00 8.42
Average c12t2d1 0.02 0.50 0 0 0.00 1.29

[A. Clay Stephenson]

I wouldn't expect to see much improvement (if any) resulting from moving the Oracle binaries to the SAN. The i/o associated with loading programs should be trivial compared to the data that these programs access. If you are seeing heavy local disk usage then it may well be termporary files or large spool files that are being written. You need to use Glance and drill down into the local disks and find the applications hitting them. You may be able to change environment variable settings to move this activity off the local disks. Of course, one thing that could cause significant local disk activity AND trigger performance losses is significant paging activity. The very first thing you should look for is swapping under periods of heavy load.

[Tim Nelson]

Looks like the following are the busiest with some good queuing happening.
Average c13t1d1 54.26 0.81 129 6029 1.18 15.23
Average c12t1d2 60.50 1.29 613 32051 0.62 4.53
Average c13t1d3 89.02 1.38 254 10768 1.75 13.99
Average c12t1d4 47.54 0.83 174 5671 0.55 6.20


Start looking at what is on these disks, how they are configured on the array. What the performance looks like from the array's pepspective.

SecurePath is a multi-pathing load balance software that provides load balancing accross the FC controllers.

There are many solutions to this issue but need to gather all the info to take the right path.

[Murat SULUHAN]

Hi

SecurePath is very good solution for SAN load balancing and failover situtations. You can set load balancing algorithms for each LUN
if you want to choose right LB algorithms you must know SAN DiskArray's controller type/working mech./active-passive ...

If you don't know the array controllers working mechanism for example which controller is primary controller for specific LUN, your 50/50 rule may be fail.

To move Oracle binaries from local to SAN does not very important effects on performance.

You can also check the Oracle SGA settings, kernel settings for Oracle.

Best Regards
Murat

[Jonathan Caplette_1]

Tim: These disk are Oracle data disks...

How can I know the wait are waiting for what?

[Murat SULUHAN]

Hi Jonathan

You should work with DBA and find out which SQL s most running, which SQLs generate high I/Os on SAN.

You should work on Oracle side if you are sure about hw and OS.

Best Regards
Murat

[Tim Nelson]

You can start drilling down on each of these paths with glance. Other than read/write information you will not know what is being waited on or what it is doing. You will have to coordinate these efforts with the DBA folks.

The typical goal is to reduce the bottleneck by either making the disk faster or by spreading out the IOs to reduce the contention( sometimes this is the same solution ).

Item 1: Segregation
If these disk share data and intent/archive logs the first option would be to separate these two.

Item 2: Configuration
The other option depending on the array configuration would be to move this data to disk groupings with more spindles.

If the disk array shows little load on these volumes then there is an outside chance that the bottleneck is at the driver queue. This defaults to 8 requests. ( again this item is an outside chance, review array performance before even bothering )

There are always cases where, IF, Oracle is not buffering well it leads to excessive IOs, how about some more SGA ?

[Ted Buis]

There are many opportunities for peformance bottlenecks. The first with Oracle that comes to mind is the potential for double buffering within the SGA and with the OS. This is not good. There are mount options that can let you avoid this issue (may need OnLine JFS for some of these). The attached document is old, but I think it is still relevant in concept. SecurePath or more appropriately AutoPath portion has support for Dynamic multipathing, but it may have issues with a Hitachi array, since the XP has a different ID string and you didn't say that it was even the XP equivalent. Easier and almost as good is to simply statically load balance so two FC adapters. You can do better by adding multiple FC adapters and using LVM to stripe across them to multiple LUNs on multiple ports of the array (assuming you have the corresponding ports on the Hitachi array). For fine tuning you can increase the queue depth using the ioctl command. More RAM can also help at times to reduce the number of physical I/Os that are actually done compared to logical I/Os, but this is application dependent. Sometimes it is best just to get an HP performance consultant out to help. They can collect data and give you a report of the best approaches to tune perfomance. They do much of this with Oracle. Otherwise, I would suggest starting with latest edition of the book from Robert F. Sauer on HP-UX Tuning and Performance.