HBA Performance on HP-UX 11.31

[John Everitt]

System: RX8640 running 11.31 with Sybase 15.0.1 attached to EVA8000 through dual HBAs (AB378-60101 4GB Single Port).

I seem to be getting very poor SAN performance. The SAN switches and EVA8000 do not appear to be overly loaded but when I do a sar -H, it seems to show the HBAs being very busy but achieving very little throughput. Any ideas as to the issue here would be greatly appreciated...

A typical SAR looks like ...

cptux12/sybase/data/tdbdata>sar -H 5 5

HP-UX cptux12 B.11.31 U ia64 03/03/08

11:46:40 ctlr util t-put IO/s r/s w/s read write avque avwait avse
rv
%age MB/s num num num MB/s MB/s num msec ms
ec
11:46:45 fcd0 93 0.59 249 228 21 0.45 0.14 1 0
9
fcd1 94 0.60 249 228 21 0.45 0.15 1 0
10
11:46:50 mpt0 100 0.04 7 2 4 0.00 0.03 1 0
2
mpt2 100 0.04 6 2 4 0.00 0.03 1 0
2
fcd0 98 0.55 231 214 17 0.42 0.13 1 0
27
fcd1 97 0.55 231 214 17 0.42 0.13 1 0
24
11:46:55 fcd0 87 0.59 256 239 17 0.47 0.12 1 0
13
fcd1 87 0.60 256 238 18 0.46 0.14 1 0
11
11:47:00 fcd0 68 0.40 198 191 7 0.39 0.01 1 0
6
fcd1 65 0.41 198 191 7 0.39 0.01 1 0
5
11:47:05 fcd0 81 0.86 354 319 35 0.63 0.23 1 0
6
fcd1 79 0.87 354 319 36 0.63 0.24 1 0
6

Average fcd0 85 0.60 258 238 20 0.47 0.13 1 0
12
Average fcd1 84 0.61 258 238 20 0.47 0.13 1 0
11
Average mpt0 100 0.01 1 0 1 0.00 0.01 1 0
2
Average mpt2 100 0.01 1 0 1 0.00 0.01 1 0
2

Thanks in advance.

John

[Hein van den Heuvel]

Why do you think it is slow?
What did you expect?
Is there a real problem or a perception issue?

An average of 258 MB/sec over a single link with a theoretical maximum of 800MB/sec with the right combination and quantity of reads and writes and IO sizes seems at least respectable to me, and maybe even good.
It most certainly will keep many disks very busy.

What was the load?
Those IOs seem very large. (1MB!)
True application or artificial test? Careful setup?
It looks like two simple "dd bs=1024k ..."
And some writes as well.

Here is a potentially relevant whitepaper...

They used Solid State Disks to avoid rotational/seek delays inherent in the real disks behind the EVA control.

http://www.qlogic.com/SiteCollectionDocuments/Education%20and%20Resource/whitepapers/whitepaper1/Scaling_SAN_Fabric.pdf

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[Patrick Wallek]

You mustn't confuse your speed terminology here. You HBA are 4 Gb (GigaBIT) per second.

4 Gigabits per second is equal 0.5 GB (Gigabytes) per second or 512 MB per second.

Your HBA's seem to be running at slightly more than 50% capacity (at least the theoretical maximum capacity).

This appears reasonable to me.

[Hein van den Heuvel]

Founn some more papers on the HP storage web site. Nothing map directly to you question it seems, but together they should help build some foundation for expectation.

I want to re-itterate a point I alluded to in the first reply. What is being measure and is it the real target load. The present output suggests few, large IOs. That is NOT typical for most applications. It could be the critical part for the target application, but if it is not, then there is very little point in mearuring it other then to get reference point.

What is more important to the applications
- MB/sec, or latency + IO/sec?
- Single stream performance
- High concurrency?

And if we want to be really serious about explaining observed performance, then more details (beyond part numbers) for the SAN and EVA are needed. Switch used? Number for disks? Grouping? Raid level used?

Here is a nice collection of whitepapers:

http://h18006.www1.hp.com/storage/arraywhitepapers.html

EVA4400 raw performance:

http://h71028.www7.hp.com/ERC/downloads/4AA1-8473ENW.pdf

EVA800 Exchange:
http://h71036.www7.hp.com/enterprise/downloads/PerformConfig_Exchange2003_EVA8000_1105.pdf
EVA800 Oracle:
http://h71028.www7.hp.com/ERC/downloads/4AA1-5658ENW.pdf

EVA800 SAP:
http://h71028.www7.hp.com/ERC/downloads/4AA1-5522ENW.pdf

Hope this helps some more,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[John Everitt]

Apologies, I must be reading this sar output incorrectly but my understanding is that across the 2 HBAs, they average 85% utilisation, I'm getting 0.60 MB/s throughput with about 250 IOPS. What should I be concluding from these figures ?

[John Everitt]

To add some more background, the system is a database server running two Sybase Database Servers. The database server metrics are suggesting that the processes are i/o bound. The SAN switches and the EVA 8000 are both suggesting, according to the HP engineer, that they are not stressed. I'll find out the model number of the SAN switches but assume they are fairly new 4GB HP switches.

Hope this helps.

John

[Patrick Wallek]

My apologies -- I misread the sar output. The formatting of the forums makes it difficult to read.

It appears that you are correct in that you are seeing 84-85% utilization and 0.6 MB/s throughput on fcd0 & fcd1.

[Hein van den Heuvel]

My apologies as well. I even tossed (cut & paste) the numbers in a spreadsheet but got the columns wrong. Looks like a very modest load indeed, yet relatively response times.

I wouldn't worry about the util%. That's just because you have a controller + many spindles help you. Take for example the Average for the fcd0. With 258 IO/sec and 12 ms response time on average it would take over 3 seconds each second to just do all those IOs. So it is clear many (most) are happening while other IOs are happening and the thus it looks busy most all the time.

Hein.

[Steven E. Protter]

Shalom John,

Based on previous posts, I'd look away from a SAN issue and perhaps take a look at the way the application reading the sybase data is working.

Its clear based on a difficult reading that you are pushing 80% of your SAN capacity.

With hard hardware, your database must be working awfully had to make that happen.

A common problem in such situations is poorly written application code.

SEP

[John Everitt]

Steven,

I'm trying to look at this issue from all angles. I'm currently looking at the application which does have a few issues, at Sybase itself, at the OS and also the SAN infrastructure. Part of this happens to be looking at the HBAs. Now it maybe that this is indeed not a problem but it does seems weird that sar -H seems to suggest the HBAs being heavily utilised but providing very little throughput. I'm just hoping that someone can shed some light on what's going on here and potentially put my mind at rest so I can concerntrate on other issues. It could even be a conclusion that the HBAs are indeed loaded and this should be expected but I could reduce this load by adding more HBAs to the system.

One thing though, as far as I know HP-UX uses processor 0 to process interrupts for the NICs. Is there something similar for the HBAs. Sybase tends to monopolise CPUs so if a processor doesn't become available for a while could there be issues with HBA performance ?

One last thing, we changed the scsi command queueing algorithm to cell based round robin. Is this recommended ? We are currently using a queue depth of 64 ...

Thanks.

John

[Steven E. Protter]

Another thought John,

Perhaps the disk arrangement on the SAN is poor based on application use.

Heavy write tables sitting on RAID 5. Thats a common problem that many people overlook.

I was feeling outside of the box today and took such an approach.

The EVA has some utilities built into its DL380 "cough" Windows workstation controller that might be able to identify hot spots.

SEP

[John Everitt]

Steven,

Funny you should mention this but we are in the process of recreating a number of disk devices Raid-1 that was previously Raid-5 specifically due to the increased number of write operations required.

But, and I apologise for harping on about this, I am still concerned / confused about the HBAs on the system reporting that they are very highly utilised but show a very low throughput and IOPS count. What does it actually mean when the HBA reports that it is 100% busy. Does the OS now have to queue requests or is this a largely irrelevant metric and in fact the HBA can still handle more load ?

John

[Hein van den Heuvel]

>> But, and I apologise for harping on about this, I am still concerned / confused about the HBAs on the system reporting that they are very highly utilised but show a very low throughput and IOPS count.

YOu are not the first, nor the last to worry about this.
I tried to address that in my reply "Apr 6, 2008 02:37:05 ". Re-read?

>> What does it actually mean when the HBA reports that it is 100% busy.

It means that 100% of the time, there was some activity to the disk: One, or more IOs in progress. But that subsystem can easiy handle 100+ IOs in process, at the same time. So the busy measure give no indication on remaining capacity.

>> Does the OS now have to queue requests

Only when "scsi_max_qdepth" is reached.
That is there to protect against overload.
It is all too often left at default (8), but can be turned up a bunch. Google for previous writtings on the subject.

The avque time reported suggest this is not an issue.

>> or is this a largely irrelevant metric

Yes

>> and in fact the HBA can still handle more load ?

Yes.


Hein.