topic Re: HP-UX 11.31, HBA usage 100% with low throughput and IOPS in Operating System - HP-UX

HP-UX 11.31, HBA usage 100% with low througput and IOPS

Francis Noël — Tue, 20 Sep 2011 18:14:39 GMT

Hello new Forum

I realize my question will be start ou vague but here we go.

Got an Rx7640.
Runs a mostly patched Feb 2007 release of 11.31
is hooked up to an SSD-based storage via HP AB378-60101 4Gb 's.
Single LUN cooked though LVM, formatted with vxfs 6.

Problem : HBA usage shoots up to 100% while througput and IOPs stay ridiculously low.

Example :

sar -H 1 10

HP-UX hp-prod1 B.11.31 U ia64    09/20/11

14:10:22     ctlr   util t-put  IO/s   r/s   w/s   read  write avque avwait avserv
                   %age   MB/s   num   num   num   MB/s   MB/s   num   msec   msec
14:10:23     fcd0    99   5.33    89     1    88   0.00   5.33   161      0      1
             fcd1   100   5.67    97     0    97   0.00   5.67   117      0      1
14:10:24     fcd0    99   5.55    90     0    90   0.00   5.55   161      0      1
             fcd1    99   5.47    89     0    89   0.00   5.47   117      0      1
14:10:25     fcd0   100   5.26    85     0    85   0.00   5.26   161      0      1
             fcd1   100   5.32    85     0    85   0.00   5.32   117      0      1
14:10:26     fcd0    99   5.28    87     0    87   0.00   5.28   161      0      1
             fcd1    99   5.44    88     0    88   0.00   5.44   117      0      1
14:10:27     fcd0   100   5.29    85     0    85   0.00   5.29   161      0      1
             fcd1   100   5.60    92     0    92   0.00   5.60   117      0      1
14:10:28     fcd0   100   5.21    85     0    85   0.00   5.21   161      0      1
             fcd1   100   5.54    90     0    90   0.00   5.54   117      0      1
14:10:29     fcd0   100   5.47    90     0    90   0.00   5.47   161      0      1
             fcd1   100   5.55    94     0    94   0.00   5.55   117      0      1
14:10:30     fcd0    99   5.38    86     0    86   0.00   5.38   161      0      1
             fcd1   100   5.49    90     0    90   0.00   5.49   117      0      1
14:10:31     fcd0   100   5.40    90     2    88   0.00   5.40   161      0      1
             fcd1   100   5.49    89     0    89   0.00   5.49   117      0      1
14:10:32     fcd0   100   5.38    88     0    88   0.00   5.38   161      0      1
             fcd1   100   5.48    92     0    92   0.00   5.48   117      0      1

Average      fcd0   100   5.35    88     0    87   0.00   5.35   161      0      1
Average      fcd1   100   5.50    91     0    91   0.00   5.50   117      0      1

Data is being restored from fiberchannel tapes.
I need to find out where the pipes are jamming and I'm running out of ideas.
This is my first time working on this box so no prior knowledge as to if this used to work or not.

Anyone ever seen something similar ?

Re: HP-UX 11.31, HBA usage 100% with low througput and IOPS

Duncan Edmonstone — Wed, 21 Sep 2011 09:09:21 GMT

You need to understand what "utilization" means in this context. It doesn't mean that 100% of the bandwidth or capability of the card is being used, it simply means that "during the interval, the HBA was carrying out IO operations 100% of the time". That doesn't mean its not capable of more... I would look elsewhere if I were you...

Incidentally if the data is being restored from FC tapes, where are the reads? I only see write below, or have you removed some additional FC controller info from the output below?

Re: HP-UX 11.31, HBA usage 100% with low througput and IOPS

Francis Noël — Wed, 21 Sep 2011 16:26:21 GMT

Hello Duncan and thank you for stepping up.

Indeed I am looking elsewhere. The SAN vendor is investigating his box and it looks like a firmware patch is needed to correct an "intensive write" bug... This would mesh well with your explanation of 100% usage if the few IOs we do manage to get through to this thing take a long time to complete. Is there such a metric as IO completion time I could look at to confirm the hypothesis ? The server does have (and old version of) Glance installed.

As for the absence of reads I have no explanation at this point. I can only assume that the tapes are twiddling their proverbial thumbs and the 10 seconds I caputred was fed by our copious Netbackup Buffers. I shall try a longer sample.

Thank you very much for your input Duncan, your comments did enhance my understanding of the situation.

Re: HP-UX 11.31, HBA usage 100% with low throughput and IOPS

Dennis Handly — Wed, 21 Sep 2011 20:54:28 GMT

>You need to understand what "utilization" means in this context. It doesn't mean that 100% of the bandwidth or capability of the card is being used, it simply means that "during the interval, the HBA was carrying out I/O operations 100% of the time".

Hmm, this seems like a strange definition of "utilization". I would think it should include "waiting" or "idle" in the mix, the same as how you measure CPU usage.

Re: HP-UX 11.31, HBA usage 100% with low throughput and IOPS

Kris_Knigga — Mon, 03 Oct 2011 20:31:23 GMT

I believe the utilization percentage is calculated with Little's Law (http://bnrg.eecs.berkeley.edu/~randy/Courses/CS252.S96/Lecture21.pdf - page 10 specifically). Basically utilization equals the average number of requests per second multipied by the average time needed to complete a request.

Re: HP-UX 11.31, HBA usage 100% with low throughput and IOPS

Kris_Knigga — Mon, 03 Oct 2011 20:47:59 GMT

Oh, my bad. This is talking about HBA utilization, I was talking about disk utilization. I would assume they are calculated the same, but I don't know.

What's odd, though, is that avserv is always reporting 1ms. sar is acting a bit goofy here, it seems.

Re: HP-UX 11.31, HBA usage 100% with low throughput and IOPS

cosmoecho — Tue, 29 Nov 2011 19:13:58 GMT

Hello

Did you solve your 100 utilization problem?

Along with average 7 Outstanding IO requests on each of 4 HBAs that looks a bit odd to me.

I will digg and if find anything I will post it here.

Thank you.

Re: HP-UX 11.31, HBA usage 100% with low throughput and IOPS

cosmoecho — Tue, 29 Nov 2011 19:14:54 GMT

7 ave Outstanding IO its on our system... with 100 utilization