Operating System - HP-UX
1753362 Members
5312 Online
108792 Solutions
New Discussion

HP-UX 11.31, HBA usage 100% with low througput and IOPS

 
Francis Noël
Regular Advisor

HP-UX 11.31, HBA usage 100% with low througput and IOPS

Hello new Forum

I realize my question will be start ou vague but here we go.

Got an Rx7640.
Runs a mostly patched Feb 2007 release of 11.31
is hooked up to an SSD-based storage via  HP AB378-60101 4Gb 's.
Single LUN cooked though LVM, formatted with vxfs 6.

Problem : HBA usage shoots up to 100% while througput and IOPs stay ridiculously low.

Example :

 

sar -H 1 10

HP-UX hp-prod1 B.11.31 U ia64    09/20/11

14:10:22     ctlr   util t-put  IO/s   r/s   w/s   read  write avque avwait avserv
                   %age   MB/s   num   num   num   MB/s   MB/s   num   msec   msec
14:10:23     fcd0    99   5.33    89     1    88   0.00   5.33   161      0      1
             fcd1   100   5.67    97     0    97   0.00   5.67   117      0      1
14:10:24     fcd0    99   5.55    90     0    90   0.00   5.55   161      0      1
             fcd1    99   5.47    89     0    89   0.00   5.47   117      0      1
14:10:25     fcd0   100   5.26    85     0    85   0.00   5.26   161      0      1
             fcd1   100   5.32    85     0    85   0.00   5.32   117      0      1
14:10:26     fcd0    99   5.28    87     0    87   0.00   5.28   161      0      1
             fcd1    99   5.44    88     0    88   0.00   5.44   117      0      1
14:10:27     fcd0   100   5.29    85     0    85   0.00   5.29   161      0      1
             fcd1   100   5.60    92     0    92   0.00   5.60   117      0      1
14:10:28     fcd0   100   5.21    85     0    85   0.00   5.21   161      0      1
             fcd1   100   5.54    90     0    90   0.00   5.54   117      0      1
14:10:29     fcd0   100   5.47    90     0    90   0.00   5.47   161      0      1
             fcd1   100   5.55    94     0    94   0.00   5.55   117      0      1
14:10:30     fcd0    99   5.38    86     0    86   0.00   5.38   161      0      1
             fcd1   100   5.49    90     0    90   0.00   5.49   117      0      1
14:10:31     fcd0   100   5.40    90     2    88   0.00   5.40   161      0      1
             fcd1   100   5.49    89     0    89   0.00   5.49   117      0      1
14:10:32     fcd0   100   5.38    88     0    88   0.00   5.38   161      0      1
             fcd1   100   5.48    92     0    92   0.00   5.48   117      0      1

Average      fcd0   100   5.35    88     0    87   0.00   5.35   161      0      1
Average      fcd1   100   5.50    91     0    91   0.00   5.50   117      0      1

 
Data is being restored from fiberchannel tapes.
I need to find out where the pipes are jamming and I'm running out of ideas.
This is my first time working on this box so no prior knowledge as to if this used to work or not.

 

Anyone ever seen something similar ?

7 REPLIES 7

Re: HP-UX 11.31, HBA usage 100% with low througput and IOPS

You need to understand what "utilization" means in this context. It doesn't mean that 100% of the bandwidth or capability of the card is being used, it simply means that "during the interval, the HBA was carrying out IO operations 100% of the time". That doesn't mean its not capable of more... I would look elsewhere if I were you...

 

Incidentally if the data is being restored from FC tapes, where are the reads? I only see write below, or have you removed some additional FC controller info from the output below?


I am an HPE Employee
Accept or Kudo
Francis Noël
Regular Advisor

Re: HP-UX 11.31, HBA usage 100% with low througput and IOPS

Hello Duncan and thank you for stepping up.

 

Indeed I am looking elsewhere.  The SAN vendor is investigating his box and it looks like a firmware patch is needed to correct an "intensive write" bug...  This would mesh well with your explanation of 100% usage if the few IOs we do manage to get through to this thing take a long time to complete.  Is there such a metric as IO completion time I could look at to confirm the hypothesis ?  The server does have (and old version of) Glance installed.

 

As for the absence of reads I have no explanation at this point.  I can only assume that the tapes are twiddling their proverbial thumbs and the 10 seconds I caputred was fed by our copious Netbackup Buffers.  I shall try a longer sample.

 

Thank you very much for your input Duncan, your comments did enhance my understanding of the situation.

Dennis Handly
Acclaimed Contributor

Re: HP-UX 11.31, HBA usage 100% with low throughput and IOPS

>You need to understand what "utilization" means in this context. It doesn't mean that 100% of the bandwidth or capability of the card is being used, it simply means that "during the interval, the HBA was carrying out I/O operations 100% of the time".

 

Hmm, this seems like a strange definition of  "utilization".  I would think it should include "waiting" or "idle" in the mix, the same as how you measure CPU usage.

Kris_Knigga
Trusted Contributor

Re: HP-UX 11.31, HBA usage 100% with low throughput and IOPS

I believe the utilization percentage is calculated with Little's Law (http://bnrg.eecs.berkeley.edu/~randy/Courses/CS252.S96/Lecture21.pdf - page 10 specifically).  Basically utilization equals the average number of requests per second multipied by the average time needed to complete a request.


Kris Knigga
Kris_Knigga
Trusted Contributor

Re: HP-UX 11.31, HBA usage 100% with low throughput and IOPS

Oh, my bad. This is talking about HBA utilization, I was talking about disk utilization. I would assume they are calculated the same, but I don't know.

What's odd, though, is that avserv is always reporting 1ms. sar is acting a bit goofy here, it seems.

Kris Knigga
cosmoecho
New Member

Re: HP-UX 11.31, HBA usage 100% with low throughput and IOPS

Hello

 

Did you solve your 100 utilization problem? 

Along with average 7 Outstanding IO requests on each of 4 HBAs that looks a bit odd to me.

I will digg and if find anything I will post it here.

 

Thank you.

cosmoecho
New Member

Re: HP-UX 11.31, HBA usage 100% with low throughput and IOPS

7 ave Outstanding IO its on our system... with 100 utilization