Operating System - HP-UX
1748143 Members
3663 Online
108758 Solutions
New Discussion

Re: troubleshooting disk io

 
user001
Frequent Advisor

troubleshooting disk io

 

Hello,

 

i'm troubleshooting some performance issues with an application, so far i haven't found the problem and suspect its the application but did notice some unexpected disk IO problems.

 

It's an integrity service gaurd cluster running 11.31.

 

CPU and Load Average seem OK e.g. 40-50% idle cpu 1-2 load average.

 

using sar -d i've noticed the following:

the avwait spikes to 40 or so which outweighs the avserv now it doesn't go to 100% busy at the time and when this happens it happens for the san disks as well.

 

Now i've been using the application during these hits and it runs OK. My thougts are to get a performance hit on the application it would have to be prolonged high IO results not a quick spike? I've been running sar and by the time the next round of results come back the spike is gone.

 

the root disk is an lvm mirrored disk, the only thing on the root volume is the hpux OS and everything else has been moved to the san.

 

Why would i be seeing large avwait times on the root disk or is this normal?

 

also sar -b seems to be high on the write cache the average is around 40 but i've had it go lower than that. Althought i'm not sure its calculation averages properly.

I've read that its best to keep wcache above 70?

 

Any help and suggestions would be great.

 

Like i said this may be normal i'm just chasing performance issues so digging at everything at the moment.

 

Thank you.

 

 

3 REPLIES 3
Khairy
Esteemed Contributor

Re: troubleshooting disk io

>> Why would i be seeing large avwait times on the root disk or is this normal?

 

just a while guess, do you have large swap space in your root disks?

 

Could you post your vmstat and sar or glance output?

 

//

user001
Frequent Advisor

Re: troubleshooting disk io

Hi Thanks for replying.

 

Good point, i didn't even think of that.

 

Yes we do its 24GB.

 

Here are some stats, i've included Glance because i noticed its constatly blocked on Terminal and Semphore?

 

 

sar -d

19:55:01     c2t0d0   10.60    0.50      24     702    0.00    6.37
           c9t0d4    0.20    0.50       7     246    0.00    0.49
                      c7t0d4    0.20    0.50       7     245    0.00    0.59
                disk17    0.20    0.50      13     491    0.00    0.54
               disk18   15.40    0.50      28     721    0.00    9.17
              disk19   10.60    0.50      24     702    0.00    6.37
19:55:06   c2t0d0   28.20    7.36     103   10866   17.26   14.62
                 c9t0d4    2.60    0.54      30    2680    0.29    5.60
             c7t0d4    2.60    0.53      30    2659    0.61    5.68
             disk17    5.40    0.53      60    5339    0.45    5.64
                  disk18   38.00    7.49     115   10916   20.22   17.13
             disk19   28.20    7.36     103   10866   17.26   14.62
 

 

 

 

vmstat

        procs           memory                   page                              faults       cpu
    r     b     w      avm    free   re   at    pi   po    fr   de    sr     in     sy    cs  us sy id
   22     1     0  3326228  858410    0    0     0    0     0    0     0   5017 194314  9338  16 14 70
 
Glance (B)
IPC                 0.2      40.78      7.0  Cache         0.0       1.56      0.3
Job Control    0.1      23.32      4.0  CDROM IO      0.0       0.00      0.0
Message        4.8     815.50    140.1  Disk IO       0.0       0.00      0.0
Pipe                0.6      99.88     17.2  Graphics      0.0       0.00      0.0
RPC                0.0       0.00      0.0  Inode         0.0       0.00      0.0
Semaphore     0.0       0.00      0.0  IO            0.1      17.61      3.0
Sleep              81.2   13890.84   2386.7  LAN           0.0       0.00      0.0
Socket             1.5     248.71     42.7  NFS           0.0       0.00      0.0
Stream            0.1      17.24      3.0  Priority      0.0       8.45      1.5
Terminal          0.0       5.82      1.0  System        1.7     291.25     50.0
Other                9.6    1636.36    281.2  Virtual Mem   0.0       1.53      0.3


user001
Frequent Advisor

Re: troubleshooting disk io

 

OK so you can see i'm not paging out so probably not a huge concern about the swap at the moment.

 

I did find one interesting setting

 

the fcache_fb_policy as been set to 1 enabling it, i'm not sure why this has been set. I checked write_throttle and it hasn't been set.

 

anyone had anything to do with the fb_policy?