Internal disk so busy it locks the rx2880-i2 hpux11.3 system

[Steve Post]

I found my error in my test.  With that fixed, I was able to compare performance by running the same pile of jobs over and over.

Here is how it went.

I ran  scsimgr set_attr -D /dev/rdisk/disk2 -q max_q_depth=8

queue of 8
00:00:00   device   %busy   avque   r+w/s  blks/s  avwait  avserv
10:18:00    disk2    4.00    0.51      22     566    0.07    3.36
10:20:17    disk2   87.61 6552.51     481   26045 12251.35   14.13
            disk2  100.00   43.58    3950  150000   48.60    8.49
    (two entries at 10:20 is not a typo).
10:21:00    disk2  100.00 3884.44     631   25356 3213.06   12.67
10:22:00    disk2   80.09  893.39     619   21720 5262.29    8.67
10:23:00    disk2    6.57    1.44      28     635    1.62    5.47

queue of 32
00:00:00   device   %busy   avque   r+w/s  blks/s  avwait  avserv
12:52:01    disk2    2.80    0.50      10     248    0.00    5.64
12:53:01    disk2    3.83    0.50      28     617    0.00    3.06
12:54:01    disk2   95.99 1915.28    1149   73149 1452.83   23.27
12:55:01    disk2   39.13 1488.82     361   26574 1085.59   19.07
12:56:01    disk2   10.14    1.84      61    1107    2.45    7.64
12:57:01    disk2    8.90    0.50      60    1135    0.00    2.98

queue of 64
00:00:00   device   %busy   avque   r+w/s  blks/s  avwait  avserv
13:06:00    disk2    8.97    0.69      48     877    0.35   10.59
13:07:07    disk2   26.95 2208.50     291   18046 1283.96   47.42
13:08:03    disk2  100.00 5702.70    1605  107088 3470.67   39.31
13:09:00    disk2   34.42    0.53     282    9685    4.06    3.86
13:10:00    disk2    8.58    0.58      49     764    0.17   12.20

when I was watching swapinfo.

NOT BUSY......
             Mb      Mb      Mb   PCT  START/      Mb
TYPE      AVAIL    USED    FREE  USED   LIMIT RESERVE  PRI  NAME
dev        8192     139    8053    2%       0       -    1  /dev/vg00/lvol2
dev       16384     141   16243    1%       0       -    1  /dev/vg00/lvol9
dev        8192       0    8192    0%       0       -    2  /dev/vg00/lvol14
reserve       -     632    -632
memory    31069   26199    4870   84%
total     63837   27111   36726   42%       -       0    -
BUSY......
             Mb      Mb      Mb   PCT  START/      Mb
TYPE      AVAIL    USED    FREE  USED   LIMIT RESERVE  PRI  NAME
dev        8192    1247    6945   15%       0       -    1  /dev/vg00/lvol2
dev       16384    1258   15126    8%       0       -    1  /dev/vg00/lvol9
dev        8192       0    8192    0%       0       -    2  /dev/vg00/lvol14
reserve       -    4490   -4490
memory    31069   26124    4945   84%
total     63837   33119   30718   52%       -       0    -
The less /dev/vg00/lvol2 was in use, the faster every was running.

 So my conclusion would be to set the Max Queue depth to 32. 

I would like to avoid swapping out also.  Perhaps throttling down the number of times my java job can run at once will help.   I can imagine two guys trying to get through the same door at the same time.  I can also imagine someone's Mom saying "take HUMAN bites."

Is it a good idea to make a swap space on a disk array? 

[Dennis Handly]

>Is it a good idea to make a swap space on a disk array?

A better idea would be to add more RAM.  ;-)

[Steve Post]

Maybe I am reading it wrong,(really?), but it looks like I have plenty of RAM.  So why is it swapping out some stuff that does not need swapping out?  My guess it that it is swapping out because the disk is getting pummelled with data.  And the "swap" area is also on the same pummelled disk. 

So it all runs like crap because it is sort of in an infinite loop. 

At least changing the max_q_depth to 32 helped a lot.  Avoiding my scenario will go further.  

I don't like Java.   

[Dennis Handly]

>it looks like I have plenty of RAM. 

I see some device swap being used:

dev 8192 1247 6945 15%

dev 16384 1258 15126 8%

>So why is it swapping out some stuff that does not need swapping out?

It could be paging out mapped files and shared memory, not just data pages.

>And the "swap" area is also on the same pummelled disk.

That won't help.

[Steve Post]

I think I am done.   I can just about eliminate my problem with a few things.....

Wait...  I'll step back and explain in my twisted, unofficial examination. (like if I land a bowling ball on my foot I can deduce that is a BAD thing).

When I run 17 of these lousy java jobs at the same time, it causes the internal disk to get really busy.  If I run 8 instead?  Hardly any impact at all.  But let's go back to running 17 of the jobs. 

If I use scsimgr to change the max_q_depth from 8 to 32? it helps immensely.  I also note that paging out is lot bigger this way.  And the memory is at 45% (a bit less that 100 I must say).   So it appears that the disk requests are in a queue that drops to swapspace.  Because I have pseudo swap on, most of these "thingies, doo-hickies whatchamacallits" are in actual memory.  I also tried a queue depth of 64.  It was not as good. 

I also know that when I move more and more of the lousy java job to external disk, things run faster.  Why?  Because the external disk is a Vnx EMC disk array that works fast. 

So to summarize:  

Increase max_q_depth on the internal disk from 8 to 32 and stop obsessing over memory.

Move what the script uses to external disks.

Do not run too many of this poor script at once.

hmmmm......  Bowling Balls

Steve