rp5470: CPU & DISK Utilization

[Bill Hassell]

>> There is no time where both disk I/O and CPUs are both being used. Is this unexpected?

That's not possible to determine without knowledge of the application.The graph is a bit hard to correlate disk vs. CPU but it is certainly possible to max out CPU and also have high I/O rates. The big question is the I/O rate...400 is not very speedy. You need to look att Glance (not gpm) and check the disk queues and especially the service times. If the service times are slow (more than 40-60ms per I/O) and the queue is very high, there are probably other factors at work here. I would expect higher performance (I/O's per second).

For instance, the system overhead percentage looks like 40-60%, and that is not expected. If increasing the buffer cache helps a lot, and multiple copies of the program are being run at the same time, additional CPUs may help (add 2 more).

And as always, look at the processes themselves. Business Basic is not known for setting any speed records. Perhaps a close look at the code to minimize I/O and CPU would do the most good. If the BB app is reading/writing very small records, this would explain the high system overhead.

[Chris Vail]

I gotta concure with Bill on the Business Basic thing. We had a 32 processor V-class machine that was brought to its knees with a basic application. We were lucky to process 20,000 transactions a day, when our goal was a million. Putting the application BACK onto the Win2k system where it came from upped the transaction volume to 2 million/day.

The V-Class had memory and disk I/O at 100%, and disk I/O was at 20,000. Yet no work was getting done. The cue we looked at was the R column for vmstat. This is the size of the run queue--jobs waiting to be serviced by the CPU's. Standard tuning guides advise that a run Q higher than 1 is undesirable, but that is per-processor. We have 8-way N-class machines with a run Q less than 5, and the systems SCREAM they're so fast.

So I suggest that you run vmstat while running your app. How goes the Run Q?

Another is the w column, or processes waiting on I/O. If this number is very high (more than 2 in your case) then you have jobs waiting for the I/O channel to open so that the job can be submitted.

Lastly is the b column, which shows processes blocked waiting on I/O. Fundamentally, this means that you have too few disk channels and drives. By splitting your data up amongst several disks and filesystems, you can lower this number.

Post some vmstat numbers, as well as your glance readings.

Chris

[Brian M Rawlings]

As Bill says, without knowing more about the app(s), it's hard to say what is expected. Some observations:

1> If the app reads in a big file, and then spends time processing it (to decide where to write all the pieces), a big I/O spike (without much CPU), followed by a big CPU spike (without much I/O) would seem like a likely outcome. Especially if that's all that was going on.

2> If several (like, 3 or more) of these processes was running, unless they all start at the same time and basically run through the same sequence in the same times, I would expect the CPU and I/O spikes to mingle, balancing things out somewhat. If this is not happening, try staggering the "start times" of the three processes you run at the same time.

3> With LUNs, as with JBOD PVs, if you don't stripe with LVM when you create the LVs, the effect you'll get is of a series of buckets (the LUNs or PVs), in which new data all goes to the first one till it is clear full. Then the next one also gets used, and I/Os will go to both the first and second (but 3rd through Nth will be unused). This is why LVM striping (using the -i arg to lvcreate) is highly recommended, to spread I/Os "round robin" across a series of drives (LUNs or PVs). From your question, my guess is that your LVs were created without stripes, and that your data all fits into one 4GB LUN.

To see if your LVs are striped, see this thread where this is discussed:

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x69015fe8b250d71190080090279cd0f9,00.html

4> Lastly, I don't know if your Business Basic allows compiles, most do these days. I'm probably not telling you anything new, but BB apps that are compiled run faster and use less resources per process than in interpreted mode. Sorry to state the obvious, but one never knows...

Regards, --bmr

[Steven E. Protter]

I would say that you need to collect data.

I would do it with my attached sar script which can collect for any time period you wish to modify it for, in the background and see if further tweaking is required.

You can take the output send it to HP if you have a support contract, and they will make recommendations.

See attachment, and good luck.

SEP

[Brad Marks]

Gents,
Thanks for your detailed responses.
We have a k-420 with 3 100mz CPUs and not a lot of memory which supports 300+ users running Business Basic and, while things sometimes get busy, averages about 65% cpu / disk utilization. Our BB is very 'light', so I don't think that's the problem.
Looking at lvdisplay, it *appears* that my disks are neither mirrored nor striped (see command's output below), but the RAID card says that the disks are at RAID5 (striped and mirrored, right?). Also, when I look at the lights on the drives themselves when I/O is taking place, they are all active; this looks like striping to me... Our VAR pre-configured the system prior to delivery and I have a good deal of faith in them (maybe falsely?).

If my disks are not striped and mirrored, do I have to delete and re-create only the logical volumes? Will having online JFS (I finally got it!)make this procedure any easier?

Chris, looking at sar -qM output the runqueue size is 1.6 at worst and averages at 1.1. The R column of vmstat -n, during a high CPU cycle is 3 while B and W are 0. During a high I/O cycle R=1, B=3or4, W=0.

Here is lvdisplay output
--- Logical volumes ---
LV Name /dev/vg02/lvol1
VG Name /dev/vg02
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 40004
Current LE 10001
Allocated PE 10001
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation strict
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c4t0d1 10001 10001

--- Logical extents ---
LE PV1 PE1 Status 1
00000 /dev/dsk/c4t0d1 00000 current
00001 /dev/dsk/c4t0d1 00001 current
00002 /dev/dsk/c4t0d1 00002 current
00003 /dev/dsk/c4t0d1 00003 current
00004 /dev/dsk/c4t0d1 00004 current
00005 /dev/dsk/c4t0d1 00005 current
00006 /dev/dsk/c4t0d1 00006 current
00007 /dev/dsk/c4t0d1 00007 current
00008 /dev/dsk/c4t0d1 00008 current
00009 /dev/dsk/c4t0d1 00009 current
00010 /dev/dsk/c4t0d1 00010 current
00011 /dev/dsk/c4t0d1 00011 current
**** This portion of the display goes on an on *****

I gotta go for the weekend but will check back on Monday.

Have a good weekend, all.
Thanks,
Brad

[Brian M Rawlings]

Brad: provided that RAID was set up correctly on your 4Si raid card, the data is probably being striped across all your drives using RAID-5. It's not quite "mirrored and striped", rather it is "parity-protected and striped", same effect but worse performance for writes.

If you are seeing all the drive lights showing activity while your app is running the disks, this is very likely a correct setup as far as it goes. However, I'm having to guess at how the drives are set up:

1> You have 4x 36GB drives in the DS2300, set up as RAID-5. That means that the space of one drive (36GB) is gone, used for parity, leaving as useable space 3x36GB, or 108GB (roughly, real number is slightly different).

2> According to your irdisplay, you have SIX 10GB LUNS, and TWO 41GB LUNs. If I add all that up, it looks like 142GB. Hard to get that out of 108GB... but maybe that's raw space being reported, which I would expect to see around 144GB, so this is probably the case.

3> The 'bdf' you listed shows the LVOL mounted on /usr3 as about 41GB, so that looks like it's using one of the 41GB LUNs. That probably explains why you are seeing activity to only one PV in your Glance graphs.

4> But wait, I thought the data from 'irdisplay' was showing raw space, so how could an LVOL using that space (set up as RAID-5, hence losing 25% of the space) have 41GB? If the data from 'irdisplay' is useable space, you must have more than 4x 36GB drives. Maybe you can clear this up?

5> I don't know what the LVOLs mounted on /usr2 and /bbj are, although they are small enough to have been carved out of any LUN(s). If you post a 'vgdisplay -v', the PVs used in each LV will be listed, with sizes. This may or may not be needed, but I'm a little confused about the disks at this point.

6> In brief, LVM creates queues, buffers, and other internal I/O structures for a VG in proportion to how many PVs you include in the VG. It looks like you have set up one LUN per VG, so LVM will allocate the minimum amount of I/O queues (etc) allowed for that VG. To let the OS pump more data through a VG, you'd want to allocate several "PVs" (LUNs, in your case) to the VG, so lots of I/O queues & buffers will be allocated to that VG. This may be one of your performance constraints. I can't really tell you how much this matters, with the setup you have, but this can make a large difference with large EMC/HDS arrays, with scores or hundreds of LUNs in use...

Hope this helps some, although I left as many questions as I answered...

Regards, --bmr

[Brad Marks]

Sorry if I mislead anyone. Along with the 4*36 JBOD, I have 2*36 internal drives. My understanding is that vg00 is on them.

Unfortunately, I'm not going to be able to alter the current configuration of lvols, etc. at this time. I think the best thing to do is let this drop for now and, in a few weeks, re-post and continue.

THANK YOU all for your help!

Later,
Brad