Performance problem HPUX 11 N-class server, JFS3.3, HP A5158A, Storagetek 9176

[Roy Tan]

Hi,

I have been having some problem with the IO through-put with the above combination of hardware/software.
I have only managed to achive about 12 MB per second using the comamnd "dd if=/dev/vg02/elvol1 if=/dev/null bs=64k count=1000".

Would like to hear from any existing users of the above combination of hardware/software on the actual through-put achived and what are the patches applied ?

According to StorageTek technical support, the problem that I encountered is a HPUX file system problem.

Thank you.

Regards,
roy

[harry d brown jr]

Roy,

12MB (MegaBytes - correct?) per sec for /dev/null isn't too bad. You shouldn't use one copying of data to /dev/null to measure throughput, you should use glance, and multiple instances of dd, until saturation is reached.

A lot of people forget the basics of the scheduler: if a process requests an IO it gets swapped out until the IO completes

And if you are doing measurement, then you have to be aware of all of the other processes running on the system.

live free or die
harry

[support_5]

Hi,

Where did you get the 12MB per second figure from? Glance? Or did you time how long the dd command took to run?

I would suggest the following to get better stats:

In a seperate hpterm window, startup glance.

Run multiple instances of the dd (6 or so should do) command (as already suggested) with the following:
dd if=/dev/vg02/rlvol1 of=/dev/null bs=64k

Once a few of these are running, have a look at glance and look in the IO By Disk option (press 'u') and you will see some of the disks with high KB/Sec values. You could tally up the KB/Sec values (the left column only in the KB/Sec section) of the main disks and you should be able to see how much throughput the system and disks can handle.

Alternatively,

instead of using a logical volume, you can use the disk instead. So replace /dev/vg00/rlovl1 with /dev/rdsk/c2t6d0 or whatever disk you want. Then look at glance and go to the IO By Disk option and select the disk you gave the dd commands and press S then the number next to the disk which we want to look at. This gives more detail on an individual disk.

Alternatively, use gpm instead of glance.

12 MB per second isn't terrible, but I've seen much higher, like 100 MB per second on my XP512.

Some notes. Is it connected via fibre? If so, then it probably has a redundant fibre link. You should therefore try to balance the load over the two links.

Hope this helps.

- Andy Gray

[Roy Tan]

Hi,

The IO rate that I obtained is through iostat and also the StorageTek Simplicity Software (the GUI configuration tools).
So this 12 MegaByte per second rate has to be correct.
I also see that the CPU% on the system went up to about 90% when the dd command is being run (there is no other processes running when the dd command is being run).

If a try to run 2 concurrent dd process, I get about 15 MegaBytes per second. With the faster process getting about 9 MB and the other slower one getting about 6 MB.

One interesting point I did not mention in my previous posting is that if I use dd against a file (dd if=/prdenv/file1 of=....)(on the same volume that I have ran the dd command), I get about 40MB per second (this rate is also captured by both iostat and Simplicity software). Any suggestion why there is such a dfference ?

I am using fibre link. So how do I balance the load over the two links ? Any additional software that I need to install ?

- Roy

[Kirby Collins]

I/O from a raw device generally gives lower sequential throughput than I/O through the filesystem. Raw I/O is not buffered and does not benefit from readahead by the filesystem. So the program issues a read, sits and waits for the disk(s) to do the I/O, then issues another read. Depending on how much readahead the disk or array hardware itself is doing, this can be as bad as one 64k block for every 10-20 milliseconds. The filesystem, on the other hand, will issue multiple requests in parallel if the I/O pattern is regular, so you can keep the disk(s) busy and working in parallel (if they are striped).

With vxfs3.3 it is also important to tune the filesystem for your access pattern (see the vxtunefs man page). The most important parameters for large block sequential I/O are:

read_nstream: number of blocks of readahead to issue (default 1). This should ideally
be large enough to cover all the devices in your filesystem. The exact number will depend on how your logical volume is constructed, and how the luns are arranged on your array, but I would start by increasing this to 4 or 8 (note that vxfs3.3 actually issues up to read_nstream*4 blocks of readahead).

read_prefio: This should be set to the most common block size you are using. In your case the default of 65536 is correct.

max_buf_data_size: This parameter controls the block size used for readahead. It should ideally match the record size you are using, but it only has two allowed values (the default of 8192 and 65536). For 64k records set this to 65536 (note if your real application uses smaller records, leave it at 8k).

discovered_direct_iosz: When a program does a read larger than this value, vxfs will automagically do direct, unbuffered I/O. Unfortunately, this means that you bypass the buffer cache and don't get any readahead, just as with raw I/O. For some programs this is a win (especially if the program itself is issuing multiple requests via async I/O), as it eliminates the copy to/from the buffer cache. For for normal, one read at a time, sequential I/O it is slower than buffered I/O unless the request size is really, really big. The default of 256k is much too small. You would not have seen this effect with your test case with 64k records, but it will be pretty obvious if you start playing with record sizes (your sequential I/O rate will drop dramatically at a 256K record size, then gradually climb back up to peak rates for very large records).

Hope this helps
Kirby Collins
HP Technical Server Performance Team

[Roy Tan]

Hi Kirby,

Your explaination makes sense to me.
If I perform a cp command on a file, I get about 40MB/s (the IO rate is same as what I get when I issued dd on a file). But when the application runs (SAS & DB2), we get about 12 to 13 MB/s, which is what I am seeing when I perform a dd on the logical volume.
So based on your explaination, I would think that my application some how issue IO requesting for transfer that is larger than the discovered_direct_iosz.
I would like to change this value but I am having some problem in using vxtunefs.
When I issued the following command to get the list of parameters :
vxtunefs -p /dev/vg00/lvol1
I get the error message :
"vxfs vxtunefs: Cannot get parameters for /prdenv/tmp: Function is not available".

How do I go about solving this problem.
Can someone please help.

Thank you.

- Roy

[Tom Geudens]

Hi,
You may already have the patches for vxtunefs ... but then again, you may not :-)
PHCO_22206
PHCO_22205
might help you with the vxtunefs problems.

Hope this helps,
Tom Geudens

[Kirby Collins]

you may need the advanced JFS optional product to use vxtunefs. Do you have OnlineJFS
installed? I think it may be included in
the Enterprise OE, if you have that.

Disabling discovered direct I/O may or may not help your database performance, depending on the pattern of accesses the database is making to the filesystem. If the database accesses its files sequentially, or does repeated accesses to the same parts of the file so that it can get hits in the buffer cache, then it is better to use the buffer cache.

If the database accesses data randomly within a file much larger than the buffer cache, then you are not going to get any benefit from readahead or any reuse of data in the buffer cache, and direct I/O may be the fastest, since you avoid the overhead to copy the data.

[Roy Tan]

Hi all,

Thanks for all the help given.
I have escalated the problem to HP support and apparently, the HBA was reporting some kind of IO timeout which is suspected to cause the slow performance.
I am now refering the problem to STK for resolution.

Kirby,
By the way, I do not have Onlne JFS installed but somehow the vxtunefs file exists on my standard JFS 3.3 installation. So I believe there maybe some patches I need to use the vxtunefs. In any case, I will probably not need vxtunefs now as the problem is suspected to be due to STK.

Thanks.

- Roy

[Roy Tan]

Hi Andy,

In my configuration, I have configured a RAID5 consisting of 9 disks in a single disk array.
What happen is that when my application starts running (SAS & DB2), I only see a maximum IO rate of about 13MB/s and this happens to be the same as what I get when I perform a dd on one of the logical volume on the RAID5 partition.
From your experience, what is the expected through-put for the dd command on the logical volume when I have a 9 disk RAID5 partition. Shouldn't I get more than 13MB/s ?


- Roy

[support_5]

Hi,

You might think that you would get more than 13 MB/s. In my configuration, we have 4 18GB disks set up in a raid 5 which produces about 10 LUNs (disks presented to the OS) each of which are about 7 GB in size. I have created a logical volume on one of those LUNS (remembering that the LUNS or disks presented to the OS, are actually striped across 4 physical disks in my case, 9 in yours). I created the file system and started to copy some large files to this new logical volume (once mounted) and iostat shows its writing at about 20 MB/s, sometimes spiking to 26 MB/s (but sometimes as low as 8 MB/s, but average about 20), reading is fairly similar. But if I run a dd if=/dev/vg_p01/lv_andy of=/dev/null bs=64k then iostat shows about 12MB/s on that disk. Seems dd is slower than file system IO as has already been discussed. A hint, notice that if you do a dd, the number of seeks per second reaches about 3000, whereas if you do a "cat * > /dev/null" the number of seeks is more like 300! That must make a big difference.

So your results are not that different to mine (unless mine is slow also and I just don't know it????!?!?!) So it would seem that the times coming from your setup are not too different from mine!.

Mind you, I am mildly suprised that the results are not faster. I would have expected more like 30-70 MB/s from each LUN, knowing that each lun is actually a raid 5 array of 4 disks (in your case it's 9 disks). This sort of performance is already being achieved by IDE raid controlers for your PC these days, and I would have thought that these SCSI controlers would have been faster. Check this out:
http://www4.tomshardware.com/storage/01q4/011023/raid-04.html#data_transfer_performance

But anyway, the stats are measured differently.

Interesting hey!

- Andy Gray

[Roy Tan]

Hi Andy,

If I were to copy files from 1 volume to another, I can achive the rate of 40MB/s. But somehow when the application (SAS & DB2) starts running, it can only achive 13 MB/s. Which is some 60% slower than a simple cp command.
This is something that I don't understand.

I did similar test on a Sun Solaris with STK 9176. The dd command on the volume group generates 50MB/s and the cp command also generates about 50MB/s.

Anyway, this case is referred to my local STK support for investiagtion.

Thanks for sharing the information.

- Roy

[support_5]

Hi,

Hmm, strange. Let us know how you go with it. C-Ya

- Andy Gray

[TOMAS BERNABEU]

Hello Roy!
I would wish to know as you solved the problem of disk Array. I have the same problem.

Tom

[Stan Sieler]

BTW, depending upon CPU,
disk drive, and interface,
one should definitely
be able to get
more than 12 MB/sec from
a raw disk device.

Our DiskPerf product (which
measures disk drive performace) on
HP-UX gets 36 MB/sec on
the internal HP drive
(HP 18.2GMAN3184MC), so
I know it's possible :)

sieler@allegro.com

[TOMAS BERNABEU]

Question :
xp512 with disc groups in raid 5.
4 FC of high performance.
8Gb memory cache.
lv's with striping of 4 discs and with block size of 256k.
dd with bs=256k gives 8Mb, the disc uses it to 12%.
5 dd bs=256k gives 80Mb/seg.
Oracle uses raw device with block size of 8k.
I need to improve much the performances.
I need that oracle writes many GB in just a short time.
ideas ?