striping performance issue

[Steve Lewis]

L2000 4x360 4gb RAM, HP-UX 11.00 patched to GR11 Sept 2000.

I am getting high busy% and poor block throughput on sar -d for 40 striped logical volumes (its 32 bit informix which has a disk chunk size limit of 2Gb and raw i/o direct to the LVs). The block throughput is low, but the avwait is low too, although the avserv is higher than normal. The request queue rarely gets above 0.6

sar -u shows the system is spending 50% of its time waiting for i/o, yet the 40 informix AIO vps are not over-busy, showing an average of 1.1 ios per wake-up. There is plenty of cpu available and it is not paging memory. Informix also reports about 575 IOs per second.

The LVM config is simple - just a test system with no mirroring and everything striped over 4 disks directly connected to 2 FC (tachyon XL2) controllers, driver version A6795A B.11.00.10. The disks are all 10k rpm 76.3Gb. Its a bit of an experiment in striping versus trying to be clever with table placement.

Glance also shows the disks to be busy 2 thirds of the time (yellow warning) with low throughput. Glance indicates about 200 reads and 400 writes per second with this application, which populates database tables.

Paradoxically when backing up the database I get about 40000 blocks per second, which is fine 10 times as much.

How can I increase the thoughput? The database is well tuned, using the combined experience of 4 DBAs, its just the disks which are slow (cf. sar -u attached). Obviously more disks would help, as would a fancy controller, but I dont have those :-(

Is there such a thing as controller contention for these FCXL2 controllers?

Is there any SCSI tuning that I could do or diagnostics to run? We already use measureware and glance to give i/o by LV, but it just confirms what sar shows.

See attachment.

[Chris Vail]

First of all, I really think you should go to a 64bit app and O/S. This lets you get the full throughput of your I/O channels. All by itself, this should reduce the number of processes waiting for I/O. This is because the system will move more data to/from the I/O
channel per clock cycle. The processes will get the data they're starved for, and complete faster.

Next, you really should consider getting faster CPU's--for the same reason. The CPU's will process the jobs faster. Again, the processes waiting on I/O will go down because the processes themselves will terminate more quickly. The 440Mhz processors are cheap these days, and 550Mhz processors are coming down in price very quickly.

The reason that the backups are so much faster than operations is that the backups are doing sequential reads. This lets the disk subsystems fill and empty cache very quickly. Its the random reads/writes that get slow on all systems. A 10x difference in speed is unsurprising.


Chris

[Steve Lewis]

Thanks Chris.

I have also just found a patch - PHSS_26798.

The system is generally 2 years behind in patch levels, so that may affect things a bit.

[Stefan Farrelly]

Well, your disks arent slow because youve stated your backup gets around 40000 blocks per second. Ive tested the internal system disks on some of our servers and the older ones are around 30k/b/s and the newer ones around 70k/b/s. And our EMC disks are around 50k/b/s so your 40k is in the right ballpark, speed wise.

The question im left with is why does the backup get 40k/b/s and yet during normal application running you dont get anywhere nears this. It must be your application (database). Its definitely waiting on i/o (wio = 52 is completely io bound) but with only 800/b/s it must be doing lots of random requests (ie. not sequential) or only asking for small amounts of data whereas a backup is completely sequential and asks for as much data as it can get hence the much faster throughput.

I think you need to look more into database tuning (read_ahead, increasing block size for reads, making the layout of your database and tables more sequential etc.) I dont think you can do much from the hp-ux point of view.

[Sridhar Bhaskarla]

Hi Steve,

You have 40 logical volumes striped over 4 disks?

If I read it correctly, you are going to perform very poor. Because accessing every LV is going to impact the other 39 LVs.

I am not surprized. Striping is to be very carefully done and is dependent on how you access the data and the corresponding stripe size. It will be beneficial if you have many disks and have the flexibility to group them.

-Sri

[Steve Lewis]

Oh dear, I was afraid someone would suggest tuning. You are probably right, partly, but I don't think there is much that can be done in this case, due to the application.
The application does do a lot of random access i/o, getting data from all over the place to build up a large unload file. By its very nature, it clobbers the database cache/buffers with data that isn't likely to be needed again. I mitigated this slightly by setting some small static data tables frequently used in joins to be memory-resident. The database is too big to fit entirely into memory at 80Gb.
The disk service times appear to show an element of head-flapping. You are confirming my fears.

It is actually 64 bit informix, but I am fairly sure that the i/o subsystem is still 32 bit, right up until DS version 9.4 which isn't out yet, but has 64 bit chunks.

Yes I agree it is too few disks for too many LVs. I did say it was an experiment but thanks for the feedback. The production system that this database was taken from doesn't use disks, but FC Luns, raid 1/0 so I don't get this problem with the live system.

[Dietmar Konermann]

Hmm, looking at the sar -d output the DB seems to perform most of all IOs in 4KB requests. Are you sure that your asyncdsk driver (/dev/async) is properly configured and in used?

Best regards...
Dietmar.

[Steve Lewis]

Well spotted Dietmar.

You are correct that this instance is not using KAIO. This is because it is already being used by another instance. Unfortunately informix cannot cope with KAIO (/dev/async) on more than one instance on a machine, so instead we are using 40 AIO VPs in this instance. As I said before, the application does lots of random reads in small blocks.

Actually some informix DBAs reckon that KAIO doesn't give much of a performance increase on HP, due to the HP-UX shared memory implementation and limitations in the PA-RISC architecture. I think that KAIO saves on cpu, memory, nfiles and processes, so its a good thing to use where you can. Agreed.

[Decio Miname]

Probably you've already checked this, but let's try...
Since you have several LV's on a few disks, you should consider the physical position of each LV inside each disk. I mean, if most of the 4K requests alternate between most-inner tracks and most-outer tracks too much, you will spend most of your time seeking tracks instead of actually transferring data.
If you suspect that, check which LV's are most accessed while you are experiencing lots of WIO and low transfer rates. Give special attention to the mostly accessed LV's by dividing its access equally through the 4 disks and placing them in the most "center" part of the disks. Then, proceed to the second-most-accessed LV's with the same distribution/placement idea. The fewest-accessed disks can be place anywhere.
That applies more for JBOD's than to EMC storage since inbound storage cache makes a lot of difference - for EMC, these calculations is very different.