Relationship between VA and LVM performance

[Tim Rotunda]

I have a single rp5470 connected to a VA-7100 on a single FC interface with a 256MB cache.

My VA has 1 redundancy group, 3 LUN???s made up of a total of 9, 36GB, 10k drives. When I run armperf I get read latencies mostly clustered in the 10-20ms range and write latencies in the 5-81ms range. The same armperf data also shows a Total I/O Throughput per second of 314 for a 6 day average with a peak of 571 and only a hand full of other Total I/O measurements above 500. My feeling is that the RAID is performing well.

On the server, I have vg10 with a single lvol configured that is made up of three PV???s, c6t0d0, d1 and d2 and sar shows the following for each PV:

Device %busy await
C6t0d0 78 73
C6t0d1 49 20
C6t0d2 31 40

Glance shows disk utilization hitting 100% much of the time also. My feeling is that there is a problem either in the measurement of this or it truly indicates a problem.

I think it is fair to say this system is pretty busy with I/O and there are a huge number of file on the system (50M+). I would say most of the I/O???s on this system are small to many files, although there are a few large files (1G+) that get hit frequently. The files are mostly flat indexed database files with a handful of PostgreSQL clusters.

A few questions, if I may, as I am trying to better understand the relationship between the VA performance and the performance of LVM.

Just how busy can a LVM device file get?

If I were to add 3 more LUN???s and was able to equalize the data across them all, increase RAID performance only to put more pressure on LVM?

Is increasing controller cache the key, if there is one.

Is there a problem I should be worried about?

How much more can the I/O take?

I am preparing to create a SAN and will be adding another controller to the VA and another FC HBA to the 5470, but also another rp5470 with dual FC HBA???s. Will the additional controller and HBA???s only add to the LVM pressure with the additional bandwidth?

Is there a tool that will help analyze this end to end, accurately?

Thanks for any information you can provide.

Tim

[Steven E. Protter]

First, December 2002 Patch bundle. There are lvm performance patches and other patches that can deal with these issues.

Second, search itrc for more lvm patches.

Third, try this little attached performance script. IT will collect data in a file and you can modify it to change the data collection period. It runs in the background.

It will help you.

Long run, consider a second fiber card and pvlinks to do a little load balancing and failover on the fiber card interfaces.

I think though you can totally deal with this issue by measuring performance and installing appropriate patches.

SEP

[Tim Rotunda]

So I assume you believe I would have a problem and that problem would be with LVM? That isn???t a hard sell for me, as that is what I have been eying. Just trying to collect as much information as I can.
Thanks,
Tim