topic Re: disk storage solution in Operating System - HP-UX

disk storage solution

Martha Mueller — Fri, 04 Aug 2000 12:11:35 GMT

On my N4000 with 3 440 MHz cpu's and 4 gig RAM, I am using jamaica subsystems (product A3312AZ) from Fast/Wide/Differential controller cards (A4800A). The data transfer speed is 20 MB/sec. These disks are mirrored and hold database data. I have two tempdb areas that are striped, not mirrored, and the data is on filesystem (vxfs), not raw partition, as the rest of the data is. Sybase suggestion so that the OS could help with some of the load. It did help the throughput. These tempdb areas are each on separate controllers, with no other data. They are striped across two and four disks, respectively. The disks are not busy, as shown by perfview's metric BYDSK_UTIL, but the request queue, BYDSK_REQUEST_QUEUE, is constantly around 20. Alan Riggs had mentioned something about the vxfs journal log having to be written at one location on the disk and therefore mechanical head movement may account for the long request queue. CPU usage is about 50%. Does anyone else have an idea, and does anyone have any suggestions to alleviate the bottleneck?

Thanks.

Martha

Re: disk storage solution

John Palmer — Fri, 04 Aug 2000 12:21:58 GMT

What vxfs mount options are you using? Options tmplog and nolog are available (see man mount_vxfs) .

Regards,

John

Re: disk storage solution

Tim Malnati — Fri, 04 Aug 2000 12:29:43 GMT

What you have pictured is about as far as you can accomplish with a Jamaica enclosure. More spindles in the stripe set could help, but I doubt you would gain much, particularly with the cost involved. Sounds like time to start considering an array where some hardware cache can help things. Avoid the model 12 though (it's slower if anything).

Re: disk storage solution

Martha Mueller — Fri, 04 Aug 2000 12:46:43 GMT

John,

/dev/volumegroup/logicalvolume /tmpdbdata vxfs rw,suid,largefiles,convosync=delay,mincache=tmpcache 0 2

I hope this displays properly, but it should be all on one line.

Thanks.

Martha

Re: disk storage solution

Martha Mueller — Fri, 04 Aug 2000 12:49:29 GMT

Tim,

Would you have any recommendations? I would probably like to stick with SCSI, due to the cost of Fibre Channel.

Thanks.

Martha

Re: disk storage solution

Tim Malnati — Fri, 04 Aug 2000 14:04:43 GMT

I'm an EMC guy, but I doubt you want spend that kind of money. They are just not very cost effective until you have significant storage needs. The same is true with the XP256 although the initial frame cost is less. Avoid the model 12h; it's slower than the Jamaica's. I've always liked the model 20 array in SCSI. But my thought is that the info that I have is more than a year old. Maybe HP has upgraded this array to improve the connection throughput or maybe HP has come out with something else to more effectively handle 80 mbs (ultra) drives. Your sales channel is probably the best bet.

Re: disk storage solution

Stefan Farrelly — Fri, 04 Aug 2000 14:46:26 GMT

What speed are the disks on your Jamaica ? They can vary by a huge amount. Use ioscan -fknCdisk to find out. The newer models have much larger cache and are considerably faster, here is a table of the different Jamica disks;

Disk_ProductID Size Speed(1)
========================
ST15150W 4Gb ~6.5 MB/s
ST34371W 4Gb ~8.5 MB/s
ST34572WC 4Gb ~10 MB/s
ST34573WC 4GB ~14 MB/s

ST19171 9GB ~10 MB/s
ST39173WC 9GB ~14.5 MB/s
ST39175LC 9GB ~18 MB/s
(1) using time dd if=/dev/rdsk/xxx

As you can see some of the newer models are massively faster than the older ones. We just replaced some of ours here with the 18 MB/s 9Gb models and the performance on our striped lvols increased wonderfully!

Re: disk storage solution

Martha Mueller — Fri, 04 Aug 2000 15:44:37 GMT

Stefan,

That is very important information. I have a mixture of disks in one of the tempdb areas: the slowest is ST34572 at 10 MB/s. Since this is a striped logical volume, can I assume that the entire logical volume can only write at this slowest disk's speed?

Re: disk storage solution

Stefan Farrelly — Fri, 04 Aug 2000 15:50:23 GMT

Hi Martha,

yes, indeed, any striped lvol will be contrained by the slowest disk in the stripeset. Try using pvmove online to move any high usage lvols to the fastest disks.

If you have a good HP VAR they may swap your disks for the faster ones at a not too expensive price because they can reuse yours in part exchange. Its certainly cheaper than buying and configuring a new disk subsystem. Also whenever we lose a Jamaica disk I always ask for the fastest model as a replacement. Has worked so far and its free!

Cheers,

Stefan

Re: disk storage solution

Martha Mueller — Fri, 04 Aug 2000 16:01:40 GMT

Stefan,

This is very enlightening. Would this explain the phenomenon of the long request queue but the disk not busy? Or do I need to look furthur for that answer?

Thanks.

Martha

Re: disk storage solution

Alan Riggs — Fri, 04 Aug 2000 18:15:41 GMT

Hi Martha.

The performance drop from the slowest disk in the volume will affect spin time and data transfer, so it might be a contributing factor. I would be surpised, though, if it accounted for the full problem. I have a couple of questions:

1) do all disks in the 4 disk stripe show similar UTIL/QUEUE pattern?
2) does sar -d report a similar uutilization pattern (sar queries different structures than the midaemon).
3) What are the BYDSK_AVG_SERVICE_TIMEs for the disks? Is there much divergence between the 4 disks?
4) What is the BYDSK_PHYS_IO_RATE for each disk?

Unfortunately, there are seldom quick and easy anserw to these type of performance issues. But the aboe wuestions might help pin something down.

BTW: please say hello to Sandy, Bob, et al fpr me. I hope you are all doing well.

Re: disk storage solution

Martha Mueller — Fri, 04 Aug 2000 18:46:55 GMT

Hi, Alan, nice to hear from you.

1. They are identical.

2. sar -d shows patterns that the disks activities are similar to each other, but the patterns are not necessarily similar to those shown by perfview. The average queue length on every disk, not just the ones in question, is 0.50 from sar. The percent busy for all four disks shown by sar was lower than that shown by perfview, but I don't have a good sar sample...I just ran it a few minutes.

3. BYDSK_AVG_SERVICE times as shown by glanceplus are about 2.2 msec, but, again, I don't have a long collection time. This metric isn't available from perfview. The four disks are within 0.5 msec of each other.

4. BYDSK_PHYS_IO_RATE is averaging around 4 requests per second, with gusts up to 25.

Re: disk storage solution

Alan Riggs — Fri, 04 Aug 2000 18:58:52 GMT

hmmm . . . what about cache hit ratio for reads and writes?

Re: disk storage solution

Martha Mueller — Fri, 04 Aug 2000 19:11:11 GMT

sar -b shows %rcache around 99 and %wcache around 90.

Re: disk storage solution

Alan Riggs — Fri, 04 Aug 2000 19:22:16 GMT

See, I told you this wouldn't be simple. I am a bit swamped ATM, but I will do some poking through a reference or two. Probably won't have anything meaningful before Monday -- maybe someone else can come up with something quicker. There are some smart folks hanging around here some days.

Re: disk storage solution

Martha Mueller — Fri, 04 Aug 2000 19:23:54 GMT

I agree with both points...Monday and the VERY smart people out there that are generously sharing knowledge. I have been spending most of the week just going through the answers on this forum and jotting down notes.

Re: disk storage solution

Dragan Krnic — Sun, 06 Aug 2000 09:56:17 GMT

I wouldt like to comment in general what I observed about disks under hp-ux:

RAIDs are nonsense. They are supposed to reduce the number of disks, like disks are
expensive. Most often the cost of the RAID controller is even higher than the value of
disks it was supposed to save. In a recent offer for 140 GB net, a model 12 with 5
disks, 36 GB each, spares 3 disks at level 5 versus straight mirroring (which can be
accomplished by Mirror/UX). The saving of 3 disks ($7,150) was more than offset by
the cost of the controller ($13,400). However, for this higher price one gets significantly
lower performance.

The RAID controllers perforce introduce moderate to severe latencies and are never
nearly as fast as Mirror/UX.

My advice is, keep away from RAIDs in general. Just a bunch of mirrored fastest disks available, professionally installed in a decently redundant enclosure (multiple power
supplies, multiple fans, multiple SCSI cabling), saves a lot of dough and increases the
performance. For legacy F/W differential systems my supplier uses an SE/Diff adapter
(a passive device) which converts off-the-shelf UW and LVD drives to F/W differential
(HP way). My most recent enclosure was 9 times 73 GB cyclically stripe-mirrored for
a net capacity of 315 GB at the total price of $16,000. The new disks just fly.

Unfortunately, many system bottlenecks are psychological. IT managers tend to build
up their status based on the $$$ invested in the hardware they manage with scant
concern for performance or sound cost/benefit analysis.

Re: disk storage solution

Dragan Krnic — Sun, 06 Aug 2000 10:02:22 GMT

My previous posting was garbled by the editor. 2nd take:

I wouldt like to comment in general what I observed about disks under hp-ux:

RAIDs are nonsense. They are supposed to reduce the number of disks, like disks are expensive. Most often the cost of the RAID controller is even higher than the value of disks it was supposed to save. In a recent offer for 140 GB net, a model 12 with 5 disks, 36 GB each, spares 3 disks at level 5 versus straight mirroring (which can be accomplished by Mirror/UX). The saving of 3 disks ($7,150) was more than offset by the cost of the controller ($13,400). However, for this higher price one gets significantly lower performance.

The RAID controllers perforce introduce moderate to severe latencies and are never
nearly as fast as Mirror/UX.

My advice is, keep away from RAIDs in general. Just a bunch of mirrored fastest disks available, professionally installed in a decently redundant enclosure (multiple power supplies, multiple fans, multiple SCSI cabling), saves a lot of dough and increases the performance. For legacy F/W differential systems my supplier uses an SE/Diff adapter (a passive device) which converts off-the-shelf UW and LVD drives to F/W differential (HP way). My most recent enclosure was 9 times 73 GB cyclically stripe-mirrored for a net capacity of 315 GB at the total price of $16,000. The new disks just fly.

Unfortunately, many system bottlenecks are psychological. IT managers tend to build up their status based on the $$$ invested in the hardware they manage with scant concern for performance or sound cost/benefit analysis.

Re: disk storage solution

Stefan Farrelly — Mon, 07 Aug 2000 06:47:53 GMT

Hi Martha,

a lot has passed since I went home Friday. Have you made any progress ? My only comment is I think the different speed disks will make a big difference. I have seen this happen before. Next step is to move your cricitcal lvols to a stripe set all on the same speed and fastest disks then see how that improves things.

Im afraid I have to disagree with Dragan. Here at HP weve got our Nike Raid disks performing much faster than Jamaicas. The hundreds of MB of cache makes a big difference - especially to write performance. For the extra money you get dual pathing so protecting against controller failure and RAID so more space for your money. We do have slightly more failures, but this is temperature related, keep em cool and their failure rate is same as any other hardware.

Re: disk storage solution

Anthony deRito — Mon, 07 Aug 2000 11:52:35 GMT

Hello Martha, just to cover your bases, have you tried to approach this problem from a different angle? Take a look at the process acually doing all the work. Use Glance and check out the wait states that the process is consuming. Check to see if the process is able to work harder than it is. Look for things like priority, IPC time delays, local network trafic. Is your database using the local UNIX domain protocol or the high overhead TCP/IP to do its local database access? (This is a common problem with local database access.)

It seems your buffer cache hit rates are very good but don't let this fool you. I've seen stranger things.

You say CPU usage is about 50%. If the %wio parameter from sar -u is not very high as you would expect from an I/O bottleneck, then SOMETHING has to be consuming the resources. (I know, that was a general statement.) This approach will also be valid if the time spent in system mode is also higher than expected.

Tony