SLOW Disk I/O on DL380 G3

[Zack Menendez_1]

Hi, We just set up 3 DL380 G3 Proliants with 3 146 GB SCSI disks in RAID 5 (1 logical disk of 292 GB) on the SmartArray 5i controller. We have been VERY disappointed in disk I/O -- a recent benchmark on a defragmented, nearly empty (logical) drive gave us read speeds of 30 MB/s sustained read and 7 MB/s sustained write! I cannot change the cache ratio to anything other than 100% read / 0% write in the ACU. Why would this be so slow? We are running Windows 2003 Server.

[Doug de Werd · Accepted solution]

There are several reasons for this.

First, the basic 5i Plus (which is what is on the DL380 G3) controller does not have a Battery Backed Write Cache (BBWC). This is one reason why your writes are so slow, and also why you cannot set the cache to anything but 100% reads. You can add a BBWC, which will improve your performance. See http://h18004.www1.hp.com/products/servers/proliantstorage/arraycontrollers/smartarray5iplus/index.html for more info on the 5i+. What you need is the Battery Backed Write Cache Enabler Option Kit (p/n 255514-B21).

Another reason that you are seeing slow performance is that you only have 3 drives in a RAID 5 configuration. In general, the more drives you have in a RAID 5, the better the performance. So adding a couple of drives to it would definitely help.

Another reason is that the integrated 5i controller runs on a slower internal PCI bus (see http://h18004.www1.hp.com/products/servers/proliantstorage/arraycontrollers/feature.html). Converting to a standard PCI Smart Controller (like a Smart 64x or 640x) would also help. See this for more info http://h18004.www1.hp.com/products/servers/proliantstorage/arraycontrollers/index.html

There's a bunch of good white papers here regarding Smart Controlelrs (advantages of BBWC, comparisons, etc.)

http://h18004.www1.hp.com/products/servers/proliantstorage/arraycontrollers/documentation.html

Hope this helps.

Thanks,
Doug

[Zack Menendez_1]

Doug, thanks for the reply! I am looking into the battery backed write cache. As a follow-up question, are there any array controllers that are compatible with the Hot Plug PCI-X slots on the machine? We are already using the non-Hot Plug slot. Is there a list somewhere of Hot Plug compatible PCI cards? At this point I'm trying to determine if it would be better (read: less expensive) to get new controller cards or the cache module for the existing controller - I've already broken the bank on these servers as it is! Thanks!

[JohnWRuffo]

Zack:
The BBWCE will be a great deal less than another RAID controller. It will give you the functionality you need for around $200.00 (depending on your vendor)

You need Option part Number: 255514-B21

[Zack Menendez_1]

Thanks, John, for the reply! I am obviously leaning toward the more cost effective BBWC solution. Now here's my question: will I only see performance enhancements up to the size of the cache? For example, suppose I have the cache set to 50% read / 50 % write, making 32 MB cache in either direction. If I write a 30 MB file, I'd expect it to write fast since it's truly writing to cache. But if I write a 100 MB or 1 GB file, will it still be slow because only the first 32 MB is written to cache while the rest is bottlenecked? Is this the case or does it not work like that? Thanks!

[JohnWRuffo]

The cache does not work exactly like that, but provides for "steaming" data on and off the RAID. You do not load data files directly into the cache per se, you just provide a buffer of sorts to allow the rest of the I/O system to streamline the movement.

That said, the more cache you provide a read or write path, the better that performance will be. The cache accelerates the opperation. If you allocate a 50% / 50% cache ratio then each path will benefit equally (which HP sets as default). You can then decide if your application requires another mix.

The BBWCE is a great option for the 5i controller. It really is a must and it can be had for rather cheap!

G'luck! -john

[Zack Menendez_1]

I installed the BBWCE yesterday and unfortunately, it did not make one bit of difference. After the batteries charged, I tried every variation of the cache settings for read/write from 100/0 to 0/100 and everything inbetween. Not one bit of difference -- well, for our problem. The benchmarks look better for write, of course, but software isn't running any faster. Please let me tell you more about the issue.

We are a software development company. The software we're using on these servers does pretty simple operations: reads from some files on disk (1 at a time), does some operations in memory, writes to disk. No external dependences like networking or a database or even the registry or anything. Imagine my astonishment last week when a user said that the process that he runs on his laptop, where it takes about 3 hours, was run on a new DL380 G3 where it took 9 hours! Of course I didn't believe him, but sure enough, it was true. Identical setup in both places.

I ran some benchmarks and assumed the problem was the disk write speed since that was reportedly slow. Even with the write speed better, performance did not improve even by 1 second! Again, all these little programs are doing is reading from disk, doing a little thinking, and writing to disk.

Benchmarks look fine for CPU, memory, and I'd even say disk with the cache enabled. I've tried turning hyperthreading on/off - no difference. 3 disks in RAID 5, Windows 2003. Latest BIOS / drivers.

Can anyone please help!? I'm getting a lot of heat for spending all this money on new servers that perform slower than laptops! But the strange thing is, I can't even pinpoint it to what the problem might be! Please help. Thank you.

[John Clark_12]

Jack,

We are having the same problems with our 380. However, ours seems to run fine locally (moving/ copying files), it is when we try to write files from a network client that the I/O slows to a crawl.

Reading/copying from the server to a client seems comparable to peer-to-peer client data transfers, but when we try to write files the transfer rate is unaccepatble.

We have had everything checked and are unable to resolve the issue. Our reseller's techs put the system on one of their test computers via a xover cable and had the same problems, so we know that the server is the root cause and not the network or clients, however they are unable to tell us why.

All diagnostics report no problems. We even tried putting in a different NIC, since it seemed that the issue only occurred during network writes, and that made no difference.

The transfer rates we are seeing are 400mb in 3.5 minutes down, and 400mb in 1+hours up. We cannot locate the bottleneck.

This brand new server is virtually worthless to us. Have you had any luck yet?

Thanks.

[Zack Menendez_1]

John - We never really came to any satisfactory resolution on this. I bought the BBWCE and it sped up write somewhat, but contrary to the advice I was given above, indeed it *was* slower on sustained writes than on burst writes.

My conclusion on that goes like this: If the write cache sped up writing without direct relation to file size, I should see no difference in writing a file that's 20 MB vs. one that's 2 GB. I should expect a write time that has roughly linear performance for the duration of the write (in this example, the 2 GB file should take about 100 times longer to write than the 20 MB file). I received exactly the performance that I had suspected and asked about. In the example above, it would take much more than 100 X longer to write the 2 GB file in a file copy because the write cache gets used up quickly and you're back to the limits of the hardware as if you had no cache.

With our servers, LAN transfers actually work a bit faster than local copies (copying a file from the disk to itself). I would guess this is because the same disk (well, set of disks, really) is doing the reading AND the writing. The read speed on these servers is good, it's just the writes that stink. I will say that if you're using a program that is not writing large files all at once, then the BBWCE does enhance write performance. File copies and the like, however, are slow.

If you come up with any answers, please post back.