I/O Performance Question (RHEL5, rx6600, 2 P600 SmartArray Controllers)

[C Knorr]

We have an rx6600 that has two (2) SmartArray Controllers (P600's) in the 2 core I/O slots. We have all sixteen drive slots filled with 15K SAS drives. One of these drives is being used for the O/S.

Our goal is to get the fastest possible I/O performance we can. We are using tools like iometer and iozone to measure our read/write times.

When configured in a simple JBOD scenario, we can get about 400MB/sec on read and 150MB/sec on write with about 4 of the disk drives being used. Beyond this, we DO NOT scale at all. In other words, we're hitting a bottleneck and we don't know what it is. We are using both controllers simultaneously.

Does anyone have any thoughts? Based on our reading of the rx6600 specifications we would surely hope to see better I/O results. Any linux tuning suggestions? (BTW, attempts to use direct I/O with linux are *disasterous* -- write times go down to 40MB/sec. No idea why, although there are some ominous notes in the linux source code about the "state" of the direct I/O code ...)

thanks and regards,

Chris

[Bertho Stultiens]

The discrepancy between read and write is mysterious. You should be able to read and write at about the same speed if you use a simple parallel test on JBOD.

I guess that you would max out at about 570MB/s with two controllers, and that would mean that each disk must read/write 36MB/s at the inner cylinder. I don't think you get this. However, on the outer cylinder you should see max performance without problems. These numbers are of course for sequential read/write; assume the controllers can follow the commandflow; no scsi bandwidth is wasted; the PCI bus/chipset can handle the load.

Test scenarios I'd try:
- create 2 RAID 0 arrays; 8 disks on each controller
- create 4 RAID 0 arrays; 8 disks on each controller; 4 disks in each array
- create JBOD; 8 disks on each controller; single disk logical array

Use LVM to stripe the logical disks together and test (see LVM doc for stripe options).

DirectIO is always a roll of the dice and may/may not give you performance. It depends on the application as far as I have experienced.

If you see bottlenecks with 4 disks on two controllers, then you should check if the controllers are on separate PCI busses. Also check the performance of each controller separately to see if other HW factors play a role.

--
Greetings Bertho

[Steven E. Protter]

Shalom,

Since you have a smart array card here is your best bet beyond getting a real disk array and fiber connection.

Set up your data on raid 1 paired disks. Try and arrange it so you don't have too many high i/o databases and whatnot on the same area.

Lets say you have a pair of 70 GB disks, arrange these two disks into a single raid 1 hardware array and put a database on it.

Avoid raid 5 for anything that has heavy write activity.

Though the system can clearly handle lots of disks, this may not be the best way to go, all local disk. If you have a lot of heavy i/o, offloading it onto a disk array that is designed for massive I/O would do better.

I agree that direct i/o is not the way to go. If you must use local disk then you need to arrange it smartly, spreading around the i/o across the disks.

SEP

[TY 007]

Hello Chris,

>> arrange these two disks into a single raid 1 hardware array
>> and put a database on it

Yes, tested before. RAID-1 does make some differences.

Thanks

[C Knorr]

Thanks to everyone for the replies. We'll try the RAID-1 configuration. One "tidbit" is that we found that the ext3 file system actually performs better than ext2 on writes. We'd been told that ext2 was better for performance, but were able to go from 150Mb/sec up to 185Mb/sec with ext3.

One question I have on this statement:

> I guess that you would max out at about
> 570MB/s with two controllers, ...

How did you determine this "maximum"?

Thanks and regards,

Chris

[Mike Stroyan]

It sounds like you don't have the Battery Backed Write Cache (BBWC) working on those P600 controllers. The RAID controller will not cache writes if it does not have a working battery connected. The battery is required to be able to recover writes after a power failure. That controller also disables write caching on the disks themselves. So write performance will seem very poor if the BBWC is not in use.
The performance of read operations can also be affected by this because reads on a file system will usually require writes to update the file access time. (That can be eliminated by mounting with a noatime option.)
Once you have a working battery connected you can configure the ratio of read cache to write cache using the "HP Array Configuration Utility". http://h18013.www1.hp.com/products/servers/proliantstorage/software-management/acumatrix/index.html

[C Knorr]

Thanks Mike. Here's the output from a ctrl all show detail from the ACU utility:

Smart Array P600 in Slot 2
Bus Interface: PCI
Slot: 2
Serial Number: P92B30N9SU50FI
Cache Serial Number: P67570P9SU30UV
RAID 6 (ADG) Status: Enabled
RAID 6 (ADG) Enabler Status: Enabled
Controller Status: OK
Chassis Slot:
Hardware Revision: Rev A
Firmware Version: 1.52
Rebuild Priority: Low
Expand Priority: Low
Surface Scan Delay: 15 sec
Cache Board Present: True
Cache Status: OK
Accelerator Ratio: 50% Read / 50% Write
Total Cache Size: 256 MB
Battery Pack Count: 2
Battery Status: OK

I don't see anything to indicate whether the BBWC is enabled/disabled. We are using JBOD (RAID0) for the disks -- is it possible the BBWC is only used with RAID1/5/6?

thanks!

Chris

[C Knorr]

Thanks Mike. Here's the output from a ctrl all show detail from the ACU utility:

Smart Array P600 in Slot 2
Bus Interface: PCI
Slot: 2
Serial Number: P92B30N9SU50FI
Cache Serial Number: P67570P9SU30UV
RAID 6 (ADG) Status: Enabled
RAID 6 (ADG) Enabler Status: Enabled
Controller Status: OK
Chassis Slot:
Hardware Revision: Rev A
Firmware Version: 1.52
Rebuild Priority: Low
Expand Priority: Low
Surface Scan Delay: 15 sec
Cache Board Present: True
Cache Status: OK
Accelerator Ratio: 50% Read / 50% Write
Total Cache Size: 256 MB
Battery Pack Count: 2
Battery Status: OK

I don't see anything to indicate whether the BBWC is enabled/disabled. We are using JBOD (RAID0) for the disks -- is it possible the BBWC is only used with RAID1/5/6?

thanks!

Chris

[Mike Stroyan]

|Smart Array P600 in Slot 2
|...
| Cache Board Present: True
| Cache Status: OK
| Accelerator Ratio: 50% Read / 50% Write
| Total Cache Size: 256 MB
| Battery Pack Count: 2
| Battery Status: OK
|
|I don't see anything to indicate whether the
|BBWC is enabled/disabled.

The "Cache Status" and "Battery Status" show that the BBWC is enabled.

It is possible that you did some performance measurements with a newly installed controller that did not have the battery charged. That would be slower until the battery did charge up. In that case repeating the measurements would give different results.

| We are using JBOD (RAID0) for the disks -- is
|it possible the BBWC is only used with |RAID1/5/6?

I don't know of any reason that the BBWC would not be used for a RAID 0 array.

You might get better write performance by increasing the percentage of the cache used for write instead of read operations. But that really depends on how large a sustained write burst is. Once the cache is full with backlogged writes it can't help anymore until some of it can be moved on to the media. The cache effects could be more significant for an actual application than for a completely I/O limited benchmark.

You could also verify that your RAID 0 configuration has good striping to get the full bandwidth from the group of disks used for the array.

[C Knorr]

We have figured out the issue with why our READ times were so slow:

HP configured the machine by placing the 2 P600 cards in the first 2 slots, which on the rx6600 are a shared PCI-X 66MHz bus. PCI-X at this speed maxes out around 500MB/sec, so the 400MB/sec we were seeing was in the ballpark.

We solved the problem by moving the P600's to independent PCI-X slots, one of which is at 133MHz. We are now getting in excess of 1000MB/sec.

Not stopping there, though, we've ordered 2 P800 PCI-Express cards, since the rx6600 we got has these slots available. We should be screaming with these.

As to the WRITE problem, we are getting around 500MB/sec on WRITES now, which is about half of our READ performance. Our working theory at present is a comment we found by HP in the linux device driver for the P600 which says:

Disabling DMA prefetch for the P600 due to bug in .... {something_or_other}

We're pretty sure disabling DMA pre-fetch would have pretty profound (negative) effect on WRITE's. We'll see if the P800's improve the situation.

Thanks to everyone for their responses. I hope this information helps someone. For sure there are a lot of bus options and issues to grapple with if you want to tune for good I/O. The knowledge pool in this space @ HP or elsewhere isn't what I'd consider "huge"!

Chris