SAS Performance reduced almost 800%

[steve drake]

Anybody else try PSP760 and the cciss 2.6.12 driver with the SLES9 SP3 2.6.5.7-244 kernel and see their SAS read and r/w performance drop from 500 MB/Sec to 68 MB/sec!

I was using the cciss 2.6.8 and 2.6.10-14 drivers to achieve 500 mb/sec.

Config is ML370G5, P400 (with the 8x extender which is NO help at all), 8 72GB 10k SFF drives configured for RAID 0.

I'm reading a 285MB file to /dev/null for the read performance numbers and then rebooting and copying the same file for the r/w performance. Read times for the file used to be .583 seconds, but with the new cciss driver this now takes 4.2 seconds.

Any comment from HP?

[chongkan]

Hi,

Rollback the driver to it prior version to see if problem is correted. If not, there may be something else downgrading the performance.

Just to be sure that it is the problem..

Regards.

[steve drake]

I rolled it back alright. I rebuilt the server and installed PSP7.52. It doesn't really matter to me if the trouble is the cciss driver or some other driver in PSP760- that's a job for HP. Obviously, the trouble occurs when I install PSP760 on top of the .244 kernel. So I won't do that again. Previous testing had already revealed a 6% drop in read and r/w performance between the original 2.6.8 driver (from a sles9 sp3 installation) and the 2.6.10-14 (from psp7.52) driver - a drop from ~495MB/sec to 465MB/sec. I was hoping that PSP760 (more specifically, the cciss driver) would correct this when I installed it. Ironically, if install SLES9 patches first, the kernel is upgraded to .276 and then PSP7.60 can do no harm as the 'new' cciss driver won't install on the .276 kernel.

My problem is solved. My point in starting this thread was to verify that others have experienced the same problem. By the way, I'm using the smp kernel and reiserfs filesystem.

More questions: Any idea why the 8x pcie expander for the p400 doesn't improve sequential read performance at all. lspci confirms that slots 6-7 are now 8x, yet performance with 8 drives remains unchanged. Is the p400 the bottleneck - is that why a p800 is coming Nov. 22?

[steve drake]

I reinstalled the OS - SLES9 SP3 and installed only the cciss driver from PSP7.60. I booted the new kernel (HP's version of .244) and tested SAS performance - over 5 seconds to read 285MB. This is no good. Clearly the new kernel with 2.6.12 cciss driver is the problem.

[steve drake]

Hello....anybody out there?

[steve drake]

Emailed HP support and learned that read_ahead is set to 0 for cciss v2.6.12 and is responsible for the poor sequential read performance. Setting read_ahead to 1024 returns seq read perf to previous levels.

[Marek Podmaka]

I think 1024 (kb) is too much, but it depends on your usage. If I understand it well, this means that for every single read operation the driver will read another 1 MB of data just in case that you might need it later. But if you have for example small files or randimized reads (database) then it is bad.
At least I observed performance degradation on RAID1 with 2 SCSI disks on Smart Array 532 over using these disks without RAID. I reduced the readahard to 256 (kb) and it is much better.

You will find these settings here (for first and second logical array configured):
/sys/block/cciss!c0d0/queue/read_ahead_kb
/sys/block/cciss!c0d1/queue/read_ahead_kb

[steve drake]

In our application large sequential read is far more important. Performance reductions in this category are instantly noticed. I'd like to try a far larger value of read_ahead. I'm curious why my seqential read performance is limited to 500Mb/sec when the interface is capable of much more. I would have expected something closer to 700-800MB/sec. Maybe this will be better with:
P800 Raid controller
MSA 60 - Storage controller
36 GB 15K SFF SAS drive

[sdrake48]

Sequential Read SAS performance seems to be dropping again.

Testing with DL380 G7, 410i/512MB (fw 3.3), and 6 x 146GB dual port 6G 15k drives configured RAID0. Negotiated link = 6G.
SLES10 SP3 installed with PSP8.50
cciss driver = v3.6.24-4

default read_ahead_kb for the cciss driver is 512

Sequential read performance(cp file to /dev/null) best case is approx 400 MB/sec

20% lower than the G5 with P400 and cciss 2.6.10

Changing read_ahead_kb to 256 results in 587MB/s sequential read - an improvement of >40%.

Any chance I can get a version of cciss driver with 256kb read_ahead_kb? or a module parameter I can permanently set.