Extremely slow io on cciss raid6

[Ulrik Holmén]

I solved the problem by forcing enable of writecache. It seems I forgot to order the BBWC and then the writecache was disabled by default. I have to order a BBWC to make sure I don't break anything in case of a powerfailure.

ctrl slot=2 modify drivewritecache=enable

The difference was quite astonishing:

sync; time dd if=/dev/zero of=/dev/mapper/VolGroup00-test bs=1M count=3000; sync
3000+0 records in
3000+0 records out
3145728000 bytes (3.1 GB) copied, 35.2907 seconds, 89.1 MB/s

real 0m35.292s
user 0m0.008s
sys 0m4.891s

[Jon Gomersall]

Check out this Forum question

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1225006

[fschicker]

dear ulrik,

i would NOT prefer enabling DWC because you are losing data on power loss. the BBWC doesnt help here because its the write cache directly on the disk.
its intresting that this helps, we have disabled disk writecache on all our servers (where we dont use p400) and get the performance you have (100-120mb/s).
i think enabling DWC helps, but ISNT the solution.

see what HP says under: http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01149818〈=en&cc=us&taskId=101&prodSeriesId=3369549&prodTypeId=329290

greets

[Ulrik Holmén]

No. I think you're right. It wasn't the real solution but it mitigates the problem with io wait. I have now ordered a battery backup to the controller so soon I feel safe with the solution as well.

I've read a lot of reports regarding what I think is the same problem with solutions ranging from BBWC to different fs.

[fschicker]

dear ulrik,

i think its not a problem with the bbwc.
we search for the error since about 1 week, we have our DL320s with BBWC and the problem occurs also.

i guess it is one of the following:

- problem with DMA handling, i looked at the cciss driver source, there is a case with P600 where problems occur
- hardware revision of the p400. we have a customer who has also the same dl320s but a newer P400 HW REV, he has NOT the problems we have
- high memory irq conflict or shared pci-x slot

can you send me some information of your system:

- hpaducli -f result
- hpacucli output from "controller slot=2 show config detail"
- lspci -v and lshw output

thank you!

Florian

[Ulrik Holmén]

Output from HP Array Configuration Utility CLI 8.0-14.0

[Ulrik Holmén]

Output from hpaducli

[Ulrik Holmén]

Output from lspci

[Ulrik Holmén]

Output from lshw.

Here you go.

I have also been through the driver a couple of times while troubleshooting the issue and I saw the P600 failure in DMA prefecth but I'm convinced that it is not the same issue as we have with the P400 as the result from DMA prefetching from memory locations outside of the memory would not only be slow write access :).

[fschicker]

thanks for your files, looks all normal i think.

my questions:

- are you using original hp disks?
- did you try to upgrade to latest fw?