Horrible Write Performance with Smartarray P400 in DL320, DL320s and DL1xx

[fschicker]

We are a small hosting company in austria and are using P400 controllers in about 20 HP Servers (DL 320(s), 160, 180)
Now, since more servers have mucher to do, we recognize very strange performanceproblems on writing to a logical drive
on the P400 Controllers even if the server is idle.

some tests we made, shows the performance problems:

OS: Debian Linux, we also tried 2.6.25 kernel and Ubuntu, CentOS and FreeBSD

Controller: P400 Rev 1 or 3, with BBWC and without, all with 256 MB Ram, latest FW or older (see detailed info attached from hpacucli controller details and hpaducli output)

Server: DL320 (FW Upgraded to latest Version with Firmware Maintenance 8.10)

if we make write stuff to the disk you can see the problem:

time sh -c "dd if=/dev/zero of=/tmp/test bs=1024k count=4000; sync"
4000+0 records in
4000+0 records out
4194304000 bytes (4.2 GB) copied, 325.159 seconds, 12.9 MB/s
real 7m49.905s
user 0m0.004s
sys 0m12.121s
(you see, syncing needs also 2 minutes after dd has finished, after this the serverload is about 5, before 0.01)

we tried almost everything to find the problem, her is a list what we tried:

- updated p400 fw to latest version
- upgraded servers fw to latest version with fw-maintenance 8.10
- tried other linux distri
- tried other linux kernel (latest 2.6.25)
- other HP Server: we tried DL320s, DL320, DL160 and DL180
- tried other p400

Now we dont know what to do now, we have no more ideas how we could solve the problem :(

It would be very very very nice if anybody could help us with this issue!

Greetings from austria (the country without kangoroos),

Florian Schicker

[fschicker]

i could only attach 1 file, here is the second!

[Hein van den Heuvel]

>> with BBWC and without

I see no sign of BBWC in the attached listings, and see 3 signs it is not there:
1) RAID 6 (ADG) Status: Disabled
2) Accelerator Ratio: 100% Read / 0% Write
3) Battery Pack Count: 0

Now I know next to nothing about the P400, so maybe I'm not looking in the right place, but the reported performance is consistent with no BBWC.

For the Battery Backed Write Cache to work you need cache (present!) and... drumroll... a battery.
The controller does not seem to see a battery, so it does not do the write cache (0% write), and refuses to do ADG as that performance would be totally horrible without cache assist.

The system appears to be running bare metal raid 0+1 for which the IO response time is the worst case for each disk for each write. Both physical disk IOs need to be ready before the OS is informed that the IO is ready. It seems to me that the 13MB/sec is a reasonable performance under those conditions.

http://h18006.www1.hp.com/products/quickspecs/12400_div/12400_div.HTML

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[Ulrik Holmén]

I kind of solved this problem in my setup. I enabled the write cache via hpacucli:

=> ctrl slot=2 modify drivewritecache=enable

It gave quite a performance bump. From 8Mb/s to 124Mb/s semi sustained rate.

[Hein van den Heuvel]

yabut... The "drive write cache" setting goes out to the mode-page in the drive, enabling the unprotected per-drive (8MB)write cache.

As per ACU Ref Manual:

"On controllers and drives that support physical drive write cache, you can use this command to enable or disable the write cache for all drives on the controller.

CAUTION: Because physical drive write cache is not battery-backed, you could lose data if a power failure occurs during a write process. To minimize this possibility, use a backup power supply"

btw... that same manual confirms the r/w cache ratio of 100/0 indeed indicates that there is no writeback cache:

"Changing the controller cache ratio.
... You can change the ratio only if the controller has a battery-backed cache (because only battery-backed cache can be used for write cache)"

Hein.

[fschicker]

hein:
the files i attached are from a dl320, but we have the problems also on DL320s with BBWC, same setup which Ulrik has.

write performance of 8 mb/s is surely not an issue of wrong raid level :) we have many other systems with other controllers and we get always a minimum of 40mb/s at writing.

greets

[Hein van den Heuvel]

And is that BBWC truly active?
Show me the money!

Did you try the "modify dwc=enable", just as a problem finding technique, or as a longer term solution if the risks are acceptable (UPS) for the application?

hth,
Hein.

[fschicker]

yes it is!

Battery Pack Count: 1
Battery Status: OK

i dont want to try dwc on our DL320s because it is a live system. on our DL320 hpacucli dont let me set it:

=> controller slot=2 modify drivewritecache=enable

Warning: Without the proper safety precautions, use of write cache on physical
drives could cause data loss in the event of power failure. To ensure
data is properly protected, use redundant power supplies and
Uninterruptible Power Supplies. Also, if you have multiple storage
enclosures, all data should be mirrored across them. Use of this
feature is not recommended unless these precautions are followed.
Continue? (y/n) y


Error: This operation is not supported with the current configuration. Use the
"show" command on devices to show additional details about the
configuration.

- config details:

controller slot=2 show config detail

Smart Array P400 in Slot 2
Bus Interface: PCI
Slot: 2
Serial Number: PA5360J9STU1LA
Cache Serial Number: PA82C0D9STJ12S
RAID 6 (ADG) Status: Disabled
RAID 6 (ADG) Enabler Status: Enabled
Controller Status: OK
Chassis Slot:
Hardware Revision: Rev B
Firmware Version: 5.20
Rebuild Priority: Medium
Expand Priority: Medium
Surface Scan Delay: 15 sec
Cache Board Present: True
Cache Status: OK
Accelerator Ratio: 100% Read / 0% Write
Total Cache Size: 256 MB
Battery Pack Count: 0
SATA NCQ Supported: False

Array: A
Interface Type: SATA
Unused Space: 0 MB
Status: OK

Logical Drive: 1
Size: 74.5 GB
Fault Tolerance: RAID 1+0
Heads: 255
Sectors Per Track: 32
Cylinders: 19146
Stripe Size: 128 KB
Status: OK
Array Accelerator: Enabled
Unique Identifier: 600508B100104A39535455314C410000
Disk Name: /dev/cciss/c0d0
Mount Points: / 70.7 GB, none 3.8 GB

physicaldrive 1:1
Port: 1I
Box: 1
Bay: 1
Status: OK
Drive Type: Data Drive
Interface Type: SATA
Size: 80.0 GB
Firmware Revision: HPF0
Serial Number: 5RW0RNZV
Model: ATA FB080C4080
SATA NCQ Capable: False
physicaldrive 1:2
Port: 1I
Box: 1
Bay: 2
Status: OK
Drive Type: Data Drive
Interface Type: SATA
Size: 80.0 GB
Firmware Revision: HPF0
Serial Number: 9RW08J3W
Model: ATA FB080C4080
SATA NCQ Capable: False

[Hein van den Heuvel]

>>yes it is!
>> Battery Pack Count: 1
>> Battery Status: OK

Ok. That looks good.

I assume you also checked the "Accelerator Ratio" for more than 0% Write


>> Error: This operation is not supported with the current configuration.

Sorry. Dunno.

Hein.

[fschicker]

yes, we tried other values, no effect.