Horrible Write Performance with Smartarray P400 in DL320, DL320s and DL1xx

[John Kufrovich]

Florian,

I'll agree with Hein, there is no battery.

You're last output shows that.

controller slot=2 show config detail

Smart Array P400 in Slot 2
Bus Interface: PCI
Slot: 2
Serial Number: PA5360J9STU1LA
Cache Serial Number: PA82C0D9STJ12S
RAID 6 (ADG) Status: Disabled
RAID 6 (ADG) Enabler Status: Enabled
Controller Status: OK
Chassis Slot:
Hardware Revision: Rev B
Firmware Version: 5.20
Rebuild Priority: Medium
Expand Priority: Medium
Surface Scan Delay: 15 sec
Cache Board Present: True
Cache Status: OK
Accelerator Ratio: 100% Read / 0% Write
Total Cache Size: 256 MB
Battery Pack Count: 0
SATA NCQ Supported: False



Battery Pack Count: 0

jk

[fschicker]

john, please read my post carefully ;)

i have one dl320s WITH bbwc and one WITHOUT bbwc, on both the same problem. (we have about 30 hp servers).
the output i pasted is from the one without the bbwc but its the same (bad) speed as with bbwc.

(i hope its clear now)

[John Kufrovich]

Florian,

Then why attach two outputs, acucli and adu, for the same system without BBWC. My concern would have been the system with BBWC and it's slow performance.

Could you supply reports from the server having poor performance and BBWC?

Let's see if that blocksize is too large for those SATA's. Instead of 1M blocks, can you try something smaller. 128k or 256k

Have you look into the default elevator queues for those OS's.

jk

[Ulrik Holmén]

Have you tried installing windows server on one of the servers. Just to see if it is a driver or hw issue?

[fschicker]

after more than 1 month of searching the failure with hp techsupport (yeah we got 2nd level support... but the person there didnt even know the hpacucli commands...) we found the problem ourself.

here what we did:
- we opened a new support case at hp where we said that our p400 controller is defect in the hope that we get a newer hardware revision
- 2 days later the new controller arrived, and yes!, with a newer hw revison!
- switched controller in one dl320s
- testet again, and got much better performance:

sync; time sh -c "dd if=/dev/zero of=/mnt/test2 bs=512k count=4000; sync"
4000+0 records in
4000+0 records out
2097152000 bytes (2.1 GB) copied, 40.734 seconds, 51.5 MB/s

real 0m48.770s
user 0m0.016s
sys 0m6.072s

(before we got 8 mb/s max.)

so, HP, please NOW believe us that its not an issue of RAID 6 or a "normal" problem of SATA Disks. No its your crappy P400 hardware revision which has major problems...

now i will try to explain this issue to the hp support guys. (ohno! that will be a horror-trip...)

thanks to all guys who tried to help - i think you have to change your controllers also.

greets from austria,

Florian

[Hein van den Heuvel]

Thanks for the update! Much appreciated.

I hope we at least helped to convince you that you are doing the right tests and looking at the right settings and thus to pursue the real cause (that HW revision).

Cheers,
Hein.

[Chirs Werfel]

As a follow up I had a similar problem with an AIO1200 storage server (DL320s with P400) running Windows Storage Server. The hardware revision on the card is newer than what you posted, I think, but the symptom was UNUSABLE write performance without enabling the physical disk write cache. I am posting my summary below in case anyone else experiences this problem. I am unable to go to firmware 5.20 due to a known incompatability issue with the AIO ASM software.

1. AIO1200 w/ASM 1.4, P400 controller w/fw 4.12, SATA drives exhibits slow write performance and variable file transfer approximated times during Windows large file copies. Perfmon counters for Physical Disk Current Queue Length and Network Adapter Bytes/Sec show that disk queue rapidly rises to a fixed number (21 on a three disk raid 5 volume) while network peaks at 300-500 Mbps then drops of to zero. Once the disk queue drops to 0 network activity jumps back up. This cycle continues until the transfer is completed.
2. Turning on the P400 Advanced Setting, Physical Disk Write Cache improves file write times by 100%. Network activity stays consistent even though the disk queue length pegs at 21.
3. Rebooting after changing the write cache setting further improves file write times by another 100% (dropping total transfer time to one fourth of the original time). Presumably the OS has awareness of this setting further optimizing write times.
4. As per the documentation Physical Drive Write Cache should only be enabled when the host server has an uninterruptable power supply.

[George Mason (Xoom)]

Hi Florian,

Just a quick note to add my experiences and maybe help explain what happened in both yours and my case.

I did extensive work on a customer machine, an ML350 G5 with E200 controller and SATA disks. Write performance (and disk IO in general) was terrible and would sometime crash the server (Windows 2003). After updating the system BIOS and E200 firmware many times, nothing changed.

The problem was that the combination of E200 driver and firmware would not allow the disks' hardware cache to be enabled on SATA drives - the driver would always disable the cache after I tried to enable it.

Finally HP released a new version of the E200 firmware which allowed SATA drive cache to be enabled (with warnings about UPS etc.) and the problem went away for good - the server is now fast and very reliable.

This issue sounds to me the same as the one you had with the P400 - I assume the new motherboard shipped with a controller running the updated firmware and therefore allowed the drive cache to be enabled.

Glad you got it working anyway. I'm looking to buy a DL360 with P400 controller + SATA disks so I hope this is now fixed on new machines!

Cheers.

George

[Ed Roche]

We have 2 DL380 G5s with this P400, same strange disk behavior even when server is unused. We were having weird performance issues on one of them where the network share would hang for 7 to 10 seconds every so often during the day. After a final shot at solving the year old problem it seems to have been fixed by one of two things. Either upgrading the cache from standard 128mb to 512 cache plus battery (Cache 50/50) or upgrading veritas backupexec from 10d to 11d. After a lot of research and speaking to a proliant consultant it seems that the p400 is a poor piece of kit.

[Ste B]

Hi George,

I was wondering how you were able to determine that the drives' cache has been switched off? I can't get hdparm to look at the drives at all.

I've managed to switch on the controllers' write cache (dwc=enabled) on my card - and now my speed has been lifted to the heady heights of 14Mb/sec!

Thanks,

Ste