2 x DL380 G6 with drastically different drive performance

[IceMachines]

Greetings,

I recently ended up with 2 used DL380 G6's that seem pretty much identical.  Both have 8 x 146GB 10k RPM SAS drives with P410i RAID controllers.  Both have 48GB DDR3 RAM.  Both have dual socket Xeon E5540's.  I flashed all the latest firmware and removed the battery backup modules (the batteries were puffed).  I switched from RAID 5 to 1+0 on both machines and installed XenServer 6.5.  Everything I have looked at shows the machines to be configured the same.  The only difference I can spot is that one machine is getting acceptable drive performance off the array while the other is pathetic.

dd if=/dev/zero of=/tmp/output bs=256k count=1k; rm -f /tmp/output
Machine 1:
1024+0 records in
1024+0 records out
268435456 bytes (268 MB) copied, 0.824632 seconds, 326 MB/s

Machine 2:
1024+0 records in
1024+0 records out
268435456 bytes (268 MB) copied, 4.46843 seconds, 60.1 MB/s

Both are set to HP Static: High Performance Mode in the BIOS

Any idea why I'm getting such drastic performance differences between the 2 machines?

[IceMachines]

I haven't been able to figure out why I'm getting different performance from virtually identical systems so instead I've tried something different.  I've taken a 4 port 3Ware9650 Raid card and installed it.  I've connected the RAID card to the top multi-lane connector and replaced the right-side SAS drives with 4 x Intel 520 series sata SSD drives.  I've configured the 4 SAS on the p401i controller as Raid 1+0 and the 9650 RAID controller as RAID 5.  I've installed XenServer 6.5 on the SAS drives and then added the SSD array as a secondary local storage.

With 4 less SAS drives I'm still seeing the same performance stats as before.  When I test the sata SSD storage I'm seeing over 400MB/s.  While this does not help me figure out what is wrong with the P401i configuration it does give me a solution to the performance problem.  I don't care if the OS is on slower drives I only care that any VMs using local storage have fast drives.

If anyone out there does have an idea why the SAS array is so slow I'd love to hear about it.