DL385 USB very very slow

[Mark Woodruff]

We're having problems with the USB performance on both our DL385 G2 and G5 with Windows Server 2003 (R1 x86 on the G5 and R2 x64 on the G2). Both have USB 2.0 set in the BIOS. Device Manager shows an Enhanced PCI to USB Host Controller on both systems (driver version 5.2.3790.1830, 2003 SP2). Firmware at 9.0. PSP at 8.30. About to load 8.40 to see if it makes any difference.

We're getting about 7 MB/sec writing a 1 gig file to an external USB drive. Same drive on other systems shows 64 MB/sec on small files, 28 MB+ on larger ones.

Drives are connected to the back USB ports.

[P Muralidhar Kini]

Hi Mark,

Do you have multiple USB Ports ?
If YES, then is the problem seen with all the USB Ports ?

Check the BIOS setting for the USB. There would be couple of options like
- USB High Speed -> USB 2.0
- USB Full Speed -> USB 1.0

You BIOS setting should be set to USB High Speed i.e. USB 2.0.
Check whether its already set to USB Full speed and hence causing the problem.

Hope this helps.

Regards,
Murali

[Mark Woodruff]

As I said in my message, the ports are already set at 2.0 in the bios on both machines. The problem is on every port.

[Mark Woodruff]

On the 2003 R2 x64 machine (DL385G2), I installed the following hotfixes:

KB949483
KB944704
KB941716
KB925528
KB938136
KB972659
KB918005-v3
KB973618

I uninstalled all USB devices, rebooted, and finally I got decent performance after Windows recreated them.

Same procedure on the 2003 x86 machine (DL385G5) yielded no difference.
Unfortunately, it's the latter machine I really need to copy half a terabyte of data from. Not easy at 7MB/second...

[Mark Woodruff]

Found and fixed the problems(s):

There are performance issues that at least one of the USB hotfixes address. But in addition to that, earlier we'd run into the problem of running out of paged kernel memory, delayed write errors, etc. when copying large files over the network. The suggested fix is in KB920739, creating the SystemCacheDirtyPageThreshold key. What Microsoft's documentation doesn't mention is that the lazy writer can only write 1/8 of its dirty page buffer per second. That means that when the threshold is hit, all file I/O is limited to 1/8 of the size, which we'd set to 256MB. 256MB/8 = 8MB/s which coincidentally happens to be the limit of USB 1.1. This doesn't cause a problem normally because the P400's write cache hides the problem well. It only shows up on non-cached I/O (USB and network).

Solution:

* Install the USB hotfixes
* If you've set SystemCacheDirtyPageThreshold, make sure it's at least eight times as fast as your network or USB connection
* Delete all USB devices
* Reboot. Make sure the USB bios setting is set to 2.0 rather than it's default of 1.1.

If you still run out of paged kernel memory with the higher value, the only option is live with slower I/O speeds or switch to the x64 bit version of Windows. It's an architectual problem that can't be fixed in the x86 version.

And now, back to our regularly scheduled database backups...

[varandian]

We're having the same problem but only on servers with AMD CPUs. I've noticed it when running the Service Pack for ProLiant or SmartStart CDs/DVDs which means that this is NOT related to Windows. This does not occur on servers with Intel CPUs. Still researching...