USB hard drive stops responding

[PierreVST]

Server: HP ProLiant ML350p Gen8
ESXi 5.1.0 1157734
Hard Drives: Seagate RSS Backup+Desk
VM USB Controller: EHCI+UHCI
Guest OS: Windows 2008 Server R2 SP1

Issue:
Having problem with USB Hard drives. ESXi seems to drop connection to external hard drives. They stop responding in the Windows Server. It's still showing the USB Device in the VM settings. I still see it available and linked to my VM. If I remove it, the status doesn't change when shown in the available devices. At that moment, I can't add or remove any other USB Hard drive to any other VMs. Even if you add an other external hard drive to the server, on any usb port, it won't show up.

Vms are running fine, but I need the USB hard drive for backup purpose. Problem started since the first backups of the server (New installation, made 2 weeks ago). Running Veeam 6.5 update 3.

To workaround this problem, I have to reboot the host. Everything is normal until out of a suddent, backup fails saying it can't write to path. I look at the hard drive and it's not responding. Looks like the hypervisor is losing connection to USB Hard Drives.

I get this event repeating every second until I reboot the host:

URB timed out - USB device may not respond
(2013-09-12T17:03:24.978Z cpu21:8736)
warning
2013-09-12 13:03:24
servername.domain.local

USB Arbitrator service is running

USB Keyboard on the host screen is working

What I tried:
Updating ESXi
Updating all HP Firmware

While the host is online, restarting the services.sh won't work either.

My question is, where do I need to look at. HP or VMWare ? Or USB hard drives ? I don't have another set of 5 external hdd to try..