HP Proliant DL580 Gen8 and GPU's

[GertDekkers]

Dear all,

I'm a PhD student at KU Leuven university.  I have a HP Proliant DL 580 Gen8 server with 3 GPU's at my disposal. Lately we've been experiencing full system shutdowns. The main thing that has changed is that I added an additional GPU. I'm certain that is is related but I don't know the exact cause. 
In the iLO we get the following message at 'Integrated Management Log':
System Power Fault Detected (XR: 10 A2 MID: BF 0F F0 00 00 FF 3F 0B 0B 04 04 0B 0B 0B 0B 04 04 ....)

The server setup:
HP PROLIANT DL580 Gen8 4X I-XEON E7-4850v2 12-CORE 2.30GHZ 512GB DDR3 RAM P830i 2GB RACK RAILS
4x 1500W Power
2x Geforce RTX2080Ti, 1x Geforce GTX1080Ti
CentOS

We think it's related to the GPU causing power issues/heat issues (although measured values are within accepted values).
By removing one GPU and limiting the average power to 220W for each CPU we haven't experienced any outages so far. 
Does someone have any experience with this server along with GPU's? I was also wondering what the 10-pin MOLEX auxiliary power connectors can provide? Now a single MOLEX is used to power one GPU. Also, does it matter which connector I use?

All information is welcome.
Thanks!

Best,
Gert

[KSINGH_2018]

Geforce RTX2080Ti & Geforce GTX1080Ti  are not listed under the compatible list of GPUs for DL580 Gen8 as per the server quickspecs.

https://h20195.www2.hpe.com/v2/getdocument.aspx?docname=c04199329

Kindly coordinate with NVIDIA and check the Hardware Compatibility List of the Vendor to confir if the GPU is supported by DL580 Gen8 Server or not.

If it is identified as supported by as per Vendor Hardware Compatibility List then please log a case from HPE Support Case Manager or call us to log a ticket for the same. For any service the server must be under Valid Active Contract/Warranty/Carepack and we'll require AHS Log for analysis as well

[nikolaandelic]

Hi, I am also wondering the same thing. Did you make it work? It should as the DL580 Gen9 at least supports 5 full size, full hight GPU's. 

[Wishgranter042]

Hi   @KSINGH_2018  

Im get in next few days USED ( Refubrished )   HP Proliant DL580 Gen8 and planning to sue Nvidia RTX 2070 Super ( 2 GPUs now, 2 wil be added later )  
So the question is it is supported on the server or it needa BIOS update or etc ? to work properly ? Older card like GTX 980 wil work in meantime ?  

[pdworzynski]

Hi,

I'm on the same boat. We've been using our DL580 Gen8 with Tesla K40 for a while but after increasing amount of RAM past 1TB we run into address limitation of K40 which makes it useless. I'm thinking about getting a couple of RTX 2080Ti's but i'm wondering about what issues may occur. Could you let me know if you run into any?

Thanks

[KSINGH_2018]

As per DL580 Gen8 Quickspecs, we have the following Graphics Cards tested & recomended by HPE's Engineering Team (Please refer to "HPE Graphic Options" on Page 31):

1. NVIDIA Quadro K6000 PCI-E Graphics Adapter (730874-B21)
2. NVIDIA Tesla K40C 12GB Computational Accelerator (753960-B21)
3. NVIDIA GRID K2 Reverse Air Flow Dual GPU PCIe Graphics Accelerator (753958-B21)

**HPE ProLiant DL580 Gen8 Quickspecs**
https://h20195.www2.hpe.com/v2/getdocument.aspx?docname=c04199329

Since Nvidia RTX 2070 Super has never been tested with DL580 Gen8 Server hence we can't confirm about the compatibility of the same. Please approach the Card Vendor (NVIDIA) for checking the Hardware Compatibility List for this card. Please do ensure to have the latest System ROM prior to the usage of this card.

[Wishgranter042]

Hi All 

So afterg a lot of reading and etc. i can confirm thaat GTX or RTX GPUs are locked out from dl580 gen8 servers. 
HP just use PCI-E Vendor IDs from the GPUs and when its not a QUADRO PCI-E IDs,  it just not work or direclty crash system  ( trying to identify card by a sw ) and card have permanent error code 43 - and it cant be solved.  so nice way to lock up using anything except QUADROs what are not so powerfull vs gaming cards in term of COMPUTE performance. 

[JBDynamics]

The HPE Proliant DL580 G8 is an enterprise product, as such is meant to only be compatible with enterprise grade GPUs. I can personally say that the Tesla M80 works on the DL580 G7. I have 2x Tesla M80's with the IO expander on the true x16 double wide slots with 4x E7-4870 and 512GB RAM. The Tesla P100 works great on the DL580 Gen8, I have 3x Tesla P100's with 4x E7-4890V2 and 768GB RAM. The server runs smoothly in the data room at my office, I had to install 2x large window air conditioners along with central air to keep the room under 24ºC, but I have 9 months uptime right now. I have the machine virtualized with VMware and I use PCIe passthrough to Ubuntu 18.04 VMs for TensorFlow containers with CUDA compute.

You are correct that GTX cards are not compatible, but it is for good reason. There are multiple layers of redundancy and CRC error checking on Tesla and Quadro products. Gaming products do not have these features. In order to keep a stable enterprise server, it must have enterprise grade peripherals. There is no wonder that an HP Enterprise product crashes when you install consumer grade GPUs. Yes, HP locks out the cards by PCIe vendor and product ID, but that is insurance for the warranties they provide. If you burn up your server because of some over clocked, hot running, consumer GPUs, that should be your responsibility, but instead of putting that burden on the owners, they lockout everything they know to not be compatible.

[6BQ5]

Just discovered this thread due to @JBDynamics reply floating it to the top. Interesting topic!

I'm impressed that people have installed multiple 2070 and better cards in their servers. At full power these cards can consume over 225W and create enough heat to warm a home. Installing three of these cards will easily take up over 650W of power! Add to that 2 x 130W power house CPUs, ~10W per hard drive, and who knows how much the motherboard and memory require ...

650W + 260W + 10W * 8 drives + 75W (??)  motherboard = 1090W

Holy smokes!! (literally) You're going to need HPE's beefiest power supply which I think is 1200W / each, I think? Remember, assume that one supply can fail putting the whole burden on the remaining supply.

Gamers plan their builds around the GPU and CPU and use online tools that add up component power consumption. I'm sure there's got to be one for HPE's servers

Side note : a humble Quadro P4000 card in comparison uses only 100W at full power.

I recenty installed a GeForce 1080 card into my ML350p Gen8 without any hitches or glitches. I also have 2 x 130W CPUs, a slew of little SFF hard drives, and enough memory to last me a few centuries. A good stress test is running SETI@Home or another distributed computing project with BOINC That will fire up all your GPU and CPU cores. The server has 2 x 750W supplies and they run fairly warm with the server at full throttle. I have been thinking of adding two more supplies to ease the burden on the first two.

I haven't heard of HPE locking out unsupported GPU cards but maybe that only applies to more newer servers or specific models?

[PeterPT]

Trying to make a K80 work on the DL580G7 with no luck...any insights?

Have the pcie expansion set up and running a k40c at the moment but need more horsepower for simulations and render.

4xpsus up and running, all green and no errors.

Thx!!!

[PabloCio]

Hey, I'm looking for a solution to a problem. I would like to install a GPU in my ML350p G8 apparently, you managed to put a 1080.

For my part I tried a 3070 and a Nvidia 570 but the fans don't run and the card doesn't boot. I think it's a power cable problem because I have a 1200w power supply.

So how did you do with your 1080?

Thanks

[techin]

@PabloCio 

I think you should consider posting a new topic for this problem.

[dyter07]

I have also some problems with a DL580 G8, 4 Sockets and 128GB RAM.

I tried several card, both RTX and GTX, all failed. A Tesla K20 works. My K80 didnt. All installed in Slot 9.

Now I purchased a K6000 which is certified for the G8. However the GPU is still crashing the server. Tried Windows 10 for Workstations and Ubuntu. Windows 10 W is working fine on the G8 btw, just the GPU problems.

In Win 10 the server starts all normal. After a few minutes Windows 10 is crashing. Same in Ubuntu 20.

Anyone else having similar issues?

[JBDynamics]

What OS are you using? I have dual K80s installed on my DL580 G7 with VMWare vSphere 6.0 U3, I pass two GPUs to Ubuntu 20.04 LTS and two GPUs to Windows Server 2019 DataCenter. I have the dual native x16 PCIe expansion chassis (the one with a regular PCI slot), with both GPUs plugged into native x16. Do you have a HP PSU to dual PCIe 8-pin connector power cable and the NVIDIA Tesla dual 8-pin PCIe to NVIDIA Tesla converter cable? Also, are your 4x PSUs 1200w Platinum? I was unable to do anything other than PCIe pass through, I could not get NVIDI GRID vCS to work.

[dyter07]

I have both DL580 G7 and G8. I use one G7 as my workstation with 1 gtx and 1 rtx gpu. Works fine, I love this system and its running Windows 10 for Workstations. I have 1 Dell R720 just for my storage with 65 TB. Another3 G7 running Linux and finaly the G8 also on Linux Ubuntu. The K6000 is working fine now.

[dyter07]

Just to give you an Update. I have my DL5800G8 up and running and may turn it into a workstation the next 2 weeks. 

Windows 10 for Workstation works fine, full support for 4 sockets.

I have 2 Nvidia RTX 3060ti running very fine in the system. I ordered more GPU power cable last week and I am just waiting for their arrival. Will try to add another 3 RTX 3060ti than, making this mashine a 5 GPU monster. at least I hope so.

I mostly use DAZ3D, Unreal Engine, Adobe and Blender.  With 5 cards it should be a render monster. With 2 cards it is already, but with 5 of them it will be a dream.

I will give you folks an update.

[RenderMan]

Could U send the Pic if really make it with RTX 3060 TI

[RockeyDA]

do i got one of these servers for 650$ but i am struggling to get a gpu to work

system specs: Proliant DL580 G8 

1tb ram, 4x xeon e7-4890v2 

GPU's tired GTX 295 GTX 480 GTX 650 TI boost, and as of yesterday Quadro M6000.

i have done biso and fiurmware updates, and downgrades, have tweeked alot of settings, my focus has been on server 2008 r2 but server 2019 as has been in my efforts as well

i have tried multipe GPU driver versions some comfirmed working on my Z800 and MANY OS re-installs. 

the driver never works

but get this... it does work in linux mint  

and it gets even stranger, LAGACY BOOT, the system gets a corrupted looking output then screems at me crashed when it tries to boot. if UEFI is on but optimised boot is off, it dosnt make noise but loads of graphical glictches and a crash. if no GPU is present there is no crash but legacy boot disables USB support untill OS drivers kick in (iLO remote console works for input still) if UEFI on, optimised boot off, and disable internal graphics active, i get the graphical glitches but sytem will boot even server 2008 and output video thogh gpu after POST. but no mater what driver issues stay the same... anyone know whats up? also i read stuff here, weather quadro or GTX its the same, runs great in linux but no drivers in windows.

[RockeyDA]

well someone in here said somthing about being over 1tb causing an IRQ problem, i was at exactly 1tb... so i pulled out every other ram modual and all my problems just went away.... wonder if i can manuly set IRQ's for things in bios like i could on older desktops, find an order that makes my GPU a higher priority and lets me have all my ram. i payed for 1tb and damnit i want 1tb!