DL380 Gen10 - Poor Memory throughput with two CPU configuration

[ThriveonEngi]

Hi everyone,

I've discovered an issue with the DL380 Gen10 servers running dual CPU's.  The issue is that memory throughput is limited to around 10GB/s.   I've benchmarked memory on the system with Memtest (Bootable USB) and NovaBench (ran on guest VM's).  I've done these tests with both CPUS, only CPU A, and only CPU B.  When testing while running one CPU, memory throughput is around 17GB/s - as soon as both CPU's are in, it decreases to 10GB/s.

I've put in two tickets with HPE support and they are not getting back to me, their last reply was on 1/18/19. I've also worked with the Vendor who was responseful and tried to help but have not been able to provide a resolution.

The system hardware is as follows:

ProLiatn DL380 Gen10

Two Intel Xeon Gold 6144 CPU's

12x 16GB Dual Rank MultiBitECC 1.2v DIMM's.  Each white dimm slot on each channel is populated (6channels per CPU) Slots 1,3,5,8,10,12 on each CPU. Memory is running at 2666Mhz.

All firmware is up to date.  BIOS workload profile is set to Virtualization high performance, power setting is static high performance.

I've tested a few other dual cpu Gen10's we have in the wild and they all show memory throughput around 10GB/s.

Typically memory throughput is not an issue, but the server in concern is running a memory intensive application server which users notice a significant decrease in performance when it's running on this host.  The old Gen9 host it's currently on has around 20GB/s memory throughput, this is a single CPU system with 4 channels for memory, 8 slots populated running at 2400Mhz.

Has anyone else noticed this issue?  If so, were you able to resolve it?

[ThriveonEngi]

*UPDATE*

I just benchmarked a Gen9 with dual CPU and memory throughput was 9GB/s.

Is the drop in memory throughput with dual CPU's expected.. Is this normal?  Do other systems have this issue?

[ThriveonEngi]

*UPDATE*

I went back to review some benchmarks on a Dell poweredge that we added a second CPU to.  Luckily I still have these results:

With 1 CPU memory throughput was around 13GB/s and after adding the second CPU it increased to about 17GB/s.  So adding the 2nd CPU shouldn't have a negative affect on memory throughput..  Curious why this is an issue on ProLiants.

[Rafi786]

Good day,

I see that HPE Engineering team is looking into this issue and will get back to you once they have more information to share.

Thanks for understanding.

[KevinSpring]

Did you/HPE get this resolved? I haven't run into it myself but I'm very curious. The only weird issues I've seen is with power saving or optimized profiles which you've already eliminated so I'd love to know what happens here.