DL380 Gen10 - Poor Memory throughput with two CPU configuration

[ThriveonEngi]

Hi everyone,

I've discovered an issue with the DL380 Gen10 servers running dual CPU's.  The issue is that memory throughput is limited to around 10GB/s.   I've benchmarked memory on the system with Memtest (Bootable USB) and NovaBench (ran on guest VM's).  I've done these tests with both CPUS, only CPU A, and only CPU B.  When testing while running one CPU, memory throughput is around 17GB/s - as soon as both CPU's are in, it decreases to 10GB/s.

I've put in two tickets with HPE support and they are not getting back to me, their last reply was on 1/18/19. I've also worked with the Vendor who was responseful and tried to help but have not been able to provide a resolution.

The system hardware is as follows:

ProLiatn DL380 Gen10

Two Intel Xeon Gold 6144 CPU's

12x 16GB Dual Rank MultiBitECC 1.2v DIMM's.  Each white dimm slot on each channel is populated (6channels per CPU) Slots 1,3,5,8,10,12 on each CPU. Memory is running at 2666Mhz.

All firmware is up to date.  BIOS workload profile is set to Virtualization high performance, power setting is static high performance.

I've tested a few other dual cpu Gen10's we have in the wild and they all show memory throughput around 10GB/s.

Typically memory throughput is not an issue, but the server in concern is running a memory intensive application server which users notice a significant decrease in performance when it's running on this host.  The old Gen9 host it's currently on has around 20GB/s memory throughput, this is a single CPU system with 4 channels for memory, 8 slots populated running at 2400Mhz.

Has anyone else noticed this issue?  If so, were you able to resolve it?

[ThriveonEngi]

*UPDATE*

I just benchmarked a Gen9 with dual CPU and memory throughput was 9GB/s.

Is the drop in memory throughput with dual CPU's expected.. Is this normal?  Do other systems have this issue?

[ThriveonEngi]

*UPDATE*

I went back to review some benchmarks on a Dell poweredge that we added a second CPU to.  Luckily I still have these results:

With 1 CPU memory throughput was around 13GB/s and after adding the second CPU it increased to about 17GB/s.  So adding the 2nd CPU shouldn't have a negative affect on memory throughput..  Curious why this is an issue on ProLiants.

[Rafi786]

Good day,

I see that HPE Engineering team is looking into this issue and will get back to you once they have more information to share.

Thanks for understanding.

[Anonymous]

Did you/HPE get this resolved? I haven't run into it myself but I'm very curious. The only weird issues I've seen is with power saving or optimized profiles which you've already eliminated so I'd love to know what happens here.

[DavorK]

I have similar problem.

Dual gold processors with 128gb memory per processor. Memory is installed per manual and system don't report any errors during post operation.

I get 4gb/s throughput but when i remove second processor, the througput go to 10gb/s.

Processor is 2 x Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz and memory is DualRank DDR4 @2666Ghz.

Can any have idea what to set or do to make this server run as i new server?

Bios profiles isn't helping a bit.

Thanks

[Sunitha_Mod]

Hello @DavorK,

Thank you for posting! Since you have posted in an old topic and there is no response yet, I would recommend you create a new topic using the create "New Discussion" button, so the experts can check and assist you further.  

[MeisterLone]

Rafi786 

Were you able to resolve this issue? Can we get an update from HPE please.  Latest firmware here many customers are having the same issue, thanks.

[Suman_1978]

Hi,

There is a similar issue discussed outside HPE, please check this.  Have you performed any tuning regarding NUMA?
https://www.hpe.com/info/server-performance-management-tuning-en

Please see the UEFI guide to setup or configure NUMA, then check the throughput.

Thank You!
I work with HPE but opinions expressed here are mine.
HPE Tech Tips videos on How To and Troubleshooting topics