Why is the 385 faster than the 585?

[Matt-hew]

Here's what I'm comparing:

HP DL385(G1) configuration
2 X AMD Opteron 252 (2.6GHz)
8 GB PC3200 MEM
2 X 72GB u320 connected to 6i controller raid 1 (OS and swap)
4 X 72GB u320 connected to 6402 contoller raid 5 (app and db)

HP DL585(G1) configuration
4 X AMD Opteron 852 (2.6GHz)
8 GB PC3200 MEM
2 X 36GB u320 connected to 5i controller raid 1 (OS and swap)
2 X 72GB u320 connected to one channel of a 6404 raid 1 (app)
14 x 36GB u320 (in a MSA30) connected to the other channel of the 6404 raid 1+0 (db)

Both systems running RHEL4U4 (2.6.9-42.0.2.ELsmp x86_64 kernel) and Oracle 10g. Exact same version of everything. All firmware has been updated.

These systems run some batch processing that takes about an hour to complete. However the DL385 always completes about 5 minutes quicker running through the exact same data. Without detailed performance stats does anyone have any idea why the 385 would perform better? I tryed disabling NUMA and that slowed the 585 down even more. The only thing I can figure is since the 385 is a dual proc and 8 GB of memory, it's has 4GB local to each proc where as the 585 only has 2GB per proc. Any other ideas? settings to look at? Thanks.

[Steven E. Protter]

Shalom,

This situation should not be so.

You may wish to try the following:

Try the EL kernel instead of the smp kernel, sometimes it introduces inefficiencies.

Also try the hugemem kernel, it might provide better results.

Also the OS is on a larger disk in the first server. Its rpm speed may be higher proficing better performance.

SEP

[Steven E. Protter]

I missed something in my intial post.

The amount of memory per processor is a factor in performance. To maximize the performance of the 585 you need twice as much memory.

I heard this at an HP demo, which hopefully wasn't NDA.

SEP

[Matt-hew]

This is the 64-bit OS. There is no hugemem kernel for it. Also, I can't possibly see how a non-smp kernel (running a single proc) would provide better results.

[George Liu_4]

1. non-smp kernel definitely won't make performance better
2. more appropriately, the question should be why 585 performed worse.
3. any high I/O when the application was running? From top or sar?

[Matt-hew]

IO wait is low on the 585 during the run, never above 5%. On the 385 it gets over 20%. I don't think it's IO related.

[David Claypool]

Matt-hew, I would believe that your hunch is correct--any addressing beyond 2GB by a processor in your 585 configuration is having to address a 'distant' memory bank. This definitely would affect performance.

[Matt-hew]

Thanks, is there any way to tell how each cpu is using local vs. remote memory?

[kcpant]

Hi Matt,

You told that it is a batch processing activity where you found the 385 faster.

Since it is batch processing on a huge database, memory would be the main factor for better performance. Pls consider all points:

1. RAM per processor, you are already aware of it.
2. cache on MSA cards. the 585 is connected to an array of 14 drives.
3. RPM of HDDs.

Regards

[Matt-hew]

Thanks for the response. The 6400 cards have 192MB of RAM. All the drives are 15K rpm. On the 385, the app and db share a raid 5 disk group. On the 585, the app and db are on seperate channels.

[Hein van den Heuvel]

May we assume a more or less single stream job with high CPU busy, which in turn often means high memory usage?

The may be two effects playing a role here, adding up to almost 10%.

1) On the 4 CPU box, with striped memory, 1 in 4 meory access will be local. Sample Cost: 1.12. 1 in 2 will be 1 hop. Cost: 1.28. And finally 1 in 4 will eb 2 hops. Cost: 1.49. Average cost: 1.29
For the 2 CPU box, 1/2 will be no hops, 1/2 will be 1 hop for an average cost of 1.20
With non-striped memory the Linux kernel will favor local, 0-hop, memory access but can not do so for the Oracle SGA which will be 'everywhere'.

See: http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/40555.pdf

2) Memory bank interleaving. Are all the dimms the same in both machines? Then the 2 cpu box has 4-way memory bank interleaving versus 2-way interleavingt for the 4-cpu case. I did not readily find hard an fast numbers comparing this, but in the pas have seen from 2% - 10% application performance difference for this on other platforms.

Nice pictures... but no numbers:
http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00388120/c00388120.pdf

http://www.celestica.com/uploadedFiles/Tech_Bulletin_Aug804.pdf

Hope this helps,
Hein van den Heuvel
HvdH Performance Consulting

[Hein van den Heuvel]

Matt-hew,

Did you make any progress on this?

Any chance to reconfigure your 585 to have all memory connected to just 2, adjacent, cpu's? I _think_ you can run memory-less CPUs. How about using that to evasluate hop-costs?

Max interleaving would be at 8 Dimms/cpu, but due to electrical characteristics may require the memory clockspeed to be reduced.

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00302781/c00302781.pdf

Cheers,
Hein.

[Matt-hew]

Thanks for the replies. We have added 8GB of memory (16GB total) to the 585 and it still slower than the 385. In fact, it's slower than the 585 with just 8 GB of memory. All the memory is the same part # (1 GB DIMMs).