Operating System - Linux의 주제 Re: DL385 + RHEL, performance issue

DL385 + RHEL, performance issue

J-F Bouchard — Mon, 06 Dec 2010 23:24:05 GMT

Hello,

I am currently building a configuration for use as our build server. The machine is a HP DL385 G7 with 2x8 cores CPU, 4x 300G 10k SSF disk, running RHEL 5.5. Everything is good, nice machine. First build (compile) that I try, I got 41 minutes via the time command. My desk computer (simple intel i5 with a 160G SATA disk does it in 14 minutes.)

I did look around, the machine does not have the BBWC, so I did force it just for the test. And I did force the Physical disk cache also. Nothing better. Still more than twice slower then my desktop computer.

I m looking for idea on how to debug this ?

Thanks in advance.

Re: DL385 + RHEL, performance issue

FLQ — Tue, 07 Dec 2010 10:05:23 GMT

Hi J-F,

you could try to mount your build directory on a RAM disk. If you have enough memory in the server.

That would speed things up a good bit. ;-)

HTH

Re: DL385 + RHEL, performance issue

Zinky — Tue, 07 Dec 2010 14:40:23 GMT

Hmm hard to believe the compile time is vastly that huge. Is your "compile" a single threaded/single process one or multiple?

Also - which Core i5 model do you have on your dekstop? if it is the 3Ghz+ model with TurboBoost -- then it will surely beat the %%&*( out of the puny Opteron 6100 which I can guess is lilely the 2.2 Ghz one?

Admittedley, the core i5's or any desktop CPUs with higher clock rates would perform their server cousins on single thread/single process tasks.

Re: DL385 + RHEL, performance issue

Zinky — Tue, 07 Dec 2010 14:42:28 GMT

One more thing... am not really familiar with the Opty based Proliant (we all have Nehalems here) -- but check if you have CPU capping on --- maybe your server happens to be set to run at low clock speed by some policy - etc?

Re: DL385 + RHEL, performance issue

J-F Bouchard — Tue, 07 Dec 2010 14:45:17 GMT

Hello,

Thanks for reply.

Yes, the server has 16cores total and the compile is fully multithread as i can see with top.

The idea of ramdisk is very good. I m trying this asap.

Thanks a lot

Re: DL385 + RHEL, performance issue

Zinky — Tue, 07 Dec 2010 14:49:04 GMT

I doubt it is I/O that you're bottlenecked on.

cat /proc/cpuinfo during your builds.

again what speed is your i5? (cat /proc/cpuinfo too)

The Gap between a 3Ghz Nehalem and a 2.0 to 2.4 Ghz AMD Opty should not be that vast. I suspect the DL385G7 is notworking hard -- maybe its clock speed is set low or per policy it is set to not be demanding.... (see cpufreq-utils)

Re: DL385 + RHEL, performance issue

J-F Bouchard — Tue, 07 Dec 2010 20:06:02 GMT

Hello,

Some news, I did nail down the problem. And its quite of a serious one. To get my performance back, I had to go in the BIOS and select "maximum" performance in power management.

Still, the Opteron don't beat the i5 (dual core HT). I now able to complete the compile in 18 minutes (The i5 do 15 minutes).

To try to get down my compile I did even try some filesystem tweak :

- noatime on ext3
- ext4

Nothing do, always same performance. It seems very CPU bound.

Thanks for all your input. If you guys have an idea on :
- How to report this problem (that seems a big one to me .. I think about all RHEL 5.5 users on the DL385.

- If you guys got more BIOS tweak.

Thanks a lot !

Re: DL385 + RHEL, performance issue

Zinky — Tue, 07 Dec 2010 20:08:58 GMT

See... the issue was with your CPU capping sir. Make sure you aer aware of your G7's capabilities and have all PSP stuff loaded sir!

the "minor" difference in performance is due to the disparity in CPU speed between your Lynnfield i5 *likely you aer hitting TurboBoost) at 3.4/3.6 Ghz versus your Opty's max of 2.4 Ghz for the 6100 series.

Re: DL385 + RHEL, performance issue

Zinky — Tue, 07 Dec 2010 20:13:33 GMT

http://www.google.com/url?sa=t&source=web&cd=1&ved=0CBYQFjAA&url=http%3A%2F%2Fh20000.www2.hp.com%2Fbc%2Fdocs%2Fsupport%2FSupportManual%2Fc01549455%2Fc01549455.pdf&ei=05T-TL_1BsmpnAf3wuDGCw&usg=AFQjCNEUXEpg3l5RLBOKIbPLagv3C2oxvg

Many a site and many admins I know who have G6's and G7's have their RHEL installations not erally optimised for the feautures of the Optys, Magny Ours and Nehalems... as most don't bother to check if PSP is isnatalled and configured correctly...

Re: DL385 + RHEL, performance issue

J-F Bouchard — Tue, 07 Dec 2010 20:17:23 GMT

Hello,

Do you mean that you see that has normal to have to change something in the BIOS to get the full power of the server ? That _double_ the power.

Also, about the PSP. Its quite old (September). I had to rollback to an old kernel to get the kernel RAID array module loaded. Is that normal ? Did a miss something ?

Thanks !

Re: DL385 + RHEL, performance issue

J-F Bouchard — Tue, 07 Dec 2010 20:20:27 GMT

Well, if I did not check the performance (bad practice) I would be very angry about HP. And just buy IBM next time.

Re: DL385 + RHEL, performance issue

Zinky — Tue, 07 Dec 2010 20:21:14 GMT

You really should not have to mess with the BIOS CPU settings as having it set at maximal power will:

1) make your server less GREEN (if you care)
2) will always have your fans at full speed
3) power onsumption
4) shorten the life of your machine

a properly configured Proliant G6/G7 should have the HP bits ON to interact with the backend BIOS/iLO "properly" and do such things as on-demand CPU throttling, etc.

RHEL 6.0 from what I have heard has the functionality "built-in".

Matti, et al will be likely able to add...

Re: DL385 + RHEL, performance issue

J-F Bouchard — Wed, 08 Dec 2010 15:22:47 GMT

Solution :

Users of RHEL 5 + DL350 (All HP server ?) should be advise to get rid of the power capping (In the BIOS). Otherwise, the machine will perform at about 50% of the CPU capacity.

Thanks to Alzhy for pointing this.