DL385G7 offering poor performance

[Michael A. McKenney]

Did you check on optimized drivers for the controller? Did you consider a better controller?

[Gary_Richards]

The exact same OS/drivers give me fine performance on the DL360 with the same controller. The bonnie++ results attached to a previous post suggest that disk performance on these systems is comparably fine too.

So I don't think drivers or the storage controller are the issue here.

It seems that MySQL doesn't perform very well at all on these DL385's, at least, I can't make it perform anywhere near as well as it does on the DL360.

[TwoProc]

I don't know if this chipset has it, but turn off any hyperthreading or any hperthreading-esque features in the bios if it does. It, by itself could explain your performance tests being 50% longer.

[Joshua Small_2]

Are you definitely running the exact same MySQL version?

If you are (so we know that's not the problem) why don't you try compiling it from source? If the issue is "different processor architectures", getting a version compiled on your platform may be the fix.

[Gary_Richards]

Hi, this CPU is a 12 core and displays 12 processors within Linux, so I don't believe there's any hyperthreading related options on these AMD systems.

We have been pondering compiling mysql from source with optimisations for this specific CPU enabled as by default we're using the Ubuntu (amd64) packaged version. The amd64 name can be deceiving as this package also works fine on 64bit Intel cpu's too, so presumably there's some slightly less specific tuning done so that it'll work on both 64bit CPU's.

[Gary_Richards]

If anyone's interested I rebuilt the Ubuntu packaged version of MySQL using the source package and a couple of modifications to compile with -mtune=amdfam10.

Once built and installed, mysql actually performed worse that the Ubuntu provided package!

So still no luck here on running MySQL on this server with any sort of sensible performance.

[shanedawg]

Hi Gary, did you ever figure out a solution to the issue?

we have some of the same sounding symptoms.

old server than ran great but was nearing capacity, new server is also running a postgres 8.2 database.

here are the details of our setup

old server
>dl385 g5 AMD Opteron 2356 2.3Ghz, 64G ram
>bios A09 03/27/2008)
>Smart Array P400 for root partition
>Smart Array P800 (firmware 5.20) attached to an MSA60 with 12 146G 15k drives for data partition (raid 10)
>running Ubuntu 8.04.1 linux
>
>
>
>New server example (we have 4 that are all the same type and exhibiting
>the same issues)
>dl385 g7 AMD Opteron 6176 SE 1.8Ghz 32G ram
>bios A18 01/29/2011)
>Smart Array P410i for root partition
>Smart Array P411 (firmware 5.06) attached to a D2700 drive array with 24 146G 15k drives for data partition (drive firmware HPDG) (raid 10)
>running Ubuntu 10.04.02 linux
>
>
>random read performance was slow from the database point of view and the
>application performance was bad so we started looking at the performance
>of the server and in doing that found iostat graphs that show the new
>server is only able to sustain about 400r/sec and the old machine can do
>about 1200r/s sustained.  We also ran bonnie++ disk benchmarks on them
>and the benchmark shows that the new servers can only do about ~350
>random seeks/sec vs ~1200 for the old server.

I have a ticket open with HP and it has been escalated but not solved yet.  They have me doing all the work of tearing apart my old servers that work well and ravaging them for parts to test each component.  Fun, huh.  Just what I wanted to be, a qa tester for HP.

-Shane

[Robert Egloff]

I ran into an issue on a ML350 G6 where one of the logical drives performance was just horrible. I downloaded a program called 'HD Tune' and checked - the array was performing worse than my laptop's hard disk.

No errors in SNMP, IML, etc.

After further testing, I found a single drive was performing very badly - pulled it out of the array and it went from maybe 15 MB/Sec to upwards of 140 MB/Sec or more.

Might want to check that. Again - diagnostics reported everything was good, the windows permon did as well - but it only really measures the OS's performance relative to itself (perfmon). So get some kind of benchmark program or other I/O performance test for disk drives.

[shanedawg]

Looks like my last post was edited by Rashmi.  Not sure why it was edited or what was changed exactly but that's weird.

Since my last post I overwrote the ubuntu 10.04 OS with Ubuntu 11.04 and the drive array performance went up substantially.  Instaed of getting ~450 random reads/sec I got 1600!!  Wow.  I have narrowed down the issue to the dirver that is used for the array controller card.  Ubu 10.04 uses the cciss driver and 11.04 uses a newer driver called hpsa.  The new driver is the one difference I can think of that may have made the difference.  HP is now trying to replicate our setup in a lab to test it and see what they can do.  Since 11.04 is not supported by them I don't really want to run it.

[shanedawg]

So HP setup a similar system in their lab.  They ran 10.04 on it and got even worse performance than we did.  Their random reads were about 350 and regular reads and writes were very slow. 

Then they called me to explain that they don't support Ubuntu.  What???

After working with them for 5 weeks on the issue, handholding their tech support through every step of the lab setup, waste a ton of time helping to convince them they have a performance issue with their driver they just abandon us.  This is sooooo unacceptable.

At this point we are investigating returning these servers and buying dell.