Extremely low disk i/o on DL 580 G2

[nishant_sharma_77]

Dear All,

I recently bought 2 refurbished DL 580 G2's on ebay (with 4 x 3.0 GHz Xeon CPU's + 8GB RAM + 4 X 146 GB USCSI HDD's which are configured in a RAID 5 array on a smart array 5i controller with ADG).
The OS on both is windows 2003 enterprise with SP2 and latest critical updates and I used Compaq smart start CD v7.2 to load the OS on it).
These 2 are going to be our primary and stadby DB server but the disk I/O on them is ridiculously low at less then 5Mbps (I also have 2 X Dell poweredge 6950 servers which has 4CPU's + 8GB mem + 5X 300GB USCSI HDD'son perc/5i controller in raid 5 config) and the disk I/O on them is in access of 150Mbps.
I am using MS SQLIO Disk Subsystem Benchmark Tool for measuring disk I/O.
Kindly guide me on how to reconfigure the 2 Proliant Dl580's so that the disk I/O improves on both.

Thanks in advance,
Nishant

[KarloChacon]

hi

my first recommendation you say that you installed them using smartstart 7.2

did you updated firmware and Proliant Support Pack to the latest?


Firmware - CD 7.91
Software - Support Pack 7.91

http://h18023.www1.hp.com/support/files/server/us/locate/69_4081.html

regards

[nishant_sharma_77]

haven't dont that yet, but doing it now.

[James ~ Happy Dude]

Hi Nishant,
Have you Got Battery Backed Write Cache for the Controller ? If not, you might wanna add it too.

Also, you may upgrade to a P400/E200/P800 Controller & Enable the DISK WRITE Cache(of HDD) for ENHANCED disk i/o throughput.(this feature is not present in 5i)
;)
Regards.

[nishant_sharma_77]

Hi Happydude,

How can I enable battery backed cache?
Is it a software setting in the BIOS on via ACU or is it an hardware upgrade?
Moreover adding or buying additional hardware is out of question, I have to improve performance on existing hardware or I am loosing my job for sure tomorrow.
I have just finished downloading the latest firware and driver disks and will be going in our co-location data centre to deploy them. (I don't know how drastically these will improve performance). But if it doesn't, I won't be returning back to office, but rather finding a new job on monday. What a crappy way to approach christmas. May god have mercy on me.

Thanks for your help though.
Nishant

[KarloChacon]

hi

check this maybe you have 5i Plus controller and Battery Backed Write Cache (BBWC) installed.

you can check doing this:

** first you need to install PSP to have Array configuration utility

Setting the accelerator ratio.

The setting that you use for the accelerator ratio feature determines how much of the cache memory is allocated to read-ahead cache and how much to posted-write cache. Different applications have different optimum settings. This setting applies only if the controller uses a battery-backed cache.


To modify the accelerator ratio:
1. In the Logical Configuration View panel, highlight the controller, and then press the Enter key.
The highlight moves to the Controller Options panel.
2. Highlight the Controller Settings option, and then press the Enter key.
The Controller Settings panel appears.
3. Highlight the accelerator ratio setting that you want this controller to use, and then press the Enter key.
4. Press the Esc key to save the settings and return to the Logical Configuration View panel.


regards

[nishant_sharma_77]

Hi Guys,

I upgraded the RAID controller on one of the servers to HP Smart Array 6400 with 192MB RAM and BBWC (installed it in one of the PCI-X slots). Reconfigured RAID 5 + loaded OS
(windows 2003 enterprise with Sp2) using HP smartstart 7.91 and loaded all latest firmware using HP Firmware disk v 7.91. Even now the disk I/O on this server is approx 14Mbps.
I fail to understand what is the bottleneck now.
Another strange problem that has started happening with this controller upgrade is when I restart the system it doesn't come online by itself, I have to manually shut it and then start it manually.
Any help will be appreciated.

Thanks,
Nishant

[~sesh]

If you are using a different hardware configuration and still getting the same performance, then I don't think it is something to do with hardware - except hard drives.

Using the firmware maintenance CD did you upgrade the firmware for the hard drives too? Also, after installing the new controller (6400), please upgrade the firmware for it.

When you reboot the server, does it recognize / initialize the 6400 controller? If it doesn't; try to remove the SCSI cable from the backplane & reboot. This will tell if it is something due to the hard drives / SCSI backplane (if it successfully initializes but doesn't detect any logical drives). If the 6400 controller is initialized, but the logical drive is not detected even when the cable is plugged in, check for the LEDs on the hard drives.

Is this an SQL server? Are there any application specific settings for disk access optimization? In Windows, using Device Manager you can select "Drive cache" option in the properties of the hard drive.

[nishant_sharma_77]

Hi Adric,

Performance isn't the same, i.e. with on board 5i controller the performance was 3-4MBps, after installing 6400 the performance increased to 14-15MBps but it is no where close to the other 2 X Dell Power edge 6950's that I have which give nearly 199MBps throughput.
These DL580's were going to be our SQL servers but my DBA's won't tough them until they see a disk I/O throughput of at least 150MBps (refer some stats from my DL580's and Dell PE 6950's).

Find below SQLIO output from HP Proliant DL580 G2 & Dell Poweredge 6950 servers.

Even Dell Business services is pretty stumped with this issue and can't seem to put a finger on the botttleneck. If all things being equal between HP DL580 and Dell Poweredge 6950, I can only point to the system bus speed, on Dell it is 1000MHz and on HP it is 100MHz.

I am really out of options now and can't seem to think of anything that might be causing such a dismal performance from such a nicely configured server.

Find below SQLIO output from HP Proliant DL580 G2 -
----------------------------------------------------
C:\Program Files\SQLIO>sqlio -kW -s10 -frandom -o8 -b8 -LS -Fparam.txt timeout /
T 60
sqlio v1.5.SG
using system counter for latency timings, -1300737296 counts per second
parameter file used: param.txt
file c:\testfile.dat with 2 threads (0-1) using mask 0x0 (0)
2 threads writing for 10 secs to file c:\testfile.dat
using 8KB random IOs
enabling multiple I/Os per thread with 8 outstanding
using specified size: 100 MB for file: c:\testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 1886.27
MBs/sec: 14.73
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 7
Max_Latency(ms): 146
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 24 19 36 12 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 8

Find below SQLIO output from Dell Power edge 6950's -
----------------------------------------------------
C:\Program Files\SQLIO>sqlio -kW -s10 -frandom -o8 -b8 -LS -Fparam.txt timeout /
T 60
sqlio v1.5.SG
using system counter for latency timings, 2000160000 counts per second
parameter file used: param.txt
file c:\testfile.dat with 2 threads (0-1) using mask 0x0 (0)
2 threads writing for 10 secs to file c:\testfile.dat
using 8KB random IOs
enabling multiple I/Os per thread with 8 outstanding
using specified size: 100 MB for file: c:\testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 25581.20
MBs/sec: 199.85
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 0
Max_Latency(ms): 2
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 100 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Thanks,
Nishant

[Mi6t0]

Ok,

So what is the PSP installed now?
And what is the firmware version of the controller?

Can you try copying some files from the network? Just to see if problem occurs locally or not.

Try changing the stripe size in the Array Configuration Utility... Try to change the read/write ratio. 14 mb for a smart array controller is completely unacceptable...