topic Re: P400 array performance in ProLiant Servers (ML,DL,SL)

P400 array performance

juice99 — Thu, 13 May 2010 08:36:53 GMT

i have problems with P400 performance (BBU, with write cache turned on)

according to iostat this is how ot looks:

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d1 434.40 0.60 197.60 8.80 3624.00 103.20 36.12 151.98 298.16 4.84 100.00

so the utilization is 100% even though it uses only 3624KB/s

this is extremely poor, can you help me solving this issue?

=> ctrl sn=PAFGL0R9SX723L show config detail

Smart Array P400 in Slot ATTR_VALUE_SLOT_UNKNOWN
Bus Interface: PCI
Slot: Slot Unknown
Serial Number: PAFGL0R9SX723L
Cache Serial Number: PA82C0J9SX81G3
RAID 6 (ADG) Status: Enabled
Controller Status: OK
Chassis Slot:
Hardware Revision: Rev E
Firmware Version: 5.22
Rebuild Priority: Medium
Expand Priority: Medium
Surface Scan Delay: 15 secs
Post Prompt Timeout: 0 secs
Cache Board Present: True
Cache Status: OK
Accelerator Ratio: 100% Read / 0% Write
Drive Write Cache: Enabled
Total Cache Size: 256 MB
Battery Pack Count: 1
Battery Status: OK
SATA NCQ Supported: True

Array: A
Interface Type: SATA
Unused Space: 0 MB
Status: OK
MultiDomain Status: OK

Logical Drive: 1
Size: 465.7 GB
Fault Tolerance: RAID 1
Heads: 255
Sectors Per Track: 32
Cylinders: 65535
Stripe Size: 128 KB
Status: OK
MultiDomain Status: OK
Array Accelerator: Enabled
Unique Identifier: 600508B10010523953583732334C0002
Disk Name: /dev/cciss/c0d0
Mount Points: /boot 94 MB
Logical Drive Label: A009030APAFGL0R9SX723LA042
Mirror Group 0:
physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SATA, 500 GB, OK)
Mirror Group 1:
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SATA, 500 GB, OK)

physicaldrive 1I:1:1
Port: 1I
Box: 1
Bay: 1
Active Path:
Port: 1I Box: 1 Bay: 1
Status: OK
Drive Type: Data Drive
Interface Type: SATA
Size: 500 GB
Firmware Revision: GK6OA70M
Serial Number: GTF400P6G26M7F
Model: ATA Hitachi HUA72105
SATA NCQ Capable: True
SATA NCQ Enabled: True
PHY Count: 1
PHY Transfer Rate: 1.5GBPS
physicaldrive 1I:1:2
Port: 1I
Box: 1
Bay: 2
Active Path:
Port: 1I Box: 1 Bay: 2
Status: OK
Drive Type: Data Drive
Interface Type: SATA
Size: 500 GB
Firmware Revision: GK6OA70M
Serial Number: GTF400P6G251LF
Model: ATA Hitachi HUA72105
SATA NCQ Capable: True
SATA NCQ Enabled: True
PHY Count: 1
PHY Transfer Rate: 1.5GBPS

Array: B
Interface Type: SATA
Unused Space: 0 MB
Status: OK
MultiDomain Status: OK

Logical Drive: 2
Size: 6.4 TB
Fault Tolerance: RAID 5
Heads: 255
Sectors Per Track: 32
Cylinders: 65535
Stripe Size: 64 KB
Status: OK
MultiDomain Status: OK
Array Accelerator: Enabled
Parity Initialization Status: Initialization Completed
Unique Identifier: 600508B10010523953583732334C0003
Disk Name: /dev/cciss/c0d1
Mount Points: None
Logical Drive Label: A01153D1PAFGL0R9SX723LBA86

physicaldrive 1I:1:3
Port: 1I
Box: 1
Bay: 3
Active Path:
Port: 1I Box: 1 Bay: 3
Status: OK
Drive Type: Data Drive
Interface Type: SATA
Size: 1000.2 GB
Firmware Revision: 03.00C05
Serial Number: WD-WMATV1162820
Model: ATA WDC WD1002FBYS-0
SATA NCQ Capable: True
SATA NCQ Enabled: True
PHY Count: 1
PHY Transfer Rate: 1.5GBPS
physicaldrive 1I:1:4
Port: 1I
Box: 1
Bay: 4
Active Path:
Port: 1I Box: 1 Bay: 4
Status: OK
Drive Type: Data Drive
Interface Type: SATA
Size: 1000.2 GB
Firmware Revision: 03.00C05
Serial Number: WD-WMATV1162723
Model: ATA WDC WD1002FBYS-0
SATA NCQ Capable: True
SATA NCQ Enabled: True
PHY Count: 1
PHY Transfer Rate: 1.5GBPS
physicaldrive 1I:1:5
Port: 1I
Box: 1
Bay: 5
Active Path:
Port: 1I Box: 1 Bay: 5
Status: OK
Drive Type: Data Drive
Interface Type: SATA
Size: 1000.2 GB
Firmware Revision: 03.00C05
Serial Number: WD-WMATV1175246
Model: ATA WDC WD1002FBYS-0
SATA NCQ Capable: True
SATA NCQ Enabled: True
PHY Count: 1
PHY Transfer Rate: 1.5GBPS
physicaldrive 1I:1:6
Port: 1I
Box: 1
Bay: 6
Active Path:
Port: 1I Box: 1 Bay: 6
Status: OK
Drive Type: Data Drive
Interface Type: SATA
Size: 1000.2 GB
Firmware Revision: 03.00C05
Serial Number: WD-WMATV1164814
Model: ATA WDC WD1002FBYS-0
SATA NCQ Capable: True
SATA NCQ Enabled: True
PHY Count: 1
PHY Transfer Rate: 1.5GBPS
physicaldrive 1I:1:7
Port: 1I
Box: 1
Bay: 7
Active Path:
Port: 1I Box: 1 Bay: 7
Status: OK
Drive Type: Data Drive
Interface Type: SATA
Size: 1000.2 GB
Firmware Revision: 03.00C05
Serial Number: WD-WMATV1230181
Model: ATA WDC WD1002FBYS-0
SATA NCQ Capable: True
SATA NCQ Enabled: True
PHY Count: 1
PHY Transfer Rate: 1.5GBPS
physicaldrive 1I:1:8
Port: 1I
Box: 1
Bay: 8
Active Path:
Port: 1I Box: 1 Bay: 8
Status: OK
Drive Type: Data Drive
Interface Type: SATA
Size: 1000.2 GB
Firmware Revision: 03.00C05
Serial Number: WD-WMATV1178981
Model: ATA WDC WD1002FBYS-0
SATA NCQ Capable: True
SATA NCQ Enabled: True
PHY Count: 1
PHY Transfer Rate: 1.5GBPS
physicaldrive 1I:1:9
Port: 1I
Box: 1
Bay: 9
Active Path:
Port: 1I Box: 1 Bay: 9
Status: OK
Drive Type: Data Drive
Interface Type: SATA
Size: 1000.2 GB
Firmware Revision: 03.00C05
Serial Number: WD-WMATV1167602
Model: ATA WDC WD1002FBYS-0
SATA NCQ Capable: True
SATA NCQ Enabled: True
PHY Count: 1
PHY Transfer Rate: 1.5GBPS
physicaldrive 1I:1:10
Port: 1I
Box: 1
Bay: 10
Active Path:
Port: 1I Box: 1 Bay: 10
Status: OK
Drive Type: Data Drive
Interface Type: SATA
Size: 1000.2 GB
Firmware Revision: 03.00C05
Serial Number: WD-WMATV1179082
Model: ATA WDC WD1002FBYS-0
SATA NCQ Capable: True
SATA NCQ Enabled: True
PHY Count: 1
PHY Transfer Rate: 1.5GBPS

this is from hpacucli. any idea what can be the cause? This is server supported by HP i guess (leaseweb server) so the configuration i assume is also supported...

Re: P400 array performance

TTr — Thu, 13 May 2010 09:37:46 GMT

I see a few problems.

1. There is no write cache configured. From the "show config"
Accelerator Ratio: 100% Read / 0% Write

2. You are using SATA drives. (Actually SATA-I at 1.5Gbps)

3. The second array, arrayB, has 8 1TB drives in it. And it is RAID5 and has about 7TB of disk space. What kind of i/o do you demand of this? Usually the larger the amount of disk space the more i/o you expect to pump from it.

4. With 2 disks in arrayA and 8 disks in array10, this is probably an external enclosure. If so, you have only one connection (4 ports) from the P400 to this enclosure. That cuts down the i/o throughput capability of the P400 to half.

The configuration is valid but it may not be appropriate for your environment. Don't expect miracles from the raid cards. Just because they support many drives does not mean they can keep up with all the demand that comes alon with a large numner of drives and large amounts of data.

Re: P400 array performance

juice99 — Thu, 13 May 2010 09:56:36 GMT

1. i just changed it before posting . it was 75%/25% etc. it didn't change anything.

2. i am, but the performance of 3.5MB/s for 8 drives array for sure is not limited by the drives being SATA I.

3. it is around 40% full, this is hosting server.

4. does it explain 3.5MB/s performance? i mean, i'm not expecting miracles, but 3.5MB is EXTREMELY low.

people have 50 or 150MB/s performance of their arrays, 3.5MB is below anything i've seen.

Re: P400 array performance

TTr — Thu, 13 May 2010 10:53:33 GMT

You did not comment on the connectivity of the P400 and the drives. As I mentioned it looks like an external enclosure.
I am not saying this is exactly the cause of your problems but it can definitely contribute to them.

Now where is the 3.5MB/s realized? Is it at the array level or the OS level? What other factors at the OS level might be contributing to this?

Was this performance always like that or it just started happening? You mentioned that the 100%read/0%write was there previously but you just chaged it so there are changes being made. Have you run some performance testing after this change? What server is this on and what are some other characteristics of the environment, OS, apps, databases?

The iostat you provided is for the 8-disk raid5 LUN, what about iostat on the boot disk? Do you get the same i/o results?

Re: P400 array performance

Patrick Terlisten — Fri, 14 May 2010 07:37:52 GMT

Hello,

do you use a HP ProLiant server or a non-ProLiant server?

Regards,
Patrick

Re: P400 array performance

CLEB — Fri, 14 May 2010 08:04:09 GMT

Perhaps a FW upgrade wouldn't hurt?

Ver 5.22 is over a year old. I think there was a recommended minimum version around V6.

Re: P400 array performance

juice99 — Fri, 14 May 2010 08:19:19 GMT

i use Proliant DL180 G5

i will suggest my company (leaseweb) to upgrade array firmware as i think i should not do it by myself

i didn't comment on connectivity, because i actually don't know. it's Leaseweb server, i'm just renting it, i don't know what are they practices.

Should i tell them to connect it differently? what exactly should i tell them?

i made changes only after problems started happening, not before

this is on OS level, and of course you are right. other important factors: millions of files on this hosting, extremely random read pattern.

maybe the array is not correctly created? chunk size too small/too big ?

is there any way to check if this is P400 slowing it down?

and yes, performance was not much better before, but it was better. so utilization was 70%-80%, but why it shoot up to 100% even though my traffic did not is very strange. (number of connections stable, number of webserver requests stable, no DoS attack, everything pretty much the same, only load average shoot up to 200 sometimes and simple things like touch or rm on array takes many seconds)

boot disk is not overloaded at all:

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
cciss/c0d0 1.42 50.63 17.29 12.30 199.19 250.20 30.38 0.95 32.19 2.08 6.15
cciss/c0d1 460.52 31.71 184.62 9.37 3829.72 164.68 41.18 109.66 565.08 4.88 94.60
dm-0 0.00 0.00 17.73 63.00 198.44 251.98 11.16 2.93 36.25 0.76 6.14
dm-1 0.00 0.00 0.03 0.00 0.13 0.00 8.00 0.00 5.23 0.44 0.00

i don't know how well you can see this,but boot disk is pretty much idle

this is free hosting server, it doesn't run database for users (only for hosting panel) , but other than that it runs everything. so ftp, webserver, dns, mail etc.

is was working fine without any problems so far, it is nicely optimized. my traffic is actually lower now because of the problems, but this high load and crashes still remain

Re: P400 array performance

juice99 — Fri, 14 May 2010 08:25:30 GMT

P.S. from what i know, these disks are internally located. any idea how to check it to be sure?

i have no multipather installed, so i guess that means disks are internal, right?

Re: P400 array performance

TTr — Fri, 14 May 2010 10:54:40 GMT

The DL180G5 comes with 4, 8 or 12 disks so the disks could be internal or external. http://h18000.www1.hp.com/products/quickspecs/12903_div/12903_div.HTML

The P400 has only 8 ports so there is definitely a SAS expander involved.

The question still remains why the jump from 70-80% to 100%. In some cases it takes very little to throw of these arrays. Did the amount of data in the LUN increase at all?

Also have you rebooted the server recently? It might help. Disk defragmentation at the OS level might help as well. A defragmented disk makes the cache less and less useful.

Do you use all the 6.4TB of the second LUN?

Re: P400 array performance

CLEB — Fri, 14 May 2010 10:59:31 GMT

The DL180G5 is a 2U server that has a maximum of 12 drive slots.

Re: P400 array performance

TTr — Fri, 14 May 2010 11:55:30 GMT

> me: A defragmented disk makes the cache less and less useful.
That should read "A fragmented disk makes the cache less and less useful."