HP StorageWorks MSA Storage P2000 G3 iSCSI: very poor reading performance

[ciopsa]

Hello, 

this post was actually posted yesterday in NAS forum and some friends suggested  me to post also here:

Hello all,

we are experiencing quite poor reading performance (order of 15 Mb/sec !!!)  on all the volumes/vdisk we created on a  HP StorageWorks MSA Storage P2000 G3 iSCSI.

Here some details: 

The HW is composed by four chassis for a total of 48 disks.  

Disks are of two kinds:

12 600GB disk 15k RPM  ST3600057SS in the first enclosure 

36 2TB disks 7.2K RPM   ST32000444SS  in the 3 other enclosures.

The storage has two controller for a total of four 10gB isci port connected to a 10gigabit switch and then connected to four i/o Linux server (iosrv[1-4]).

The configuration we setup is attached here in the picture (as png file: hope it is ok) 

On the storage we created 5 different vdisks (one raid6 for the 600GB disks and 4 for the 2TB disks (raid 5 with  global spare).Here below the details: 

# show vdisks
Name Size Free Own Pref RAID Disks Spr Chk Status Jobs Serial Number Drive Spin Down Spin Down Delay
--------------------------------------------------------------------------------------------------------------------------------------------------------
vd01 5995.1GB 3522.5KB A A RAID6 12 0 64k FTOL 00c0ff1153d10000f74e844e00000000 Disabled 0
vd03 15.9TB 704.5KB B B RAID5 9 0 64k FTOL 00c0ff1154c50000c8815b4e00000000 Disabled 0
vd04 15.9TB 704.5KB B B RAID5 9 0 64k FTOL 00c0ff1154c5000042825b4e00000000 Disabled 0
vd05 15.9TB 704.5KB B B RAID5 9 0 64k FTOL 00c0ff1154c5000076825b4e00000000 Disabled 0
vd06 13.9TB 983.0KB B B RAID5 8 0 64k FTOL 00c0ff1154c5000073eb794e00000000 Disabled 0
-------------------------------------------------------------------------------------------------------------------------------------------------

On each vdisk we just created one single  volume:

# show volumes
Vdisk Name Size Serial Number WR Policy Cache Opt Read Ahead Size Type Class Qualifier Volume Description
------------------------------------------------------------------------------------------------------------------------------------------------------------------
vd01 fast1_iosrv2 5995.0GB 00c0ff1153d10000b64f844e01000000 write-back standard Default standard standard N/A
vd03 slow1_iosrv1 15.9TB 00c0ff1154c50000530c654e01000000 write-back standard Maximum standard standard N/A
vd04 slow1_iosrv2 15.9TB 00c0ff1154c50000e40d654e01000000 write-back standard Maximum standard standard N/A
vd05 slow1_iosrv3 15.9TB 00c0ff1154c50000e3db644e01000000 write-back standard Maximum standard standard N/A
vd06 slow1_iosrv4 13.9TB 00c0ff1154c50000e5ec794e01000000 write-back standard Default standard standard N/A
-------------------------------------------------------------------------------------------------------------------------------------------------

Writing on any of the volumes from I/O server we measured around 350Mb/secs that we consider satisfactory.

Reading from them we measured no more than 15Mb/sec !, no matter which combination of cache parameters we choose.

Measures were performed by  dd command  on 50giga files just to avoid cache problems on the i/o servers ( each of the them have 24GB ram).

Investigating the problem we discovered that firmware on the device is one year old:

# show versions

Controller A Versions
---------------------
Bundle Version: TS210R016
Build Date: Mon Aug 30 18:59:22 MDT 2010

Controller B Versions
---------------------
Bundle Version: TS210R016
Build Date: Mon Aug 30 18:59:22 MDT 2010

(even if the system was delivered in july 2011) and we plan to upgrade it.  

Yet we still  think we cannot just blame the old firmware for such a problem.

Is there anybody in the forum who can give us some insight  ?

Is there something dramatically wrong on our setup ? 

Please advice

thanks in advance !

Stefano&friends 

[Johan Guldmyr]

Hi, dd is not recommended to test performance.

Two examples: IOR or iometer.

http://ior-sio.sourceforge.net/
http://www.iometer.org/

If you want to see if there are any performance improvements in the firmwares: read the release notes.

All the slow volumes are owned by Controller B - intentional?

[ciopsa]

Hello,

thanks for the reply.

Here some comments:

we used several other tools (like iozone for instance or) and numbers are consistent: 

reading is is the order of  few megabytes per second no matter which tool we are using..

Bad performance are in any case confirmed by our users that send us such simple test:

du /lustre/projects/genome/casonato/homo_h19_cs_14.bin

15230340  /lustre/projects/genome/casonato/homo_h19_cs_14.bin

 /usr/bin/time cp /lustre/projects/genome/casonato/homo_h19_cs_14.bin homo_h19_cs_14.bin

0.01user 108.29system 16:58.30elapsed 10%CPU (0avgtext+0avgdata 11392maxresident)k

0inputs+30460672outputs (0major+760minor)pagefaults 0swaps

= ~14 MB/s

We read carefully release notes; nothing is said about performance.  

Regarding slow volumes ownen by controller-B: yes it is (was) intentional: originally we though to dedicate controller A for the 12 fast disks (6 disks for volume) to setup a faster partition.... 

I will post soon a more detailed analysis with some more tests we are currently performing to understand this issue.

Stefano

[Johnc_3]

Hi Stefano,

You may be interested in the performance tool for the P2000:

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&swLang=8&mode=2&taskId=135&swItem=MTX-5e878f833e2c4d078c0e8368bd

I know it's only v1.0.0 but it may help your investigation.

John

[ciopsa]

Hello John,
thanks for the suggestion. We will try the tool as soon as we update the firmware.

In the meantime we tried to better understand the problem in the following way:

We just delete one vdisk (the one composed by 12 disks) and created three different vdisks: two formed by one single disk and one formed by 6 disks n RAID5.
The idea is is to measure which kind of performance numbers we can get out from one single disks and then compare against raid5 volume.

Here the details of the situation:
#show vdisks
Name Size Free Own Pref RAID Disks Spr Chk Status Jobs Serial Number Drive Spin Down Spin Down Delay
------------------------------------------------------------------------------------------------------------------------------------------------------------
vd_noraid 599.5GB 99.5GB A A NRAID 1 0 N/A UP 00c0ff1114220000293f9d4e00000000 Disabled 0
vd_noraid2 599.5GB 99.5GB A A NRAID 1 0 N/A UP 00c0ff111422000016559d4e00000000 Disabled 0
vd_raid5 2997.5GB 997.5GB A A RAID5 6 0 64k FTOL 00c0ff111422000087fa9e4e00000000 Disabled 0

# show volumes
Vdisk Name Size Serial Number WR Policy Cache Opt Read Ahead Size Type Class Qualifier Volume Description
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
vd_noraid noraid_volume 499.9GB 00c0ff1114220000d83f9d4e01000000 write-back standard Default standard standard N/A
vd_noraid2 noraid_volume2 499.9GB 00c0ff11142200005b559d4e01000000 write-back standard Default standard standard N/A
vd_raid5 raid5_volume 1999.9GB 00c0ff1114220000e4fa9e4e01000000 write-back standard Default standard standard N/A

On the three disks above on our i/o server we run the sgpdd_survey tools coming from lustre_iokit.12 (our ultimate goal is to setup a Lustre FS on the the top of our P2000) and this give us the following picture:

Disk Threads Write Write x thread Read Read x thread
Sde 2 109 2 x 54.90 106 2 x 53.04
2 109.79 2 x 54.90 14.49 2 x 7.25
4 109.79 4 x 27.45 50.44 4 x 12.61
Sdf 2 110.28 2 x 55.14 116.51 2 x 58.26
2 109.94 2 x 54.97 50.57 2 x 25.29
4 110.28 4 x 27.57 74.60 4 x 18.65
Sdh 2 353.58 2 x 176.86 187.64 2 x 93.82
2 412.41 2 x 206.22 11.58 2 x 5.79
4 449 2 x 112.41 10.56 4 x 2.64

Please note the awful reading performance on the RAID5 device with four threads !

Here the command line we used;

scsidevs=/dev/sd[efh] rslt=/tmp/sde.sgpdd-survey size=26000 crglo=2 crghi=256 thrhi=4 /opt/tmp/lustre_iokit/lustre-iokit-1.2/sgpdd-survey/sgpdd-survey

Finally we created on each device an ext3 filesystem and run iozone on the top to check the performance by means of standard i/o benchmarking tools. Here the command line and below again the numbers:

/opt/iozone/iozone -t 1 -i 1 -i 0 -s 32g -r 1M

Device Write Re-Write Read Re-read
Sde 122 117 49 35
Sdf 118 119 35 -
Sdg 75 72 31 32
Sdh 210 197 21 26

Writing performance are almost ok (even if it drops significantly on the RAID5 system..). The thing that worries us is however to see the really poor performance on reading on the RAID5 device: order of 20Mb/sec !!

Any clue ?

thanks
Stefano

[Gian Luigi De Lorenzo_1]

hi Stefano,

we have the same problem using a P2000G3: poor read performances, about 1/3 of write performances !!

we are testing the storage with sgpdd and xdd.

did you find any solution to your problem in the meanwhile ??

Gianluigi

[Jan Soska]

Hello Stefano,

reading 15MB/s is really strange. is your read test random or sequential?

We saw similar problem even on FC arrays under linux, problem was in linux IO scheduler.

Have you tested your disks under windows?

We fixed our random read problem by changing default linux io scheduler method.

Jan

[Dileep P S]

Hi,

Please make sure that you are following iSCSI best practices

[DarrenStokes]

Hi Stefano,

Did you ever resolve this issue.  Have similar performance problem with P2000 .

Thanks

Darren

[hlmaldon]

I am experiencing this exact issues, however this post has no solutions, do you actually find any solution to the slowness problem?

I been trying lot of things, including all suggested here without success.

This thread is old but the issue is the same. Worth a try if someone got to fix it.