HP MSA2012i terrible throughput

[sean anderson]

I have a backup exec server that mounts a snapshop of various file servers that have volumes on the MSA2012i and the throughput is literally 10-20MB/Min.

The backup exec server is running windows 2008 x64 and I see that people are saying this OS is not supported so the question is if I put a Qlogic HBA in the server will this be supported configuration and if the slowness continues what could the issue be?

[Uwe Zessin]

It is normal to see low backup throughput on a server with lots of small files. The disk drives need to push their heads to position between data and meta-data during which no data transfer is possible.

[sean anderson]

I understand backup times are will be slower when dealing with many small files but I dont think literally 1MB/Sec is reasonble. We also run a windows storage server with the falconstore iscsi product as well as an instransa ISCSi solution and they have speeds during backups approaching 80X this speed I see with the MSA2012i.

Any other suggestions with regards to what I may not being doing? I cannot believe this is such a difficult problem to solve.

some information, our backup server is an ML350G5 with 12GB RAM, 1 quad core 2.5Ghz CPU and multiple nics inlcuding NC380T and NC382T. Everything has the latest firmwware and drivers. The SAN network is Procurve 2848 switches and cabling is all cat 6.

[Uwe Zessin]

I'm sorry, I have missed the "per Minute". That is really slow.

Before buying any hardware, have you tried to backup without snapshots to see if this is limiting the speed? The MSA2012i also allow some I/O monitoring - maybe the snapshots do produce too many addition I/Os. The speed on the file servers is OK during the backup?

I don't have any experience with random I/O on the MSA2012i, but I know it can do sequential quite well. What you could try is load the system with massive large, sequential read or write I/Os and look at the throughputs - maybe there is a problem in the network infrastructure which is visible with seq. I/O as well.

[sean anderson]

Thank you for the response. I suppose I should use a tool like IO meter to measure the IO's just to be sure I understand what is going on. As best as I can tell the fileserver behaves normally during the backup.

Thanks

[Kevin Hand]

There seems to be a major issue with performance on the MSA 2012i. I posted a detailed entry onto this forum as I have an outstanding support issue with HP on this very subject.

I have just checked and my posting has disappeares.

There are a number of other posting on this forum regading poor performance and BXOIS errors in the system event log

[Bart_Rajchel]

I have a similar issue. We are using a HP DP6.0 for backups and have exactly the same problem. I can also say that copying files from the MSA2012i over a gigabit LAN is also 10-18MB/s it's not only backup. I can copy from other storage technologies with much higher speed then from MSA2012i. I was told by a HP storage rep that HP does not deal with performance issues. Funny we only have HP equipment (servers. switches, LTO4 drives HP software). Time to switch vendors.

[sean anderson]

Bart,

I have noticed the same thing. Transfer speeds are generally quite poor but in our case the backups are dismal. I thought it was related to Windows 2008 iSCSI initiator software so I installed qlogic 4062C HBA's but the problem is still quite unsatisfactory.

[Kevin Hand]

I have had a support call placed with HP since 11th Feb and have sent email after email requesting assistance and have received very little response from HP at all

[Bart_Rajchel]

This is extremely upsetting. How can HP do this? O I know. They know about the problems but can't figure it so I guess it's easier to ignore it.

[Robert Sumners]

Thought I would throw in my two cents. We've been fighting the MSA2012i box for 8 months now. HP support is terrible and unresponsive. We are also having slow backup speeds with it and when backing up, our cluster resources (disks) will go offline. Would never buy an HP SANS again.

[Al Lopez]

I was curious if you could post more information. I am considering the 2012i bundle and wanted to see how you were configured.

Did you use a VLAN for your iSCSI traffic?
What cluster size are you using for the vdisk?

If this is a systemic issue with the 2012i, I won't order it.

Just curious if more people are dissatisfied than happy with this model.

Thanks!

[Bart_Rajchel]

Don't look at the MSA line of storage. This storage solution is nothing but problems. We even had CDW and HP design, quote and approve this configuration and now we have about 200lb of modern looking dust collectors.
Here's an idea. I bet we can cluster one hundred of our old RW CD-ROMs sitting in the storage room and have better performance then MSA2012i. I bet Linux community can write some app for that!

[ZeroAdmin]

Been having issues too. Backup Exec 12.5, Hyper V environment with Windows 2008 (POS) , G5 server with the 2012i and HP 6000 tape library.

Been seeing some chatter about the HP's nic teaming software and iSCSI.

You may want to try them un-teamed to see what happens (if they are).

Regards...

[sean anderson]

Thanks for the information. I am not teaming adapters under Windows 2008 anymore. The result of no teaming has lead to any performance increase overall.

Strangley, I have discovered that I can do a file level backup via the host instead of through snapshots at 2.5GB/Min. I dont understand how file based backup are faster than block based but since it is effective I have been using it.

I have no confidence that HP will help me anymore. It is a shame when you consider I have been a great advocate of HP for the past 12 years...

I am now looking at eating crow and committing to a much more expensive solution from EMC or NetAPP. I need the performance and it does not seem that anyone offers good performance at the MSA price point.....

[Dennes Meeusen]

Did you try loading up the latest (april) firmware in your msa2012? Performance increase in our iSCSI msa2012i was very noticeable when we applied the latest f/w

IMHO the 2012 is a good device, considering it's price and targeted audience. Comparable SAN's from Equallogic/Dell or Netapp cost about twice as much. If you really want high end performance, consider the EVA line

[sean anderson]

Dennes,

I dont see an april firmware update. Can you give the URL for this update?

I agree in principle with what you are saying. There are some very rough edges that HP really should smooth out to make it more fully functional for those who purchase the device.

[Dennes Meeusen]

This should be it (for the iSCSI version, not the FC)

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=12169&prodSeriesId=3687128&swItem=co-69015-1&prodNameId=3687132&swEnvOID=4025&swLang=13&taskId=135&mode=4&idx=0

[Dennes Meeusen]

Btw. When Googling you'll find that there seem to be lots of problems with iSCSI in Windows Server 2008. You might ant to test/check with 2003 and see if throughput is any better.

[sean anderson]

Thanks Dennes,

I do have this firmware. It did make a difference in overall performance but I still have major issues with performance.

I have tried testing with Windows 2003 and do some some improvement over 2008. However, the performance is still not acceptable.

[Oto Novotný]

Hi There,
my config:
-cluster 2xHP DL360G5
-MSA2012i
-W2008/64 Hyper-V, 25 Virtual machines

I had the same problem with very poor speed, solution for me was:

Upgrade a latest firmware of MSA2012
10-20% more performance + no more iSCSI error messages in event log.

Switch off on HP NIC card (HP NC373i)
-IPv4 Checksum Offload
-IPv4 Large Send Offload
This give me triple-more throughput for standard operations
Backup(Symantec VERITAS 12.5) has double more speed after NIC changes have been implemented.
I have now no issues with the speed of MSA 2012i.
Hope this help :)

[sean anderson]

I just switched back over from a qlogic card to an NC380T so I will try those NIC settings and see what happens.

[Troy Newman]

Did the NIC card settings help? Turning off the IPv4 Checksome Offload and Large Send Offload?
Did you do this on the iSCSI NIC and LAN cards?

[Juan C. Rodriguez]

I have the same issues with a 2x dl360g5 2003 r2 running a sql cluster server, using standalone switches for just 2 servers,,, the performance on copy or backups is terrible,, is there any specific recomendation on the config of the nics or the san to improve this?

[Jeroen H_1]

I had the same problem. I solved it by uninstalling the HP Network Config Utility.
I just set all the settings on the adapter itself and I got from 10-20 to 80-90 MB/s.