Very slow write speed on MSA20

[Marco DV]

Hi, I'll try a different approach to a problem I recently talked about here.

I have an HP StorageWorks MSA20 array which is connected to a Smart Array 6400 controller inside an HP ProLiant DL380 G4 server.

The problem which was noticed, and which I confirmed, is that disk access to the array can be VERY slow.

Based on my tests, read speed is OK, and the problem is only with write operations.

e.g.: "hdparm -tT /dev/cciss/c1d0" shows results around 100 MB/s under normal conditions; when doing simple tests instead like "dd if=/dev/zero of=./testfile" I can see that writing starts fast, but begins crawling as soon as the amount of data written reaches 800-900 MB, and then I get terrible transfer speeds between 5 and 20 MB/s.

If I stop the operation, disk activity continues for minutes, slowing down any other disk access.

The write cache batteries were old and needed replacement, so they have been replaced now - all four of them, two in the server and two in the array. And they are fully recharged now.

Using ACU CLI I enabled the write cache on the MSA20, while I can't enable it on the 6400 controller, which maybe is normal, according to what someone else replied in the other thread, because there is no disk connected directly to it.

The installed OS is Fedora Core 5.

All disks in the array are 250 GB models, subdivided into 3 logical drives of around 700 GB each, in RAID 5 configuration.

The firmware in the controller should need an update, according to what ACU CLI tells me, but I doubt it can really fix the problem.

Also one strange thing is that the ACU graphical interface "crashes" when I select the MSA20, the server instantly reboots. I?m using the latest ACU offline CD I found on the HP web site.

Does anyone have anything else to suggest? Any ideas about possible causes for this problem?

Thank you very much.

Marco

P.S. This thread has been moved from ProLiant Storage Systems to Storage Area Networks (SAN) (Small and Medium Business). - Hp Forum Moderator

[Torsten.]

First I would update the firmware (see the scsi bus downshift problem for example):

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=329290&prodSeriesId=1132643&swItem=MTX-d953e02acf304fa4bc2c893b8b&prodNameId=378442&swEnvOID=1005&swLang=8&taskId=135&mode=5

[Marco DV]

Thank you Torsten,

a firmware upgrade was planned, although I was not expecting it to help much, because as far as I've been told the speed was good in the past with the old firmware.

Anyway I had the opportunity to do the upgrade some days ago, and then executed more tests, mostly by transferring or creating 2GB files.

Now the situation looks much better when compared to old tests, but there's still something strange:

• write speed on the first logical drive, which takes up HDs in bays 1-2-3-4, is very slow: 10 MB/s
• write speed on the second logical drive (bays 5-6-7-8) is very good: 90 MB/s
• write speed on the third logical drive (bays 9-10-11-12) is average: 40 MB/s

What can be the cause for this difference? As far as I can tell the configuration is the same for all drives, and there is no rebuilding going on.

Plese note that those speeds have been confirmed by repeating the tests more times.

In case it can help, here is the full report from ACU after the firmware upgrade:

http://dl.dropbox.com/u/40900343/ADUReport.txt

Thank you.

[Torsten.]

From the report:

Smart Array 6400 in slot 2 Warning  The cache for this controller is not configured

This could be the reason!

But maybe I did not find the right information within the file, could you create the ADU report in XML format?

[Marco DV]

As I wrote before, I already noticed that warning, but ACU CLI tells me that the cache for the 6400 controller cannot be enabled in this configuration.

Someone else suggested that this is because no disk is directly connected to the controller, which I believe is correct. Only the controller inside the MSA20 is responsible for the write cache.

I have the report in HTML/XML, I just thought the TXT one was more immediate to view in the browser, sorry.

Here is an archive containing the other format: http://dl.dropbox.com/u/40900343/adu-report.zip

Thanks for your help.

[Torsten.]

Sometimes it is hard to find a thread again ...


If I understand right you see slow writes - or slow reads too?

Unfortunatelly I cannot see a problem, all reported problems are at least 3 years old. It looks like the array just cannot write faster to the RAID5's. Looks like the 6400 in the server has a larger (unused) cache than the MSA has, it is worth to try to swap them to give the array more cache. Google btw. found a page that recommends a stripe size of 128k. Maybe this helps, maybe not, maybe there is another problem somewhere else ...

[Marco DV]

Er... I mentioned at the start of this same thread that the cache on the 6400 cannot be enabled ;).

I only see slow writes. Read speed is excellent.

Are you saying that cache modules in the 6400 and the MSA20 are of the same size? Can they be swapped without problems?

This could be an idea, thanks.

Anyway I reached a situation where I think the performance is satisfying. Of the three RAID5 arrays which have been created in the MSA20, only the first one has a very slow write speed, and I cannot understand why.

The third one has average write speed, while the second has excellent speed.

I can see no reasons for this difference. But I can live with it at the moment.

[Torsten.]

>> I mentioned at the start of this same thread that the cache on the 6400 cannot be enabled ;).

I know, I told you.

http://h30499.www3.hp.com/t5/Proliant-Storage-Systems/Cannot-Enable-Write-Cache-on-Smart-Array-6400/m-p/5578197#M2499

I think the cache modules are the same (i did not verify that) but have a different memory size, so you can try to swap them. Put the larger module into the MSA.

Does each of the logical drives has the same stripe size?

[Marco DV]

Ooops!

So it looks like it's me, who needs to read threads more carefully ;).

I do not know if I will be able to check the stripe size soon on site. But if it is what the ADU report calls "Block size" for the logical drives, then yes, they all have 512 bytes per block.

[ryanwilliams]

Hi Macro DV

I'm so please i stumbled across your post.
I too am experiencing write speeds of 10MB/s to a RAID5 array in an MSA20 connected to a 6400 series controller.
I'm using a 5 drive RAID set in bays 1-5.

Did you ever get to the bottom of the poor write performance?
Do you have any additional test data?

I'm considering running tests on the same drive housed in different bays in an attempt to identify the bottleneck.