Very slow write speed on MSA20

[Marco DV]

Hi, I'll try a different approach to a problem I recently talked about here.

I have an HP StorageWorks MSA20 array which is connected to a Smart Array 6400 controller inside an HP ProLiant DL380 G4 server.

The problem which was noticed, and which I confirmed, is that disk access to the array can be VERY slow.

Based on my tests, read speed is OK, and the problem is only with write operations.

e.g.: "hdparm -tT /dev/cciss/c1d0" shows results around 100 MB/s under normal conditions; when doing simple tests instead like "dd if=/dev/zero of=./testfile" I can see that writing starts fast, but begins crawling as soon as the amount of data written reaches 800-900 MB, and then I get terrible transfer speeds between 5 and 20 MB/s.

If I stop the operation, disk activity continues for minutes, slowing down any other disk access.

The write cache batteries were old and needed replacement, so they have been replaced now - all four of them, two in the server and two in the array. And they are fully recharged now.

Using ACU CLI I enabled the write cache on the MSA20, while I can't enable it on the 6400 controller, which maybe is normal, according to what someone else replied in the other thread, because there is no disk connected directly to it.

The installed OS is Fedora Core 5.

All disks in the array are 250 GB models, subdivided into 3 logical drives of around 700 GB each, in RAID 5 configuration.

The firmware in the controller should need an update, according to what ACU CLI tells me, but I doubt it can really fix the problem.

Also one strange thing is that the ACU graphical interface "crashes" when I select the MSA20, the server instantly reboots. I?m using the latest ACU offline CD I found on the HP web site.

Does anyone have anything else to suggest? Any ideas about possible causes for this problem?

Thank you very much.

Marco

P.S. This thread has been moved from ProLiant Storage Systems to Storage Area Networks (SAN) (Small and Medium Business). - Hp Forum Moderator

[Torsten.]

First I would update the firmware (see the scsi bus downshift problem for example):

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=329290&prodSeriesId=1132643&swItem=MTX-d953e02acf304fa4bc2c893b8b&prodNameId=378442&swEnvOID=1005&swLang=8&taskId=135&mode=5

[Marco DV]

Thank you Torsten,

a firmware upgrade was planned, although I was not expecting it to help much, because as far as I've been told the speed was good in the past with the old firmware.

Anyway I had the opportunity to do the upgrade some days ago, and then executed more tests, mostly by transferring or creating 2GB files.

Now the situation looks much better when compared to old tests, but there's still something strange:

• write speed on the first logical drive, which takes up HDs in bays 1-2-3-4, is very slow: 10 MB/s
• write speed on the second logical drive (bays 5-6-7-8) is very good: 90 MB/s
• write speed on the third logical drive (bays 9-10-11-12) is average: 40 MB/s

What can be the cause for this difference? As far as I can tell the configuration is the same for all drives, and there is no rebuilding going on.

Plese note that those speeds have been confirmed by repeating the tests more times.

In case it can help, here is the full report from ACU after the firmware upgrade:

http://dl.dropbox.com/u/40900343/ADUReport.txt

Thank you.

[Torsten.]

From the report:

Smart Array 6400 in slot 2 Warning  The cache for this controller is not configured

This could be the reason!

But maybe I did not find the right information within the file, could you create the ADU report in XML format?

[Marco DV]

As I wrote before, I already noticed that warning, but ACU CLI tells me that the cache for the 6400 controller cannot be enabled in this configuration.

Someone else suggested that this is because no disk is directly connected to the controller, which I believe is correct. Only the controller inside the MSA20 is responsible for the write cache.

I have the report in HTML/XML, I just thought the TXT one was more immediate to view in the browser, sorry.

Here is an archive containing the other format: http://dl.dropbox.com/u/40900343/adu-report.zip

Thanks for your help.

[Torsten.]

Sometimes it is hard to find a thread again ...


If I understand right you see slow writes - or slow reads too?

Unfortunatelly I cannot see a problem, all reported problems are at least 3 years old. It looks like the array just cannot write faster to the RAID5's. Looks like the 6400 in the server has a larger (unused) cache than the MSA has, it is worth to try to swap them to give the array more cache. Google btw. found a page that recommends a stripe size of 128k. Maybe this helps, maybe not, maybe there is another problem somewhere else ...

[Marco DV]

Er... I mentioned at the start of this same thread that the cache on the 6400 cannot be enabled ;).

I only see slow writes. Read speed is excellent.

Are you saying that cache modules in the 6400 and the MSA20 are of the same size? Can they be swapped without problems?

This could be an idea, thanks.

Anyway I reached a situation where I think the performance is satisfying. Of the three RAID5 arrays which have been created in the MSA20, only the first one has a very slow write speed, and I cannot understand why.

The third one has average write speed, while the second has excellent speed.

I can see no reasons for this difference. But I can live with it at the moment.

[Torsten.]

>> I mentioned at the start of this same thread that the cache on the 6400 cannot be enabled ;).

I know, I told you.

http://h30499.www3.hp.com/t5/Proliant-Storage-Systems/Cannot-Enable-Write-Cache-on-Smart-Array-6400/m-p/5578197#M2499

I think the cache modules are the same (i did not verify that) but have a different memory size, so you can try to swap them. Put the larger module into the MSA.

Does each of the logical drives has the same stripe size?

[Marco DV]

Ooops!

So it looks like it's me, who needs to read threads more carefully ;).

I do not know if I will be able to check the stripe size soon on site. But if it is what the ADU report calls "Block size" for the logical drives, then yes, they all have 512 bytes per block.

[ryanwilliams]

Hi Macro DV

I'm so please i stumbled across your post.
I too am experiencing write speeds of 10MB/s to a RAID5 array in an MSA20 connected to a 6400 series controller.
I'm using a 5 drive RAID set in bays 1-5.

Did you ever get to the bottom of the poor write performance?
Do you have any additional test data?

I'm considering running tests on the same drive housed in different bays in an attempt to identify the bottleneck.

[BOYAN BIANDOV]

Hi Marco,

Would you share the details on how you "Using ACU CLI I enabled the write cache on the MSA20"?

I even downloaded the latest version of the ACU CLI but I can't see how to specify that a command would be executved at the MSA controller rather than the physically connected controller inside the sererv?

All I see are commands that execute locally; not on the MSA controller?

Thanks

~B

[ryanwilliams]

Example of modifying a setting on an MSA20

=> ctrl all show

MSA20 in E057MLJ18V (sn: PAAACADMQUUNBP, csn: E057MLJ18V )
Smart Array 6400 in Slot 0 (Embedded) (sn: P57820GDAQ2IYE)
Smart Array P400 in Slot 1 (sn: P61620E9SV46J1)

=> ctrl sn=PAAACADMQUUNBP modify dwc=enable forced

[Marco DV]

Hi all,

sorry for the delay but I hoped to collect more information before replying.

Unfortunately I couldn't. I rarely go to the place where the MSA20 I manage is, and even harder is the possibility to find some spare time there to repeat tests and collect info.

Thank you ryanwilliams for answering about ACU CLI usage.

Indeed, Boyan, you can define a "target" for each command, by using some identification data related to the controller you want to configure. And you can query all controllers to find out how to identify them.

I cannot remember how I did it specifically, but I think I just followed the instructions found in the document called "Configuring Arrays on HP Smart Array Controllers Reference Guide" (google for "c00729544.pdf").

Ryan, as far as I recall I never solved the slow write problem on the first logical drive, and just lived with it. The situation is better than it was initially, anyway, so users can now bear with it. The array and the drives are getting old, and the boss is also evaluating replacing it to get more capacity and performance.

Sorry I can't help.

[BOYAN BIANDOV]

Thank you gentlemen,

I saw from the sample that Marco posted that you are using P600: I have a specific P600 issue. How do you force the controller to either reset the battery health status using the CLI

Or alternatively how do you force P600 to:

(a) recondition the batteries

(b) reset status and do a true electrical detection to either initiate charging or perform full charge test on the batteries

(c) is there a mechanical switch to signal that one has replaced the batteries hence the "battery failed" error is fake and should be reset in order for the true status of the new batteries to be reported?

Thank you

[BOYAN BIANDOV]

CLEAR CONTROLLER CONFIGURATION did the trick. Of course I made sure the SAS cable is unplugged because otherwise it would have whipped out the logical drive. Then the controller finally "saw" the newly replaced batteries and all is well. What a terrible design??

[Torsten.]

Older firmware version had battery related issues solved in newer versions. What firmware do you run?

Latest I found is 2.04A

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=329290&prodSeriesId=466290&swItem=MTX-f931052606cc48dc98ae37b075&prodNameId=466291&swEnvOID=54&swLang=8&taskId=135&mode=5

[BOYAN BIANDOV]

My P600 is running 2.04 so that's the latest it appears; yet the battery issue was present up until I did the clear configuration. Then after reboot no more error. Terrible lack of exposing the internal arhitecture to CLI.

Long story short it is working now but far from the elegant expecations once common for Compaq Servers...