MSA 1000 reset and disabling write/read cache

[Todd Mermell_1]

I'm new to HP storage, particularly MSA 1000s, and just inherited a Linux NFS cluster doing up to 20,000 IOPS on a single MSA 1000. Some of the SAs have had instances where the write performance (latencies over 1000ms) have occurred and found that resetting the MSA controller fixes the problem - at least for a week or two.

This was noticed after failing over to a second node in the cluster and seeing the same performance issues.

Question - are writes acknowledged while in cache by the MSA before destaging to disk? If so, does a soft and/or hard reset potentially result in lost writes to disk?

The soft reset is done using the msaflash utility.

Anyone else experience performance issues on the MSA 1000 resulting in a reset to the MSA controller to resolve the issue?

Is it possible to disable read and/or write cache from a Linux host or using the cpqacuxe utility?

Thank you in advance.

-Todd

[Brian Bleijenberg]

Hi Todd

I far as i know ... If you have a fullblown
filled with disks,MSA the max IOPS = 8000!!
you can calculate this.
The Drive Subsys is the limitation here, not the controller.

Hope this helps

Brian

[Todd Mermell_1]

Actually, that makes a lot of sense for a JBOD. If the queue depth on the drives is 255:

42*255 ~ 10,000

Is that what you were referring to?

But that would be I/O directly to disk. Assuming some kind of cache hit ratio, I would expect the theoretical limit to be much higher - maybe 20-30k.

Thanks for the response.

-Todd

[Steven Clementi]

On the MSA, you can change the read/write cache ratio from 100%/0% to 0%/100% in 10% increments. I think you can also disable the cache.

Since the cache is battery backed, I think that you woule not lose any cached data since the memory does not actually get flushed. Leave it off for ten days and you can probably say buhbye to any cached data.

As for performance, I have not been in a situation were I can "test" performance. I install the MSA's all the time, but am not around to see the customers face when they see that it is not the right array for their application needs.

Since the backplane is still scsi based, not fibre... i would not expect to get real high IOPS out of it, thus one of the reasons for it being "Entry Level".


Steven

[Uwe Zessin]

I would not change the read/write ratio of the writeback cache (=array accelerator). For the OpenVMS operating system HP says:
"""HP recommends customers run with the default MSA1000 cache settings of 50% read cache and 50% write cache.

However, if a change to the default cache setting is required, then the MSA1000 cache MUST be set with 50% write cache or higher. Lower than 50% write cache is not supported.""

You can try and turn the WBC off on an individual LUN basis:
CLI> SET UNIT 0 CACHE=DISABLE

Of course that will lower the performance of the storage array. I suggest that you check if the MSA is already running with the latest firmware for Linux environments (V4.32) - there have been improvements over the last releases.

[Todd Mermell_1]

Thank you for the responses. We're at the latest firmware - 4.32 and still having problems with high (> 1000ms) wait times on I/O. This is very intermittent - occurs about once a week.

As mentioned before, we use the msaflash utility to reset the controller, but what's actually taking place during a soft reset?

Does the controller go through a POST and initialization as if it had been power recycled?

Can anyone think of why write throughput crawls to a standstill? Could it be a race condition in the firmware or some other deadlock that is timing out?

I'm totally okay with reaching limitations on the storage but having to manually reset the controller every time this happens is not very supportable.

Thanks again for all the responses.

-Todd

[Steven Clementi]

Have you tried HP Support? It sounds like you have a serious issue which they can possibly help with, either with answering the question... "Are we trying to push too much data into the MSA, MORE than it can handle at a given point in time?"
or with providing assistance with tweaking the performance of the MSA.


Eitherway, you should have SOME type of paid support for this MSA. If you don't, you can probably still get HP Support to look at the issue because it sounds like other companies could be experiencing or can experience the same issue, especially if there is a firmware/hardware problem that is fixable.


A soft reset is simply that. It is a power cycle without powering off the shelf. The difference is that the drives do not power down, then have to spin up again.

There could be a little more to it than that, but that is the basic function.


Steven

[Todd Mermell_1]

Steve,

Thanks for the reply. There is a case opened by the DBA team and they were recently advised to go to 4.23 of the firmware, which is the version I'm using on pairs of NFS clusters.

So we're no closer to a solution via support at this point. We may opt to put a Windows machine on the SAN if more advanced diagnostic tools are supported on Windows -for monitoring the throughput and performance on the MSA itself.

Thanks again,

-Todd

[Steven Clementi]

What type of san switches are you using? The internal 8 port switches? or external switches?

How about the firmware on those?

I am sure you probably checked all this already, but I would like a bigger picture of the environment.

Bios/firmware on the HBA's? Specifically, is it up to date.

What about your switch zoning? Can you grab the config files and post them?


Steven

[Todd Mermell_1]

HBA info:

QLogic PCI to Fibre Channel Host Adapter for QLA2340:
Firmware version: 3.02.24
Driver version 6.07.02-RH2

I don't know what level the switch is - it should be the 8 port that comes with the unit.