MSA 1000 reset and disabling write/read cache

[Todd Mermell_1]

I'm new to HP storage, particularly MSA 1000s, and just inherited a Linux NFS cluster doing up to 20,000 IOPS on a single MSA 1000. Some of the SAs have had instances where the write performance (latencies over 1000ms) have occurred and found that resetting the MSA controller fixes the problem - at least for a week or two.

This was noticed after failing over to a second node in the cluster and seeing the same performance issues.

Question - are writes acknowledged while in cache by the MSA before destaging to disk? If so, does a soft and/or hard reset potentially result in lost writes to disk?

The soft reset is done using the msaflash utility.

Anyone else experience performance issues on the MSA 1000 resulting in a reset to the MSA controller to resolve the issue?

Is it possible to disable read and/or write cache from a Linux host or using the cpqacuxe utility?

Thank you in advance.

-Todd

[Brian Bleijenberg]

Hi Todd

I far as i know ... If you have a fullblown
filled with disks,MSA the max IOPS = 8000!!
you can calculate this.
The Drive Subsys is the limitation here, not the controller.

Hope this helps

Brian

[Todd Mermell_1]

Actually, that makes a lot of sense for a JBOD. If the queue depth on the drives is 255:

42*255 ~ 10,000

Is that what you were referring to?

But that would be I/O directly to disk. Assuming some kind of cache hit ratio, I would expect the theoretical limit to be much higher - maybe 20-30k.

Thanks for the response.

-Todd

[Steven Clementi]

On the MSA, you can change the read/write cache ratio from 100%/0% to 0%/100% in 10% increments. I think you can also disable the cache.

Since the cache is battery backed, I think that you woule not lose any cached data since the memory does not actually get flushed. Leave it off for ten days and you can probably say buhbye to any cached data.

As for performance, I have not been in a situation were I can "test" performance. I install the MSA's all the time, but am not around to see the customers face when they see that it is not the right array for their application needs.

Since the backplane is still scsi based, not fibre... i would not expect to get real high IOPS out of it, thus one of the reasons for it being "Entry Level".


Steven

[Uwe Zessin]

I would not change the read/write ratio of the writeback cache (=array accelerator). For the OpenVMS operating system HP says:
"""HP recommends customers run with the default MSA1000 cache settings of 50% read cache and 50% write cache.

However, if a change to the default cache setting is required, then the MSA1000 cache MUST be set with 50% write cache or higher. Lower than 50% write cache is not supported.""

You can try and turn the WBC off on an individual LUN basis:
CLI> SET UNIT 0 CACHE=DISABLE

Of course that will lower the performance of the storage array. I suggest that you check if the MSA is already running with the latest firmware for Linux environments (V4.32) - there have been improvements over the last releases.

[Todd Mermell_1]

Thank you for the responses. We're at the latest firmware - 4.32 and still having problems with high (> 1000ms) wait times on I/O. This is very intermittent - occurs about once a week.

As mentioned before, we use the msaflash utility to reset the controller, but what's actually taking place during a soft reset?

Does the controller go through a POST and initialization as if it had been power recycled?

Can anyone think of why write throughput crawls to a standstill? Could it be a race condition in the firmware or some other deadlock that is timing out?

I'm totally okay with reaching limitations on the storage but having to manually reset the controller every time this happens is not very supportable.

Thanks again for all the responses.

-Todd

[Steven Clementi]

Have you tried HP Support? It sounds like you have a serious issue which they can possibly help with, either with answering the question... "Are we trying to push too much data into the MSA, MORE than it can handle at a given point in time?"
or with providing assistance with tweaking the performance of the MSA.


Eitherway, you should have SOME type of paid support for this MSA. If you don't, you can probably still get HP Support to look at the issue because it sounds like other companies could be experiencing or can experience the same issue, especially if there is a firmware/hardware problem that is fixable.


A soft reset is simply that. It is a power cycle without powering off the shelf. The difference is that the drives do not power down, then have to spin up again.

There could be a little more to it than that, but that is the basic function.


Steven

[Todd Mermell_1]

Steve,

Thanks for the reply. There is a case opened by the DBA team and they were recently advised to go to 4.23 of the firmware, which is the version I'm using on pairs of NFS clusters.

So we're no closer to a solution via support at this point. We may opt to put a Windows machine on the SAN if more advanced diagnostic tools are supported on Windows -for monitoring the throughput and performance on the MSA itself.

Thanks again,

-Todd

[Steven Clementi]

What type of san switches are you using? The internal 8 port switches? or external switches?

How about the firmware on those?

I am sure you probably checked all this already, but I would like a bigger picture of the environment.

Bios/firmware on the HBA's? Specifically, is it up to date.

What about your switch zoning? Can you grab the config files and post them?


Steven

[Todd Mermell_1]

HBA info:

QLogic PCI to Fibre Channel Host Adapter for QLA2340:
Firmware version: 3.02.24
Driver version 6.07.02-RH2

I don't know what level the switch is - it should be the 8 port that comes with the unit.

[Steven Clementi]

Hmmmm Any idea how the switch is zoned, if at all?

I have seen funny things when zoning is not configured in a Heterogeneus Linux environment, specifically in a cluster environment.

You should be able to telnet into the switch from the built in nic or use the serial port. If you can post the configuration, it might be helpful.

If you need help getting into the switch, just let me/us know.


Steven

[Todd Mermell_1]

To my knowledge, there is no zoning configured on the switch.

I say this because I do not believe there is even Ethernet or serial access to it.

I'll check with the folks at the data center and see if there is any way we can get a LAN connection to it.

Thank you,

-Todd

[Todd Mermell_1]

If I can get a LAN cable connected, will the switch try and pick up a DHCP address if we reboot it? Or does it have to be statically set?

-Todd

[Steven Clementi]

No to the DHCP. The default IP is 10.77.77.77. You will need to use a crossover cable to a server/laptop/workstation to get into it or use the serial cable.

The login for the switch is...
user: admin
password: password

Once on the switch, you can see/set the IP using IPADDRSHOW/IPADDRSET.

You can also use a web browser to view the switch once you set the IP (or use the default). You will need the Java Runtime though. I suggest J2EE 1.4xxx

a couple of additional questions...
How old is the MSA?
How many ports are on the switch?
Do you have the second controller/switch for it? or is the MSA a single controller setup?


Steven

[Todd Mermell_1]

Stephen,

I checked and there is an 8-port switch used in the middle - so the single controller port is connected to a SAN switch, not a 2/8 switch in the MSA itself.

I'm going to have a serial cable connected to the switch from one of the cluster nodes and should have the information available later today.

There is a redundant controller in the MSA, however, I was told that SecurePath did not work for this release of Linux (REL 3.0) and the HBA.

HP's document site does not have updated information with regard to configuring SecurePath for the MSA 1000 and this release of RedHat.

-Todd

[Uwe Zessin]

The problem with SecurePath is currently being worked on. You should be able to use Qlogic's multipath driver.
http://h18000.www1.hp.com/products/sanworks/softwaredrivers/multipathoptions/qlogicv70.html

Make sure you grab the 7.00.03 driver. One of my customers only looked for 'Red Hat' and installed the old one. System crashed on the first failover.

[Cass Witkowski]

We have done some performance testing with the MSA1000s. With an ES40 running OpenVMS V7.3 and using the TESTDEV utility we set up 32 LUNs each consisting of a 36GB 15K disk. Cache was disabled.

Using 20 concurent I/O streams to each disk we performed random reads 1KB in size. We got around 10,000 I/Os per second.

[Todd Mermell_1]

Switch firmware level:

Kernel: 5.4
Fabric OS: v3.1.0
Made on: Thu May 1 12:32:20 PDT 2003
Flash: Thu May 1 12:33:13 PDT 2003
BootProm: Tue Oct 30 10:24:38 PST 2001

[kelly trosper]

Cass,

You said you have done some performance testing with the MSA1000s.

What type of write speeds were you getting to Raid5 vs. ADG?

Thanks!

KT