P2000 MSA G3 SAS & Proliant Gen9 Server

[CB-IT]

Same problem here with the H241 controllers, DL360 Gen9 servers and MSA2040 SAS dual controller storage. All firmware up to date (version 2.14 running on the H241's). Symptoms are the same: a dead path is not revived and new LUNs are not visible unless you use some hpssacli commands (unreliable for me at best - doesn't always work).

Tried playing with the "Missing LUN response" option in the MSA2040 but both "Not Ready" and "Illegal Request" lead to the same result (according to VMware Best Practices it should be set to "Illegal Request").

Anyone have a case running or have a solution?

Edit: Opened a case, nr 4651214555

[CB-IT]

I just got a call from an HP support representative regarding the case I created, unfortunately he wasn't able to help me on this.

As it stands, the H241 HBA and the MSA 2040 combo is NOT supported.

He said it is being worked on but that's the only thing he could tell me. No release dates on updated firmware or whatever, couldn't tell me in which direction they were working (to get it supported or not). He recommended I subscribe to the new firmware alerts and closed the case on this.

[stevegolab]

After reading the thread, it appears the issue is still unresolved and has been costly. I wonder if it would help to post your question today during HP Servers Expert Day - April 28th/29th. This is an event where Server Experts across HP join the HP Support Forums to answer all your questions. Simply go the Proliant board on the HP Enterprise Business Community and post your question between 7am EST April 28th thru 7am EST April 29th.

Let me know so I can follow along.

[masterase]

I got involved into a simmilar issue yesterday.

The customer is running into identical problems with MS HYperv 3.0 (Server 2012R2)

Configuration:

Hyper-V Cluster with 3 Hosts:

DL380 Gen9 2x Intel CPU E5-2620 V3 256 GB RAM Ethernet 331i

ALL Array Controllers manually set to function as HBA adapter.

1.

1x SAS HBA H241 initially running 1.34 updated to firmware 2.14 after numerous crashes

Used first - had various crashes per week

2.

1x P441 Array Controller mit Firmware 2.14

(crashed after two days)

hp recommended downgrade  to 1.18

crash after 1/2 day

3.

replaced by

1x P431 Array Controller mit Firmware 2.40

remark: SUPPORTED on MSA2040 but UNSUPPORTED in G9 Servers

Stable sicne 3 days, but unsupported.

Die MSA2040 SAS running firmware GL200P002 (initially last V1 Firmware)

Customer is no longer amused spending 100k + EUR for the whole solution.

At least 6 other similar HYPERv projects of my company are on hold, because of the actual situation.

We urgently need proffesional support from hp (or Dothill) and will ask for an escalation manager tomorrow.

HP CASE:4651226095

 best regards from Germany

Christian

[Ken Krubsack]

Well, add me to the list too.

2 ea. 380 Gen9's running VM ESXi 5.5 with P441 controllers attached to an MSA 2000 series storage and I'm having issues with the hosts not always having connectivity to the storage.

Case opened with HP - this one happens to be in Spain.

Flippin' lovely...

Ken

[CB-IT]

Just had one server become unresponsive (running VMware ESXI 5.5.0), I connected to the console and switched to the logging terminal and saw these messages constantly scrolling by:

As you can see by the timestamps, these messages are being generated multiple times per second. Either the driver or the controller crashed, rendering the host unusable. I couldn't login to the VMware console either, the only way to regain control of the host was to hard reset it.

[Torsten.]

I remember a very similar scenario some time ago with P2000 SAS and Gen8 servers. At the end we converted the P2000 SAS to  P2000 FC and the problem was gone. The result of investigation by hp support was a bad SAS driver installed on the servers. Once corrected, the P2000 SAS was working fine.

Just my 2 cents ...

[Ken Krubsack]

Torsten,

At this point HP hasn't come up with a fix for the Gen9 issue yet.

I have good reason for that statement (which I would be willing to share in a private email).

Ken

[Ken Krubsack]

FOLKS - UPDATE FROM HP.

This customer advisory is available by doing a Google search on the advisory number (c04654888).

hth

Ken

=======================================================

SUPPORT COMMUNICATION -

Document ID: c04654888

Version: 1

CUSTOMER ADVISORY: HP Storage MSA 2040 SAS Arrays Temporarily Halted Support of SAS Arrays Configured with HP ProLiant Gen9 Servers and HP H241 HBAs in VMware ESX/ESXi Environments

Release Date: 2015-04-28

Last Updated: 2015-04-28

---

DESCRIPTION

HP has recently discovered an issue with certain configurations of the HP Storage MSA 2040 SAS arrays when connected to an HP ProLiant Gen9 server configured with an HP H241 Smart Host Bus Adapter (HBA) in VMware ESX/ESXi environments. The HP H241 HBA controller was originally listed on the Single Point of Connectivity Knowledge (SPOCK) as a supported configuration with the HP ProLiant Gen9 servers configured with the HP MSA 2040 SAS array. For detailed specifications about the operating systems and connectivity components supported with the HP Storage MSA 2040 arrays, go to the SPOCK website. Sign up for an HP Passport to enable access:

http://www.hp.com/storage/spock

Currently there are no other HP Smart Arrays (such as a P441 or P741) qualified for use in the HP ProLiant Gen9 server that are qualified with the HP MSA 2040 SAS array in a VMware ESX/ESXi environment. If you are currently in this type of configuration and it is working, no further action is needed. If you recently bought this configuration and are having issues, contact HP Support.

SCOPE

This advisory affects HP Storage MSA 2040 SAS arrays and HP H241 Smart Host Bus Adapters.

RESOLUTION

HP is working to resolve the issue quickly. Once the issue has been resolved, this advisory and SPOCK will be updated.

[JeroenL]

Update:

HP provided us with SAN controllers, FC SFPs and FC HBAs for our setup.

First I replaced the SAS HBAs and updated the server with the latest Service Pack for Proliant and reinstalled ESXi using the latest 5.5 U2 HP OEM installer. Made some alternations for the default path selection and roundrobin settings. Don't know if it has any function with direct attached FC, because there are maximum 2 paths with only one to the storage controller hosting the LUN, so only one really active IO path and so no RoundRobin load balancing being performed.

After this I replaced the MSA2040 controllers and installed the latest firmware on the MSA. Made the proper settings for VMware usage and I removed ALL the old SAS mappings for all LUNs.

Next I connected the servers one by one, modified the auto detected initiators, added them to a host, and added the hosts to a host group.

After this I mapped all the luns to this host group. And tada...everything works just as expected.

Some more testing and dead paths are automatically detected when reconnected and reconfigured as the IO active path.

After this I set the scratch storage location to a LUN and storage wise we are finally ready for production usage.

Although it isn't really sure this FC "upgrade" is temporary or not?

We also had an horrable 10Gbe network issue. This we traced to the HP J9546A 10GbT V2 module and some crappy firmware. HP send us new Intel based LOMs and although these didn't caused PSoDs like the 533FLR-T Flexloms did, we weren't able to get stable network at 10Gbe. With the switchport set to 1000-Auto, not problems, but at Auto-Auto it was unusable.

After using a simpel Netgear S3300 switch, we were able to eliminate the server hardware and get stable 10Gbe performance and no more PSoDs.

HP provided a new firmware on may 1st 2015, and we are planning to update the switches to this new firmware and see if this fixes the problem.

I think it is very, very bad that these things can happen these days. We are paying HighPrice for our HP hardware and HP is skipping basic connectivity testing when releasing new firmware. For a full month HP was unable to provide good support.

At this moment we spend more than 40hours of testing and troubleshooting for HP while expensive carepacks were bought for the servers and the storage. Me, as a technician, and our customer lost confidence in HP and I am really interested to see what HP is going to do for this customer and the company I work for to get back this confidence.