MSA2012i iSCSI MPIO + Windows 2008 R2 BXIOS Errors

[Anthony Kearns]

Has anyone out there sucessfully set up MPIO with Windows 2008 R2 and an MSA2012i array?

We have a number of servers running Windows 2003 and Windows 2008 (x64) using the array and MPIO with no problems whatever. However we have recently introduced the following server:

HP DL360G5
2 x HP NC382T dedicated NICs for iSCSI
Latest firmware applied to NICs
Windows 2008 R2

The current environment:
MSA2012i (dual controller)
2 x Cisco 2560 Gigabit switches (seperate IP subnets)
Latest firmware applied to MSA2012i

The server has been setup exactly the same as the previous Windows 2008 (x64) machine in that one NIC is connected to one switch the other to a different switch. A total of 4 paths are then created manually 2 to Controller A, 2 to Controller B spread over the two subnets available. MPIO is enabled as Round Robin.

Now the problem. With both ports connected the server reports the following errors in the System Event Log every few seconds:

BXIOS, Event ID 20, 3/1/2010 8:59:09pm
"Connection to the target was lost. The initiator will attempt to retry the connection".
BXIOS, Event ID 48, 3/1/2010 8:59:11pm
"Target failed to respond in time to NOP request".

and then an "Information":
BXIOS, Event ID 34, 3/1/2010 8:59:11pm
"A connection to the target was lost, but Intiator sucessfully reconnected to the target. Dump data contains the target name"
<-the dump date then shows the disk on controller A

The server will appear sluggish in Disk Manager and is prone to hanging. Rebooting the server will effectively either hang it on shutting down or when it starts up. Removing one of the network cables used by iSCSI will make the server recover.

All in all it sounds like the issue is defintely MPIO but I at a loss as to what. I've tried removing the entire iSCSI configuration at least twice and also followed HP's document:

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&objectID=c01715951&jumpid=reg_R1002_USEN

to the letter (just in case).

Again just to reiterate none of the other servers in this environment are experiencing this problem. All use MPIO and two dedicated network cards.

If anyone can shed any light on this it would be greatly appreciated.

Many thanks.

Anthony

[Prokopets]

>Latest firmware applied to MSA2012i
NICs drivers? NICs firmware? (sounds stupid, but anyway...)

[Anthony Kearns]

Hi P.P.

Thanks for the reply. Unfortunately the array and the NICs are using the latest firmware available. The NICs are using the latest drivers as well as the latest verion of the HP Network Configuration Utility (NCU).

Any other ideas would be gratefully accepted.

Regards,

Anthony

[Caleb Lucas]

I have had similar issues with DL 380 G6 servers running 2008 R2 Core and NC382T with the latest drivers and firmware.

In my case the issue occurs whether or not MPIO is enabled, so even with a single path. I performed a packet capture which revealed that the HBA stops sending NOP out requests at some point.

The packet capture clearly shows:

- Every NOP out request from initiator is getting a good response from the target within a few mS.

- At some point, the initiator stops sending NOP out requests.

- By the end of the packet capture, the loop is this:
1. Initiator logs into target successfully
2. Initiator DOES NOT put a NOP Out request on the wire at any point
3. Capture shows Initiator resetting the connection, then doing ARP request, then connecting again successfully to the target. Event logs on server claim target did not respond in time to a NOP request. However, after this point the initiator NEVER PUTS A NOP OUT ON THE WIRE so why would it expect a response????
4. Loop repeats. It will go on like this indefinitely without ever sending a NOP out request. I have to break it out of the loop by taking away rights to the storage volume.

Just as in your situation, machine will freeze if I try to shut it down - taking away rights to the volume ends it though.

To me this looks clearly like an issue with the NIC or driver.

[Anthony Kearns]

Hi Caleb

Thanks for the info although it sounds like your situation is even worse. I've found that if I pull the 2nd path (by disabling the port on the CISCO switch or physically removing patch lead) it brings the server back to life after a few minutes. With 2 paths and MPIO any disk operation (like Disk Manager, formatting the drive, logging on, logging off, shutting down etc) will result in the server hanging until the 2nd path is pulled (or after about approximately 30 minutes).

Did you manage to get anywhere with your configuration? At this rate this server will have to go live with only a single path which is not desirable at all.

Anyone out there from HP shed any light on this?

Regards,

Anthony

[Prokopets]

Anthony, is seems that your problem is not a configuration issue, so i recommend you to open a support case in HP.

[Anthony Kearns]

Hi all

Haven't had a chance to open the call with HP yet but I have just noticed that there is another version of the controller firmware out
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=12169&prodSeriesId=3687128&swItem=MTX-0f8bb8756caa4d3d90cccc829e&prodNameId=3687132&swEnvOID=4024&swLang=13&taskId=135&mode=5

Although there is anything in the "fixes" to suggest this will solve the issues I am seeing I feel it is worth a shot as the previous firmware release was July 2009.

I will update you all later.

Regards,

ACK

[Travis Langhals]

Does anyone have the correct procedure for configuring the NC382T iSCSI acceleration in Windows 2008 R2. Not sure if an IP address needs to be assigned outside of the NCU iSCSI settings. I have assigned the same IP in network connections and for the iSCSI settings in the NCU but the iSCSI statistics are at 0.

[akearns]

Dear all

A Solution!!!

Just thought I would bring everyone upto speed on this one. After building a seperate test machine using Windows 2008R2 and different NICs but getting the same error I then embarked on some more in depth troubleshooting as and when I could. This involved different drivers, firmware, Round Robin vs Failover etc etc.

On one of these sessions I noticed that the 2nd switch used in the SAN design (CISCIO 2960G's) was still running the old MTU size (i.e. 1500) despite all the equipment plugged into it trying to use Jumbo packets (ie 9000). After changing this and reloading the switch config all the BXIOS errors vanished.

Have done extensive testing using both failover/round robin and everything is looking good so I can safely say this issue can be CLOSED.

One thing to bear in mind though: Windows 2008R2 is definitely very sensitive to this type of error as (and I checked this again) the other servers running Windows 2008 (64 bit)and Windows 2003 (32 bit) never reported this as a problem (which is why it was never noticed). The other servers had even gone through extensive site acceptance tests (i.e. disconnecting nics, switches , controllers etc) before going live and this error still didn't appear anywhere.

Watch out everyone...

[Anthony Kearns]

Sorry everyone did the last update using the wrong account so it went as an update rather than a closure.

See the above for the solution.