Getting iScsiPrt errors 7 and 20 all of a sudden?

[Cajuntank MS]

I have a 2node (DL360G7) 2008 R2 cluster connecting to a pair of 2910al switches and 4300 SAN nodes connected at 1Gb ALB, flow control, but not jumbo-frames. Two NICs in each server for failover and the NICs are HP iSCSI feature-set supported. I have the latest HP DSM installed along with latest 9.x software on the SAN nodes. Noticed this week that I had been getting a lot of event errors in Windows from iScsiPrt with code 7 and 20. 20 being "Connection to the target was lost. The initiator will attempt to retry the connection."

7 being "The initiator could not send an iSCSI PDU. Error status is given in the dump data."

One thing I notice is I get a lot of 'drops Tx' on the switchport interfaces that have flow-control enabled where my SAN nodes and server connections to the SAN are connected.

Any ideas?

[Jitun]

I Hope Flow Control is enabled across the SAN, that is
On the P4300s,
Switch Ports on which they are connected.
Switch Ports that connects to the Host.
And on the Host nics as well.

[Aart Kenens]

We also had massive TX drops with a Multisite HP P4500 & 2910al switches.

We enabled flow control on the lefthand/vmware and switches. The drops were gone after that.

Also enable flow control on the uplinks from your iscsi switch!

Btw call HP support and ask for a new firmware for the 2910al ==> w_14_65.swi

We had some major issues with our switches going on 100% cpu bringing down our entire network.

Rebooting the switch didn't help so ask the FW asap.

[Cajuntank MS]

I already had flow-control enabled on all SAN node NIC ports, all SAN ports on the 2910al switches, on the uplink ports to my 5412zl switch, and on the server NICs I have assigned as SAN NICs as well. It's the ports that have flow-control enabled on them that I'm seeing the dropped TX packets on the switchThe uplinks are 10Gb on both 2910al and 5412zl. I'm on W.14.49 on the 2910al and K.15.0.4.0003 on the 5412zl. I'm waiting on a call back from the LeftHand group at HP, but I'll check with them or the Procurve group about the W.14.65 firmware.

When I find out something for sure, I'll post back, but in the meantime, if anyone has any other ideas, please let me know just in case.

Thanks.

[Jay Cardin]

You really shouldn't have Flow Control enabled between switches.  That is a very bad idea and will cause dropped iSCSI connections.  Flow control should only be enabled on the LH nodes and on the switch ports that they are connected to; same for server NICs. 

[Cajuntank MS]

Still waiting on Lefthand group to call back, but I did talk to the Procurve group briefly. Sent them a show tech on both my switches and they are supposed to reply back with a firmware up while they look at my tech logs. In the meantime, I noticed one other piece of info... when I get the iScsiPrt errors, I get them a few times, it eventually connects back up and then 30 minutes later, I will get another batch of errors. It looks fairly consistant on this 30 minute threshold as I go back for several hours. Every 30 minutes it will flood with a bunch of those errors for a minute or so and then go normal. I have also, just to see what would happen, turned off flow-control on one of my server's host nics and amazingly, even after 30 minutes, I have not gotten another iScsiPrt error (knock on wood)...but I find that weird.

[Cajuntank MS]

I made sure I turned flow-control off between the switches "Jay". I also turned flow-control off at one of my other servers, but I could not replicate the stopping of the iScsiPrt errors on it like the first one did. Same make/model/spec server, same OS and HP software level, etc...  Wierd is all I can say.

Still waiting on a call back from HP :-(

[Cajuntank MS]

Finally got a hold of someone today after lunch and spent the next 5 hours on the phone with a level 2 engineer. Finally got rid of my iScsiPrt errors as I am using NC382T NICs for my SAN connectivity and those NICs are supported for HP's accelerated iSCSI feature set in the network configuration utility. Come to find out that it's not supported with HP's P4000 product; bummer. So after removing that (he also had me turn off all of the offloading features), a different problem arose with my HP DSM service as it would stop. It would give no reason and it would start just fine, but stop just as easy. He gathered all of my logs and server reports so he could forward them to a level 3 engineer on Monday. I think it boiled down to the engineer having me manually make the connections to the targets from both my server host NIC's IP addresses instead of just letting it default everything.I figured that one out on my own after I got off the phone with him. So far, no errors, HP DSM is working, and I've made the changes on 3 of my 4 machines.

I think I'm still having a switch issue though with the dropped TX packets on those flow controlled ports, but I want to get my servers "kosher" before I tackle that.

[Cajuntank MS · Accepted solution]

OK, after everything said and done, here's what it looks like my result is. Never did talk to level 3 tech support at HP like I was promised, but as I mentioned in earlier post, with some help from the level 2 engineer, my epiphany at 2:00am, and helpfull info from Aart on the switch firmware issue, it seems my issues are resolved.

So a recap, I was using the HP iSCSI Accelerated feature-set of the server NIC (through NCU) that was not supported in conjunction with the P4000 series product (shame on me for thinking HP stuff would work with HP stuff :-)

Turned that off, turned off the inherent offloading features of the NIC, made the connections to the targets with multipath option but let everything default (level 2 guy had me making those connections manually which was failing my HP DSM service), and finally upgraded the firmware on the 2910al(s) which took care of my large DropTX numbers. My highest number is now 1 out of 56000 packets might get dropped which from another posting I read 1 out of 5000 was acceptable.

[Cajuntank MS]

Ok, I thought I had this resolved and I sort of did, but come to find out, I was not getting connectivity from both of my iSCSI NICs in my servers. I finally heard back from L3 support and they had my L2 guy I had been talking to issue these commands after I disconnected my targets and removed my favorite targets.

net stop dsmcontrol
iscsicli listtargets T
net start dsmcontrol

mofcomp.exe %systemroot%\system32\wbem\iscsiprf.mof

Rebooted server and reconnected back to my targets using the manual method of specifying from each source IP address to connect. That worked and my DSM stayed running and my connections doubled that I had to my targets and also confirmed the additional DSM connections via the CMC.