Procurce 5406zl iSCSI Jumbo packet+Flow control packet loss

[Dirk Oosting]

We use a 5406zl for our iSCSI SAN with Jumbo packets and Flow Control, but when I look in this VLAN with a networkanalyzer there are many packetresends. The reason that I use a 5400 is the combination of Jumbo packets and FC, the switch has a almost default configuration and is a dedicated SAN switch with 2 VLANs. All server nics and hba's support Jumbo packets+FC.
Does anyone have a similar setup? I use the latest firmware.

[rick jones]

While I may be "HP" I'm not "ProCurve" so don't have specifics on "how," but have you checked the switch stats for drops from other reasons besides flow-control? FCS/CRC errors? Other?

Have you done the same on the servers?

Do you know that when/if the link flow-controls your hosts have "deep enough" queues to keep from overflowing those in the hosts?

[Igor Ybema]

I have recommended a lot of 3500's to our iscsi customers. The 3500 is just like the 5406 but is the fixed model instead of chassis based. These types are well enough for jumbo+fc because they have large buffers.

I haven't heard any complaint about the combination with jumbo+fc but I haven't been using a analyzer yet to check for resends.

The question is indeed why it is retransmitting. If the counters on the hp switch stay at 0 then could also be your hosts which are to blame. Which kind of NICs are you using? My experience (4 years iscsi) is that e1000 from intel are the best iscsi nics. Broadcom based nics are not that good if the usage is around 100%.

[Gerhard Roets]

Hi Dirk

This is a toughie for a forum. I would recommend the following

walkmib ifname ( to get the .### to user name for ports )

and then

walkmib ifindiscards ( Interface inbound discards )

walkmib ifoutdiscards ( Interface outbound discards )

Do this once .... let it run for a bit and repeat the above commands.

show interfaces all can also be used to identify other types of drops i.e. FCS errors

Now if you notice an increment in ifoutdiscards.

Look at the Advanced Traffic Guide and QoS Queue Configuration ( In chapter 6 )
You might want to go to 4 queues, if you do not use and QoS / CoS / Routing Protocols you might be able to go to 2 queues safely but I would first check the 4 queue mode out as en experiment if possible.

be aware of the forced reboot etc etc

HTH
gpr

[Dirk Oosting]

Thank you for your response.
We have a mix of hba's from Qlogic and Alacritech and run VMware ESX and NAS servers with large datafiles, we use 3 Equallogic's in one storagepool meaning 6Gbps
bandwidth.
I must say we are stessingout our SAN measering speeds over 400MByte/s.
After some further testing I have found the resends source, the networkanalyzer was the problem. It couldn't buffer the huge amount of traffic so it not track all packets send.

Gerard what settings do you use with QoS? I tried TCP/UDP port priorty with 4 queues but could not see a better performance. The TCP ports I used were:
| Port Apply rule DSCP Prio
-------- + ---- ---------- + ------
TCP | 3260 Priority | 3
TCP | 3205 Priority | 3
UDP | 3205 Priority | 3
UDP | 3260 Priority | 3
TCP | 860 Priority | 3
UDP | 860 Priority | 3

[Gerhard Roets]

Hi Dirk

You should not need to tune the QoS settings unless you have contention( Except if it is required by the manufacturer ).

Look at Table 6-12. "Default GMB Percentage Allocations per QoS Queue Configuration"

Since the switch is 100% dedicated to your "LAN-SAN". I would suggest the following

1. Don't mess with QoS it is dedicated to your storage Ethernet traffic :)
2. Configure it for 2 queues

HTH
Gerhard

[rick jones]

"After some further testing I have found the resends source, the networkanalyzer was the problem. It couldn't buffer the huge amount of traffic so it not track all packets send."

So, what you were seeing were gaps in the TCP (?) sequence space but not actual retransmissions of given TCP sequence numbers? That can be a _very_ common problem especially if you have full duplex communications and a monitor port setup - the full duplex traffic of even one port cannot "fit" in one direction on the monitor port. Hence the reason to also always check the stats on the ends, and if possible the middle :)

[Dirk Oosting]

I did some more testing with portmonitoring, its disasterous for heavy used ports you will get restransmits. Very strange behaviour, sounds like a performance problem when copying data too the monitoringport.
The 5400 have more portbuffer then the 2800/2900 series but not enough for heavy usage?
With the 2800 you can free more buffer using the qos-passthrough-mode, does this do the same for the 5400 when using 2 queue QoS even when you're not using QoS?

[Gerhard Roets]

Indeed :)