topic Re: Procurve 4160 and Netapp filers, slow routing in Switches, Hubs, and Modems

Procurve 4160 and Netapp filers, slow routing

jan_68 — Fri, 27 Oct 2006 07:47:11 GMT

Hi everyone.
We've got a couple of Netapp filers F820 and F940. Our network consists of HP 4000s and 4100s with the 4100 at the core doing all the L3 routing. We have discovered that any time a filer talks to the core switch at layer 3 it's very, very slow. Basicaly clients from subnets different than the filer's time-out or get very slow response. We did some packet capture and it shows a lot of retransmissions of TCP. We created a test network with Intel 480T L3 switch (which is the same as Extreme 5i) and there is no such issues whatsoever. I am hoping that somebody might have some suggestions. Is the problem with the Filers or HP gear ? Please see attached scenario. Thx

Re: Procurve 4160 and Netapp filers, slow routing

Mohieddin Kharnoub — Fri, 27 Oct 2006 09:13:46 GMT

Hi Jan

I would like to ask you if you can attach the config of your 4100 which does L3 routing, and some output of the packet captured also if its possible.

Good Luck !!!

Re: Procurve 4160 and Netapp filers, slow routing

jan_68 — Fri, 27 Oct 2006 09:22:35 GMT

Show tech attached here

Re: Procurve 4160 and Netapp filers, slow routing

jan_68 — Fri, 27 Oct 2006 09:24:02 GMT

Wireshark (Ethereal) capture attached here.
Where 24.199 is the client and 25.254 is the Filer

Re: Procurve 4160 and Netapp filers, slow routing

Mohieddin Kharnoub — Sat, 28 Oct 2006 02:29:06 GMT

Hi

After looking in the Wireshark output after sorting by time,
I've noticed that the retransmission packets are identical to it previous ones, just compare them, and you won't find any difference except the time of course.

Also, i noticed that the Don't Fragment Flag is set to on.

One important thing is, the retransmission happened 2 times, both are identical on L3 info, but on L2, the first one has DST MAC = to the original packet, but second one with different MAC.

With all above i suspect the following:

1- The returning traffic is being dropped because the packets are too large MAYBE, since its an SMB protocol.

2- The 4000 has a problem with Single forwarding Database and this issue can be arise in Multiple Vlans environment such the one you have.

Suggestions:
1- On the 4100 disable the high security.
2- Try to connect the client 24.199 on the 4100 switch and do the same test and try to get an output from wireshark.

3- Try do the the basic tests with trace route, and see where the packets being dropped or delayed.

4- Run the Wireshark after you connect the 24.199 client on the 4100 switch, try do work with both filers F820 - F940 and capture with Wireshark for a bit long period.

Good Luck !!!

Re: Procurve 4160 and Netapp filers, slow routing

jan_68 — Mon, 30 Oct 2006 08:31:18 GMT

Thx a bunch for reply.
Couple of things. 1.Ping or trace route does not show any issues. ICMP flies with not delays. The problem might be with SMB/HTTP protocols (?). 2.Netapp filer is a mid-range NAS device, for those unfamiliar.
I will check the defrag settings.

Re: Procurve 4160 and Netapp filers, slow routing

jan_68 — Mon, 30 Oct 2006 11:23:36 GMT

Thx Mohieddin, you made me review my testing procedures, especially suggestion about single forwarding database. I started digging in the Netapp manuals and here is what I found:
What fast path is ?
Fast path is an alternate routing mechanism available in Data ONTAP. Instead of using the routing table of the storage system to route, this mechanism uses

The source Media Access Control (MAC) address of the incoming packet as the destination MAC address of the outgoing packet for NFS-over-UDP and all TCP traffic transmitted from the storage system
The same interface for incoming and outgoing traffic
Using this mechanism provides the following advantages:

Load balancing between multiple storage system interfaces on the same subnet
The load balancing is achieved by sending responses on the same interface of the storage system as incoming requests.

Increasing storage system performance
The increase in storage system performance is achieved by skipping routing table lookups.

Fast path is enabled automatically on the storage system; however, you can disable it. "

I know it's a bit long but basically that's the problem. After I turned the option off everything seems to be working OK. Tested same setup on the HP 2848 and this one works properly. So there must be something in the way that fastpath and 4108 resolve routing requests.

It's been a long struggle, but at least now we know where the problem is. I will most likely make the 2848 our core switch and all will be dandy. Thx a bunch for the suggestions.

Re: Procurve 4160 and Netapp filers, slow routing

OLARU Dan — Tue, 31 Oct 2006 00:44:03 GMT

It is a good ideea to perform routing on some other device than 4100, since these switches are extremely sluggish in doing routing.

Re: Procurve 4160 and Netapp filers, slow routing

jan_68 — Tue, 31 Oct 2006 08:13:29 GMT

Looks like even though I have found the solution to the main problem I will be moving core to 2848. I guess that one might a bit better. Thx