4140gl reordering ACK packets

[T. Joebstl]

Hello,

I've got a very weird issue with a 4140gl reordering ACK packets in the initial setup of a tcp/ip session.

The unit has the following hw configuration:
Module A J4908A - connected to half a dozen Citrix/SQL/Exchange Servers (IBM and HP systems running Win2k Server - Broadcom GBit NICs with different driver releases ranging from 3 years old to the latest)
Module B J4908A - connected to a dozen 2424/2524/2626 switches with 1000BaseT and FX links
Module C J4864A - connected to 2 2524 with the GBit stacking kit and a 2524 with a 1000BaseFX module (serverroom basement, housing the other half dozen of the servers)
Module D J4862 - connected to a bunch of 100BaseFX fibre connectors to give some of the servers in the basement a direct connection to the main switch. Connected to all the WAN equipment and a rented fibre to building block "TZ" somehow shaped to 30MBit/s.
J4839A RPS

All floors and building blocks are in different VLANs, all servers are in a separate VLAN with the 4140 doing ip routing between them.
Switch and network load is minimal, max. load is when the backups run, and that is only 100MBit as the backup server only has a 100MBit connection.

Firmware is G.7.70 but the issues occured first back when the unit was running G.7.50. All other switches in the network are running latest SW releases as well.

Since about a month clients in the building block TZ show spontaneous connection issues to the ica redirector so clients have to try to log on 2 times and more till the connection actually works.
I've finally got permission to put ethereal on one of the terminal servers and that shows some very weird reordering of the packets during the setup of the tcp/ip session.
I've put a sniffer on the connection between fibre and the 4140 (Module D port 19) with a spare 2424 I had lying around which shows the proper SYN, SYN/ACK, ACK, DATA... sequence.
The sniffer on the Terminal Server (Module A) shows SYN, SYN/ACK, DATA, DATA, ACK so somewhere through the switch the packets get reordered. It doesnt matter if I set the port of the Terminal Server to 100MBit or even 10MBit - the issue still persists. Only fix so far has been connecting the terminal server used as ica redirectors to module B. The ica redirectors on modules C or D dont show that weird behaviour. Clients connected to the switches linked via module B dont show that either. If I'm connecting a client with the spare 2424 between the fibre and the 4140 it works, too.
Only clients connected via the fibre to building block TZ show that behaviour - but only if VLAN routing is involved (doesnt matter which client VLAN). If I put the client into the server VLAN the problems vanish.
I've gone so far as to alter VLAN tagging on the fibre link, keeping the client-VLAN untagged but that didnt to solve the problem. The fault finder doesnt show anything weird on the fibre, ethereal doesnt show anything weird on the fibre - it just gets weird when routing on the 4140 is involved.

Since somebody has to be in building block TZ to handle the client and somebody here to change connections on the switch I thought it would be nice to have a client here to trigger the issue without constantly having to drive between 2 locations.
So I put up a cheapass linksys router in TZ, put the internal side into a separate VLAN + subnet and added a static route to a new subnet on the 4140.
Guess what - the issue vanished behind the linux router....
Thank god I had a spare 2626 to use as router so now I put in that one and (finally) I can trigger the issue and do some work on the 4140 at the same time.
Only difference is that in building TZ i can trigger it every 10 seconds - here it takes up to 10 minutes till the error occurs.


But I'm at my wits end:
Why does it reorder packets - only when the servers are on module A - and only for clients behind the fibre?
Why does it suddenly work when there's another router involved which isnt actually a layer3 switch?

I'll drive there tonight to do some more tests without disturbing work but honestly I dont know what more I can try besides reseating or swapping the modules A+B.
So I'm very open to suggestions.

Regards,
Thomas

[T. Joebstl]

So, now it's not just ACK packets getting reordered - it's packets just vanishing when passing in on a port on module A.

I've tried swapping modules A and B since they're the same model - didnt change anything.

I've tried connecting the fibre to modules A and B - didnt change anything.

I've connected the servers to module B and D - works.

So the conclusion is:
Packtes vanish/get reordered when
a) they're going out/coming in on that specific fibre and
b) servers are connected to module A and
c) layer3 routing on the 4140 is involved.

Guess the 4104 is broken in a very subtle way and I'll have to get it replaced.

Regards,
Thomas

[OLARU Dan]

I once tried to use routing on 4104GL: it was a mess - so I stopped that project and used a specialized router