Intermittent screen freeze due to 'out of order packets' ???

[Sajith V Mannadiar]

Hello there,

We have 3 application servers running Oracle 9iAS which are load balanced using Cisco CSS.

User sessions on two servers experience intermittent screen freeze issues, while the sessions on 3rd server is working fine..

Also we get '408 errors' on the HTTP access log of the problem servers.

we have been analysing the 'netstat -p tcp & beforeafter' on all 3 servers and we found that the count of 'out of order packets' is increasing on the two problem servers.

Could someone help me understand what could be causing this 'out of order packets'? It is increasing in the order of 40 per hour.

We have asked the network team to verify the load balancer and also the network switches, but also want to confirm if everything is fine on the HPUX servers

Please advise..

Thanks,
Sajith

[Pete Randall]

Is there any chance of a duplex mismatch between the server and the switch? With 100 Mb interfaces, you need to force them to 100FD in order to get good performance. If left to auto-negotiate you end up with lots of problems like collisions, dropped packets, etc. This could account for your symptoms.


Pete

[Sajith V Mannadiar]

Thanks for your reply, but we have both ends configured in 100 Mbps Full Duplex

Any other thoughts..please?

[rick jones · Accepted solution]

Out of order packets can happen when there is packet loss - the packets arriving after the lost one(s) are out of order.

Check your lanadmin statistics on the problem servers. If the interfaces are in full-duplex and you see FCS errors, it suggests a duplex mismatch. If the interfaces are in half-duplex and you see _late_ collisions, it suggests a duplex mismatch. Many's the time that folks beleive they have hardcoded both sides, when someone else has "helpfully" straightened-out the cabling at the switch and moved things around ports - so _triple_ check your settings (frankly, I much prefer to leave everything at auto).

Another possible cause of out-of-order packets is trunking/bonding/link aggregation and poor choice of algorithms.

Having said all that, a rate of 40 out of order packets per hour is nothing - unless your actual packet rate is rather low indeed. Now, if you can also check the retransmission rates on the servers - netstat -p tcp > before; sleep 60; netstat -p tcp > after and then run the two through beforeafter (ftp.cup.hp.com/dist/networking/tools/

beforeafter before after > delta

look at data packets retransmitted relative to data packets. Also check on the "other" ends if you can - if there is packet loss, and many of the transactions across the network are small, they can induce 500 millisecond or greater delays in things happening.

Finally, it _could_ be the case that the load balancer is messing-up. Checking that requires taking packet traces on both sides of the load balancer.

[Sajith V Mannadiar]

Hello Rick,

Thanks very much for the detailed information...

The tcpdump analysis indicates that there are packet losses...I shall check other parameters using netstat -p tcp / beforeafter and let you know..

Your help is much appreciated


Thanks,
Sajith

[rick jones]

you should follow-up the tcpdump analysis with a check of the lanadmin statistics for your NICs.