Where Did the Packets Go?

[Paul Nunez]

Hi Everyone,

First, this isn't an OpenVMS problem, but does involve an OpenVMS application. I'm much more comfortable starting here than in a general networking forum, so I hope you'll bear with me. I just need some knowledgeable people to bounce this off of...

I'm trying to isolate an issue where _the majority_ of a specific UDP packet type are either not getting to the destination host or are being ignored/dropped by that host.

The short of it is the remote host is connected to a Cisco switch. Using the port spanning/mirroring capability, a sniffer was connected to a port which was mirroring this particular remote host's port. That trace captured/showed every UDP packet sent from the local (VMS) host. A trace later obtained on the destination system (using Ethereal) captured only a few of these UDP packets (about 2% of the total sent).

Since all packets were seen in the capture obtained on the mirroring port, yet few were seen in the capture obtained on the destination system, doesn't that _clearly_ point the blame at the destination system? And since Ethereal is running in promiscuous mode on that destination system and only "sees" a few of these packets, doesn't that implicate the NIC in the destination system?

Unfortunately, I don't have other resources available to use for testing. Also, a workaround is now in place and we are not able to disrupt operations to return to the non-working configuration for testing (I'm hosed :O). Yet a solution is expected.


Thanks,


Paul

[Garry Fruth]

I don't believe UDP guarantees delivery. Is it possible that the VMS box is sending packets at a rate faster than they can be recieved on the destination box; e.g. VMS sending at 100MB Full duplex and the reciever is at 10MB half? BTW, what is the destination box?

[John Gillings]

Paul,

You need to work out the path(s) between the systems. What routers or firewalls may be between them? Routers are allowed to just drop certain packets, and firewalls are *supposed* to!

Remember that IP is not connection based, so consecutive packets don't need to follow the same path. It could be there are multiple paths, and one or more of them have some box that doesn't like your packets.

I guess the only way to find out for certain is to draw yourself a map, and sample each segment between any routers or firewalls to see who's responsible for dropping them.

[Wim Van den Wyngaert]

1) It can be a router saturation. It simply drops what is too much.
2) It can be bad hardware. Do as ucx sho prot udp and ucx sho prot tcp and check for abnormal things (both origin and destination)
3) It can be a double IP address. In that case some device (router) may be sending the packets to a bad destination.
4) Who knows. Try ucx sho prot icmp to find other abnormalities.

Wim

[Paul Nunez]

Thanks to everyone for contributing, but I still don't get it.

How could an intermediate system be dropping the packets if we see ALL the packets arrive on the spanned port on the switch to which the remote host (Windows 2003 Server on a DELL box) is directly connected?

If they all arrive at the spanned port, doesn't that imply they all reached the port being mirrored? These ports are on the same switch. If the packets were dropped before reaching the switch, they would not appear in the trace obtained on the spanned port, no?

And the only thing between the port to which the remote system is connected and that remote system is the cable to the remote system. I guess a bad cable could produce these results.?.

I don't understand how an intermediate router/firewall could be involved (but that's why I'm here :o).

My hands are being tied by the "customer"; he can't help isolate the problem through testing, so I'm left with conjecture. I'll likely need to push this up to the folks who handle complex problem management, but want to first be diligent in my efforts to isolate where the fault lies.

Thanks again...

Paul

[Robert Gezelter]

Paul,

Unless you are actually on the cable to the recipient server, it is "probable cause", not a conclusion.

Since all the packets show up at the mirrored port, the problem could be in the switch or in how things are being received.

My suggestion is that you install a real hub on the Dell side of the switch, and hook a sniffer and the Dell server directly into the hub:

Cisco Switch --- Hub (100BaseT/Full Duplex)
| |
Dell Server Sniffer

Then cross compare the packets seen on the sniffer with the actual traffic reported on the Dell Server. Then analyze the resulting trace on BOTH systems.

I hope that the above is helpful.

- Bob Gezelter, http://www.rlgsc.com

[Paul Nunez]

Ok, so I still have work to do to find the faulty device. Since I can no longer utilize the customer's environment to do so, I'm left with duplicating their environment in-house.

This issue has not been reported by any other user of our application and the code changes which resulted in this problem have been in wide use since Oct 2003. This customer's environment is not unique in regard to the network equipment (widely used Cisco gear). But heck, I can't even convince them to try a different port on the switch at this stage...

Thanks all,


Paul

[Andy Bustamante]

You know that the traffic is getting to the Cisco switch.

Has the customer loaded any software updates from Microsoft or any anti virus/firewall software? Check the switch and the NIC on Windows system, do they argree on speed and duplex settings? Change the network cable from the switch to the Windows box. Change the network card on the Windows box.

You know the traffic is getting to the the destination Cisco switch, it's the last leg that I would focus on.

[Robert Gezelter]

Paul,

Andy's comments are well taken.

I also must apologize about the diagram in my earlier posting. The HUB was connected to the Dell server and the sniffer, and thence connected to the switch (the ITRC posting software mangled my diagram).

I would also not be surprised if you are unable to replicate the problem in your lab. If the problem is indeed a incorrect setting, or a defective port/switch you will more likely than not be unable to reproduce it.

- Bob Gezelter, http://www.rlgsc.com

[Doug Phillips]

>Ok, so I still have work to do to find the
>faulty device. Since I can no longer
>utilize the customer's environment to do
>so, I'm left with duplicating their
>environment in-house.

There is no way you could do this. But, the question I have is: What "workaround" did you put in place?

>This issue has not been reported by any
>other user of our application and the code
>changes which resulted in this problem
>have been in wide use since Oct 2003. This
>customer's environment is not unique in
>regard to the network equipment (widely
>used Cisco gear).

It's unique in that something isn't working;-)

You've checked the obvious things (settings on NIC match setting on switch port), so that leaves: The switch's port is failing, you have a bad cable or connector, the NIC driver is corrupted, or the NIC is bad.

>But heck, I can't even convince them to
>try a different port on the switch at this
>stage...

That's the only way you'll find the problem without bringing down the system. Swap the system's port with the mirror port. Switch the system cable. These things you can do without major disruption. These things will either point to the port or cable as the problem or eliminate them as as suspects, leaving the driver or card. You can reinstall the driver first, and if that doesn't work replace the card but these will both disrupt things.

Otherwise, the customer is being unreasonable. How does the customer expect the problem to be fixed if you can't troubleshoot it?