HPE Community read-only access December 15, 2018
This is a maintenance upgrade. You will be able to read articles and posts, but not post or reply.
Hours:
Dec 15, 4:00 am to 10:00 am UTC
Dec 14, 10:00 pm CST to Dec 15, 4:00 am CST
Dec 14, 8:00 pm PST to Dec 15, 2:00 am PST
Operating System - OpenVMS
cancel
Showing results for 
Search instead for 
Did you mean: 

Where Did the Packets Go?

 
Paul Nunez
Respected Contributor

Where Did the Packets Go?

Hi Everyone,

First, this isn't an OpenVMS problem, but does involve an OpenVMS application. I'm much more comfortable starting here than in a general networking forum, so I hope you'll bear with me. I just need some knowledgeable people to bounce this off of...

I'm trying to isolate an issue where _the majority_ of a specific UDP packet type are either not getting to the destination host or are being ignored/dropped by that host.

The short of it is the remote host is connected to a Cisco switch. Using the port spanning/mirroring capability, a sniffer was connected to a port which was mirroring this particular remote host's port. That trace captured/showed every UDP packet sent from the local (VMS) host. A trace later obtained on the destination system (using Ethereal) captured only a few of these UDP packets (about 2% of the total sent).

Since all packets were seen in the capture obtained on the mirroring port, yet few were seen in the capture obtained on the destination system, doesn't that _clearly_ point the blame at the destination system? And since Ethereal is running in promiscuous mode on that destination system and only "sees" a few of these packets, doesn't that implicate the NIC in the destination system?

Unfortunately, I don't have other resources available to use for testing. Also, a workaround is now in place and we are not able to disrupt operations to return to the non-working configuration for testing (I'm hosed :O). Yet a solution is expected.


Thanks,


Paul
10 REPLIES
Garry Fruth
Trusted Contributor

Re: Where Did the Packets Go?

I don't believe UDP guarantees delivery. Is it possible that the VMS box is sending packets at a rate faster than they can be recieved on the destination box; e.g. VMS sending at 100MB Full duplex and the reciever is at 10MB half? BTW, what is the destination box?
John Gillings
Honored Contributor

Re: Where Did the Packets Go?

Paul,

You need to work out the path(s) between the systems. What routers or firewalls may be between them? Routers are allowed to just drop certain packets, and firewalls are *supposed* to!

Remember that IP is not connection based, so consecutive packets don't need to follow the same path. It could be there are multiple paths, and one or more of them have some box that doesn't like your packets.

I guess the only way to find out for certain is to draw yourself a map, and sample each segment between any routers or firewalls to see who's responsible for dropping them.
A crucible of informative mistakes
Wim Van den Wyngaert
Honored Contributor

Re: Where Did the Packets Go?

1) It can be a router saturation. It simply drops what is too much.
2) It can be bad hardware. Do as ucx sho prot udp and ucx sho prot tcp and check for abnormal things (both origin and destination)
3) It can be a double IP address. In that case some device (router) may be sending the packets to a bad destination.
4) Who knows. Try ucx sho prot icmp to find other abnormalities.

Wim
Wim
Paul Nunez
Respected Contributor

Re: Where Did the Packets Go?

Thanks to everyone for contributing, but I still don't get it.

How could an intermediate system be dropping the packets if we see ALL the packets arrive on the spanned port on the switch to which the remote host (Windows 2003 Server on a DELL box) is directly connected?

If they all arrive at the spanned port, doesn't that imply they all reached the port being mirrored? These ports are on the same switch. If the packets were dropped before reaching the switch, they would not appear in the trace obtained on the spanned port, no?

And the only thing between the port to which the remote system is connected and that remote system is the cable to the remote system. I guess a bad cable could produce these results.?.

I don't understand how an intermediate router/firewall could be involved (but that's why I'm here :o).

My hands are being tied by the "customer"; he can't help isolate the problem through testing, so I'm left with conjecture. I'll likely need to push this up to the folks who handle complex problem management, but want to first be diligent in my efforts to isolate where the fault lies.

Thanks again...

Paul
Robert Gezelter
Honored Contributor

Re: Where Did the Packets Go?

Paul,

Unless you are actually on the cable to the recipient server, it is "probable cause", not a conclusion.

Since all the packets show up at the mirrored port, the problem could be in the switch or in how things are being received.

My suggestion is that you install a real hub on the Dell side of the switch, and hook a sniffer and the Dell server directly into the hub:

Cisco Switch --- Hub (100BaseT/Full Duplex)
| |
Dell Server Sniffer

Then cross compare the packets seen on the sniffer with the actual traffic reported on the Dell Server. Then analyze the resulting trace on BOTH systems.

I hope that the above is helpful.

- Bob Gezelter, http://www.rlgsc.com
Paul Nunez
Respected Contributor

Re: Where Did the Packets Go?

Ok, so I still have work to do to find the faulty device. Since I can no longer utilize the customer's environment to do so, I'm left with duplicating their environment in-house.

This issue has not been reported by any other user of our application and the code changes which resulted in this problem have been in wide use since Oct 2003. This customer's environment is not unique in regard to the network equipment (widely used Cisco gear). But heck, I can't even convince them to try a different port on the switch at this stage...

Thanks all,


Paul
Andy Bustamante
Honored Contributor

Re: Where Did the Packets Go?

You know that the traffic is getting to the Cisco switch.

Has the customer loaded any software updates from Microsoft or any anti virus/firewall software? Check the switch and the NIC on Windows system, do they argree on speed and duplex settings? Change the network cable from the switch to the Windows box. Change the network card on the Windows box.

You know the traffic is getting to the the destination Cisco switch, it's the last leg that I would focus on.
If you don't have time to do it right, when will you have time to do it over? Reach me at first_name + "." + last_name at sysmanager net
Robert Gezelter
Honored Contributor

Re: Where Did the Packets Go?

Paul,

Andy's comments are well taken.

I also must apologize about the diagram in my earlier posting. The HUB was connected to the Dell server and the sniffer, and thence connected to the switch (the ITRC posting software mangled my diagram).

I would also not be surprised if you are unable to replicate the problem in your lab. If the problem is indeed a incorrect setting, or a defective port/switch you will more likely than not be unable to reproduce it.

- Bob Gezelter, http://www.rlgsc.com
Doug Phillips
Trusted Contributor

Re: Where Did the Packets Go?

>Ok, so I still have work to do to find the
>faulty device. Since I can no longer
>utilize the customer's environment to do
>so, I'm left with duplicating their
>environment in-house.

There is no way you could do this. But, the question I have is: What "workaround" did you put in place?

>This issue has not been reported by any
>other user of our application and the code
>changes which resulted in this problem
>have been in wide use since Oct 2003. This
>customer's environment is not unique in
>regard to the network equipment (widely
>used Cisco gear).

It's unique in that something isn't working;-)

You've checked the obvious things (settings on NIC match setting on switch port), so that leaves: The switch's port is failing, you have a bad cable or connector, the NIC driver is corrupted, or the NIC is bad.

>But heck, I can't even convince them to
>try a different port on the switch at this
>stage...

That's the only way you'll find the problem without bringing down the system. Swap the system's port with the mirror port. Switch the system cable. These things you can do without major disruption. These things will either point to the port or cable as the problem or eliminate them as as suspects, leaving the driver or card. You can reinstall the driver first, and if that doesn't work replace the card but these will both disrupt things.

Otherwise, the customer is being unreasonable. How does the customer expect the problem to be fixed if you can't troubleshoot it?
Paul Nunez
Respected Contributor

Re: Where Did the Packets Go?


One idea I had was to take an OVMS system on site and use it for testing. That way we minimize the impact on their operations.

I agree there's little chance of duplicating in-house and have expressed so to the customer. Hopefully he'll agree to my idea, otherwise, I'll have to get respective management together on this...

Appreciate the help,

Paul