Network response

[Brian K. Arnholt]

I have an interesting problem, and am by no means a network expert. Here's the problem: I have two C-class workstations connected to a LAN through a router, and have found, about every 20 minutes, there is a real big slow down in the network speed. I ran a simple script to ping the machine from a remote system and found that about every 20 minutes, I get 100% data loss. My networking folks used an extended ping from the Router for 1000 times. They received back the first 420 packets before receiving a Source Quench (Destination Busy) message from the server. This means that the server is asking for a reduction or stop in IP datagrams.

Any suggestions? From my initial investigations, the workstations themselves seem to be running fine (no big CPU-hogging jobs, plenty of available memory.) Local connections to the machine work fine, we only see the slow down when performing remote connections via the particular router.

Thanks for the help,

Brian

[Sachin Patel]

Hi Brian,
I am guessing.
May be every 20 minutes someone is starting cronjob which is taking all resources.

Sachin

[Sandip Ghosh]

May be the remote M/C where you are connecting, is either doing some heavily loaded cronjob or it is downloading or uploading huge amount of data from the network. Have a look at the remote Server. Or you can look in the network on which it is connected. May be some other Server is doing the same so that the total network comes to a halt situation.

Sandip

[hpuxrox]

I have seen network backups bring a network to a slowdown. maybe check and see at what time your client runs their backups.

[MANOJ SRIVASTAVA]

Hi Brian

I am sure you must be having HPUX 11.00. We had a same problem and HP suggested to do a

ndd -set /dev/ip ip_send_source_quench 0


here is detail of the same
This problem has been identified and is addressed in SR 5003435396.
This problem will be fixed in the 11.01 version of the HP-UX operating
system. These messages can be safely ignored as they have absolutely no
impact on the operating system (performance or otherwise). Alternatively
these messages can be prevented by disabling source quench. For more
information see the sections below.

What is causing these messages?

At 11.0 the Streams Xport layer now passes the ICMP echo request to any
other process that has a socket open and bound to raw IP. The rpcd
rpcd/dced deamon opens a raw socket to listen to ICMP messages. This raw
socket is open by icmp_monitor routine of rpcd. The main function of
this routine is to check for error messages from DCE servers registered
in endpoint database of the host and it checks the socket every 5
minutes. It does not respond to or use the ICMP echo requests; however,
the socket queue becomes filled during the 5-minute delay causing the
source quench message. The fix being implemented in 11.01 will be to
increase the buffer size to 128 K and shorten the wait interval from 5
minutes to 2 minutes, thereby flushing the queue of these unwanted
messages before the queue becomes filled.

Why is it safe to ignore these messages or to turn them off?

A good disscussion of this is in TCPIP Illustrated Volume 1,
by Richard Stevens, pages 160-162.Here is an excerpt from page 161:

Although RFC 1009 [Braden and Postal 1987] requires a
router to generate source quenches when it runs out of
buffers, the new router requirements RFC [Almquist 1993]
changes this and says that a router must not originate
source quench errors. The current feeling is to deprecate
the source quench error, since it consumes network bandwidth
and is an ineffective and unfair fix for congestion.

Also, see RFC 1812, section 4.3.3.3 Source Quench for a good
discussion of the issues.

Also RFC 2001 has discussion on Congestion and why TCP should handle
this not ICMP:

Congestion avoidance and slow start are independent
algorithms with different objectives. But when congestion
occurs, TCP must slow down its transmission rate of packets
into the network and then invoke slow start to get things
going again. In practice, they are implemented together.

Exactly how do I disable source quench?

You can disable source quench in HP-UX 11.0 by executing this command:

ndd -set /dev/ip ip_send_source_quench 0

To disable Source Quench so that it can survive a reboot,

modify the /etc/rc.config.d/nddconf file as follows :
TRANSPORT_NAME[X]=ip
NDD_NAME[X]=ip_send_source_quench
NDD_VALUE[X]=0

Where X is the next logical numerical sequence in a table
of values, with X starting at 0.
--------------------------------------------------

DCE patch and dependencies:

3 PHCO_23651 fsck_vxfs(1M) cumulative patch
2 PHCO_23876 cumulative SAM/ObAM patch
1 PHCO_25883 cumulative 10.20 libc compatibility support
3 PHKL_18543 PM/VM/UFS/async/scsi/io/DMAPI/JFS/perf patch
3 PHKL_20016 2nd CPU not recognized in G70/H70/I70
3 PHKL_23956 Profile, virtual timers and disabling fix
3 PHKL_24027 VxFS 3.1 cumulative patch
1 PHKL_25906 Probe,IDDS,PM,VM,PA-8700,asyncio,T600,FS
1 PHKL_25999 syscall, msem_lock, umask cumulative patch
2 PHSS_21614 HP DCE/9000 1.7 Runtime cumulative patch


all the best

[Brian K. Arnholt]

Thanks for the suggestions about errant cronjobs, that did not seem to be the problem, but I did find a stray one that I killed.
I will forward Manoj's information to our networking folks, my brain hurts from just reading it!

Thanks for the help so far.

Brian

[Ron Kinner]

Appears we are talking about a Cisco router here. Have the network guys do a "show interface" on the interface that goes to your PC. See if they have any drops or errors. Then do the same thing for the interface which goes to the remote location. If the pipe to the remote is a serial link then they need to check the router on the other end too.

Drops indicate that the pipe is too full for some reason.

If they get drops have them turn on "ip accounting out" on the interface where the drops occur and then do a show ip accounting every few minutes. Look and see if you have one source - destination pair that is hogging the pipe.

Also ask them if they are running OSPF. Every 30 minutes it sends an update which might require routing changes or might hog too much bandwidth.

My bet is that the serial link is filling up but there could be noise bursts or route flapping or broadcast storms or network hogs. They may need to look into custom queuing or a bigger pipe.

The source quench is more a nuisance than a problem. It just makes it hard to check the LAN tho in reality a Q means the same as a ! as far as the LAN is concerned => good two way traffic.

Have them do their extended ping from the router to the remote and see what happens. THey may need to do more than 1000 pings to catch your problem since it only happens every 20 minutes unless you can give them an idea of when it is about to happen. Is it exactly 20 minutes or just about?

Ron

[Brian K. Arnholt]

Thanks for the information. I did apply your suggestion, however, I don't think that was soley my problem, we also reconfigured our network a while back, and they had to "ring out" some issues with it.