Finding out who's hogging the network

[thewho?]

Hello,
HPUX11.0, N4000, GlancePlus C.03.05.00

Sometimes the Network Graph in gpm shows a huge peak on PktOut. I'm trying to pin point who or what is responsible for this. I've been using:
netstat |grep -v "tcp 0 0"|grep -v "dgram 0 0"|more

And it comes back with some interesting stats, if I assume the Recv-Q and Send-Q fields show packet transmission, but I'm not certain of this and therefore can't tell for sure what's going on.

Thanks in advance.
Luis

[rick jones]

what is a "huge" spike in your opinion? if you want to find the source of the spike, some combination of tcpdump,ethereal,tcptrace et al might do the trick

www.tcpdump.org
www.tcptrace.org
www.ethereal.org?

to get some of the Gnu utilities required to build bits and pieces try devresource.hp.com

[thewho?]

Rick,
Thanks for the reply. Normally the Packets out is below 600. "Huge pike" is when I see that value above 2000 and it's only for a couple of samples (every 15 sec) several times a day.
What is the meaning of those 2 fields in the netstat output (Recv-Q and Send-Q)?

[Steven Sim Kok Leong]

Hi,

Another useful tool you would want to use for pinpointing the culprit is lsof. Use lsof to identify the processes associated with each established connection listed in the netstat table.

Hope this helps. Regards.

Steven Sim Kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

[Vincenzo Restuccia]

get lsof from http://hpux.cs.utah.edu/hppd/hpux/Sysadmin/lsof-4.55/

[Brian Hackley]

Luis,
If you can trap the issue quickly e.g. within seconds or as it occurs, nettl / netfmt will get you the PID of the user space process that is doing the packet burst. Recall that it is likely that nettl won't be able keep up during this burst period, but you should be able to create a nice big nettl trace file. Example is below. Also trace packets at ns_ls_ip so you get all IP packets. This will not capture non-IP traffic.

Step 1: Begin LAN Tracing to a Raw Trace File:
# nettl -tn 0x30800000 -e ns_ls_ip -size 1024 -tracemax 99999 -f /tmp/raw0

Step 2: Reproduce the Network "Event" or Error Condition

Step 3: Stop Tracing to the Raw Trace File As Soon As Possible
# nettl -tf -e all

Step 4: Format the Raw LAN Trace So You Can Read It using netfmt.

Hope this helps,
-> Brian Hackley

[Carlos Fernandez Riera]

Gpm is a graphic utility that can cause peeks if the measure time is too lower.

5 seconds is recomended in sar utility.

Rcv-Q and send-Q are queue of packets waiting to be processed by conection( if i recall well).

[thewho?]

Brian,
I tried the nettl/netfmt pair in one of our development servers and it does give me a lot of information, but how do I find the PID? It seems like the PID and UID are always [ICS] and -1, respectively.
I turned the trace on, made an rcp and turn it back off. I can see the packets and its contents but like I said the PID and UID are always the same.

[rick jones]

the sendq is the data sent to the remote for which we are awaiting an ACK. the recvq is data we have received from the remote, that has yet to be received by the local application.

as for getting the pid, you would use nettl/tcpdump to get the full four-tuple of local/remote IP, local/remote port number, and then use that information in lsof to find the pid.

all inbound driver processing happens on the ICS, data queued waiting to be sent (waiting say for window) will also end-up being sent out on the ICS.

2000 packets per second, even if full 1460 byte TCP segments, is still less than 2.8 MByte/s. so, if your system is connected to 100BT networks, it isn't all that much of a "spike" relative to network capacity.

you might also do some periodic ps -ef's - if you see ftps or rcp processes, those could generate short spikes - does anyone transfer files to/from this system?

[thewho?]

Rick,
Yes, there are some people that ftp to this server that's the reason why I'm looking for a method to determine who is responsible for the load.
Eventhough those peaks are relatively small, one can notice them (you actually wait for the cursor to come back when typing a command).
As you can see, I'm not very familiar with networking issues, let me analyze the information and tools people in the forum has provided ( that's why I have not assigned more than 7 points to any answer, I have not tried all the suggestions). Thanks to all that responded.

[Philip Chan_1]

Hi Luis,

I used to use a software called LinkView Pro to trace the people who hogged the network.

http://www.tinwald.com/linkview_classic/index.html

Rgds,
Philip