Monitoring network traffic?

[roose]

Hi folks,

Is it possible to determine what process is contributing to a server's high network utilization from a VMS node? If yes, what commands should I use and how to interpret the output of that command, meaning, which value will lead my to my top-network-using process?

Our servers are on OpenVMS v7.3-2 and TCP/IP v5.4 ECO5.

[Karl Rohwedder · Accepted solution]

From VMS you may use a simple 'show DEV BG/Fu' and check for high/changing operations counts.
Inside TCPIP there is a SHO DEV BGxxx /Full or /CONTIN to monitor candidates.

regards Kalle

[Volker Halle]

Roose,

network IOs are typically buffered IOs, so you could start with MONITOR PROCESSES/TOPBIO

Once you have found a process with high BUFIO, you can use SHOW PROC/ID=xxx to see the devices allocated to this process and can use SHOW DEV/FULL BGxxx or NETxxx (or OSxxx) to watch the IO operation rates. Depending on the network protocol, you could then also use more specific TCPIP or DECnet (NCP/NCL) commands to look at read/write IO counters.

Volker.

[labadie_1]

And you can monitor a bg device with

tcpip sh dev bgxxx: /continuous

[labadie_1]

For Decnet IV, you will see the potential problems between nodes with a command similar to
$ pipe mc ncp sh k node cou | sea sys$pipe remote,"Response timeouts"

[roose]

Thanks for the quick replies. So, there is no 1-quick way of identifying these processes then? I was thinking of a command that might be similar to: monit proc/topnetwork_user :)

How about VMS freeware tools then?

[labadie_1]

$ monitor decnet
will show the decnet traffic.

$ @sys$startup:tcpip$define_commands
will setup the netstat symbol and others

So you will be able to do
netstat -an
and other commands, but you are right, there is a need for your new monitor item !

:-)

[Wim Van den Wyngaert]

A prototype I made just now because I also need this info.

@xxx 5

will show the bg devices with more than 5K traffic per minute (output after 1 minute !)
With usefull info such as destination, username, image, etc.

Wim (ucx 5.3 on 7.3)

[labadie_1]

useful code, Wim !

A Sda extension would be a good idea for this...

[Wim Van den Wyngaert]

Optimized it a bit and sorted the output.
Was amazed with wath I found on my prod system (1 program pumping 10 MB per minute all the time).

Wim

[labadie_1]

with tcpip 5.4 eco 5 on Alpha Vms 7.3-2 your code breaks, at least because it needs an extra read somewhere.

May be I will try to write something more generic next week, as I will have some spare time :-)

[Wim Van den Wyngaert]

That's not that difficult. The output of ucx show dev/fu with output directed to file just STINKS. Full of garbage.

I now wrote something simular for decnet too (tested with ncl/NSP, no osi here). No NCP either.

I'll just post it here too.

Wim

[labadie_1]

Wim wrote
The output of ucx show dev/fu with output directed to file just STINKS. Full of garbage.

Should I say that I agree at 102% ?

May be it is better to do, instead of
$ tcpip sh dev bgxxx:/fu
something like
$ ana/sys
tcpip sh dev bgxxx:/fu

The output has more data, but it seems to be correctly formatted.

And we go nearer an Sda extension, as building a command file with sda> tcpip commands, and parsing the output...

:-)

[labadie_1]

The simple command
$ pipe wr sys$output "tcpip sh dev /fu" | ana/sys | sea sys$pipe "Device_socket",Bytes tr",Service,Host,Port

gives good "data".

Just add some awk ( awk :== $ sys$common:[syshlp.examples.tcpip.snmp]gawk ), Perl or Python formatting, and it should be fine.

[labadie_1]

As we now have more and more very useful Sda extensions, may be we should ask Vms Engineering some API to get easily data from any Sda extensions ?

Some votes for this on the advocacy site ?

[Paul Beaudoin]

Roose.

Enclosed is a procedure I have been using for some time. It records each IP or DECnet (IV +V) address that connects to the local box and records the traffic count to/from each address and some simple averages. One file collects/ caclulates and stores, the other displays the results. Best results when run over long periods.

Good Luck

Paul

[Sebastian Bazley]

Paul B:

the attachment seems to contain the output of some data collections, rather than the scripts to do the collection ...

[Sebastian Bazley]

Also worth considering combining the output of

TCPIP SHOW DEV

with

SHOW DEV BG /FULL

The BG information includes the PID and process name, which is not in the TCPIP output, and removing the /FULL qualifier makes the TCPIP output much easier to process.

===

BTW, for an example of how peculiar the TCPIP output is, try the following two commands:

$ TCPIP SHOW DEV BGxxx/FULL

$ PIPE TCPIP SHOW DEV BGxxx/FULL | TYPE SYS$PIPE

They should show much the same information; however the TYPE PIPE loses most of the input.

[Paul Beaudoin]

Roose,

My applogies - I picked up the wrong file! This one contains the procedures and one aditional one that will monitor phase V only.

Regards

Paul

[WWarren]

For a telnet, no byte or I/O counts appear to be kept (at least for TCP/IP for OpenVMS v5.5). So does a SH DEV/FULL BGnnn output really capture everything you want?


TCPIP> sh dev/full bg17106
Device_socket: bg17106 Type: STREAM
LOCAL REMOTE
Port: 23 49611
Host: w.x.y.z1 w.x.y.z2
Service: TELNET

RECEIVE SEND
Queued I/O 0 0
Q0LEN 0 Socket buffer bytes 0 2
QLEN 0 Socket buffer quota 4380 4380
QLIMIT 0 Total buffer alloc 0 512
TIMEO 0 Total buffer limit 35040 35040
ERROR 0 Number of XONs 0 0
OOBMARK 0 Number of XOFFs 0 0
I/O completed 0 0
Bytes transferred 0 0

Options: REUSEADR KEEP
State: ISCONNECTED PRIV ASYNC
RCV Buff: ASYNC
SND Buff: ASYNC

[Sebastian Bazley]

Same with VMS 7.3-2 - the counts do not seem to increase for TCPIP SHO DEV BGnnn.

However the system command

$ SHOW DEVICE BGnnnn /FULL

does show the I/Os.

[labadie_1]

Some formatting with awk.

Do
$ set term/wid=132
$ gawk :== $ sys$common:[syshlp.examples.tcpip.snmp]gawk.exe

then
$ cre a.awk
ty l.awk
BEGIN { format = "%-107s"
printf format, "Device | type | ploc | prem | host local | host remote | b
ytes rec| bytes sent| I/O rec | I/O sen |\n"
printf format, "________________________________________________________________
__________________________________________\n"}
/Device_socket/ {dev=$2; typ=$4}
/Port:/ {plocal=$(NF-1); pdist=$NF}
/Host/ {hlocal=$2; hdist=$3}
/Bytes/ {bytest=$3; bytesr=$4}
/O completed/ {iot=$(NF-1); ior=$NF ;
{printf("%-7s,%-8s,%5d,%5d,%16s,%16s,%10d,%10d,%10d,%10d\n",dev,typ,plocal,pdist
,hlocal,hdist,bytest,bytesr,iot,ior)}}
END { format = "%-107s"
printf format, "Device | type | ploc | prem | host local | host remote | b
ytes rec| bytes sent| I/O rec | I/O sen |\n"
printf format, "________________________________________________________________
__________________________________________"}


Then the command

$ pipe wr sys$output "tcpip sh dev bg/fu" | ana/sys | sea sys$pipe "Device_socket","Bytes",Service,Host,port,comp
leted | gawk/input=l.awk sys$pipe

gives some stats

You can add a sort at the end, for example

$ pipe wr sys$output "tcpip sh dev bg/fu" | ana/sys | sea sys$pipe "Device_socket"
,"Bytes",Service,Host,port,comp
leted | gawk/input=l.awk sys$pipe | sort/key=(pos:64,size:10,asc) sys$pipe sys$output

will show the bg device stats with the most bytes received at the end of the display.

replace ,asc) by ,des) in the previous command and you will have the most at the beginning of the display

To sort on the I/O received, use
/key=(pos:85,size=10)

There is a buglet at the moment with the formatting lines being both at the beginning or the end of the display after the sort.
:-(

This hack could be modified with the pid of the bg device - just f$getdvi("bg123","pid") and/or the processname on the bg device - f$getjpi(f$getdvi("bg123","pid"),"prcnam")

Have fun !