Socket queue/buffer introspection

[Ralph Grothe]

Hello,

I need to know how to get a glance at the contents of socket queues.

The reason are inexplicable tcp server latencies (e.g. telnets), and I thus wish to monitor the queues' states.

Regards
Ralph

[Sridhar Bhaskarla]

Ralph,

Did you consider looking at "Recv-Q" and "Send-Q" in netstat -an output?. YOu can get some information out of it though they may not be true representations of bottlenecks.

-Sri

[Ralph Grothe]

Sridhar,

yes, the two columns of send and receive queues from a "netstat -an -f inet" was also the first thing that came to my mind.

But I hoped for some tool that would give a more thorough introspection (e.g. contents, or state changes in queues on different layers of the tcp/ip stack, as well as maybe pointers to possible "memory leaks").

[rick jones]

If one is within the context of a process with access to the socket, there is always MSG_PEEK. However, from outside the process, it would basically mean finding the socket structure in the kernel and then following the pointers to the mblks chained off of it and such. I'm not aware of such a tool.

Lsof may give slightly more information, but perhaps not all of what you might want.

As for state changes in the stack, the only place that really has "state" would be TCP, and perhaps the socket/streamhead. TCP's state (connection anyhow) is displayed in that netstat -an output.

There is also the output of "ndd /dev/tcp tcp_status" which will have a little more information about the TCP connection.

It might help to know when in the lifecycle of these telnet connections you have these inexplicable latencies - there may be other things to check.

[Ralph Grothe]

Rick,

what's wrong with my invocation of tcp_status?
See ndd's responses below:

# uname -srv
HP-UX B.11.00 U

# ndd -h|grep status
ip_ill_status - Displays a report of all physical interfaces
ip_ipif_status - Displays a report of all logical interfaces
ip_ire_status - Displays all routing table entries
ip_udp_status - Reports IP level UDP fanout table
tcp_status - Get netstat-like TCP instances information
udp_status - Get UDP instances information.
ip_ill_config_status - Internal configuration option
ip_mc_filter_status - Internal Multicast filter option

# ndd -get /dev/tcp tcp_status
operation failed, Invalid argument

# ndd -?
usage: ndd [-get] network-device parameter
[-set] network-device parameter value
[ -h ] [ supported | unsupported ]
[ -c ]

# ndd -get /dev/tcp \?|grep status
tcp_status (read only)

[rick jones]

I seem to recall there being some issues with STRMSGSIZE and the quantity of data some ndd calls into the kernel wish to return. IIRC, the fix is to be on th elatest ndd patch and perhaps set STRMSGSIZE to 0 and regen and reboot.

The usual song and dance about being on the latest ARPA trasnport patch and its dependencies would likely apply as well.

[Ralph Grothe]

Rick,

there is a cumulative ARPA patch on this box

# swlist |grep -i arpa
PHNE_20436 1.0 cumulative ARPA Transport patch

However, as you mentioned, it may not be that recent.

And it doesn't seem to address any streams segment size

# swlist -l fileset -a readme PHNE_20436|grep -i sgsize
Blocking sendmsg() returns EMSGSIZE when recieve side is
overflow of revieve side resulting in the EMSGSIZE.


Nor is in the accompanying readme a match against tcp_status.

However, I've got an 11i box that shows a tcp_status dump, where the query succeeds.

But then I fear I'm not register-literate enough to interpret this correctly.

# uname -srv
HP-UX B.11.11 U
# ndd -get /dev/tcp tcp_status|wc -c
16389
# ndd -get /dev/tcp tcp_status|head
TCP dst snxt suna swnd cwnd rnxt
rack rwnd rto mss [lport,fport] state
00000000431b1be8 000.000.000.000 cc9be10e cc9be10d 00000000 00000000 0000
0000 00000000 00000000 01500 00536 [c001,0] TCP_LISTEN
00000000431b1828 000.000.000.000 cc97ae96 cc97ae95 00000000 00000000 0000
0000 00000000 00000000 01500 00536 [0,0] TCP_IDLE
00000000431b10a8 000.000.000.000 cc950084 cc950083 00000000 00000000 0000
0000 00000000 00000000 01500 00536 [16,0] TCP_LISTEN
00000000431b1468 000.000.000.000 cc96cd0a cc96cd09 00000000 00000000 0000
0000 00000000 00000000 01500 00536 [6f,0] TCP_LISTEN
00000000482818e8 000.000.000.000 1eb99c0b 1eb99c0a 00000000 00000000 0000
0000 00000000 00000000 01500 00536 [d7e2,0] TCP_LISTEN
0000000048281528 000.000.000.000 c76a13c1 c76a13c0 00000000 00000000 0000
0000 00000000 00000000 01500 00536 [0,0] TCP_IDLE
0000000061d9b468 000.000.000.000 ce79cc52 ce79cc51 00000000 00000000 0000
0000 00000000 00000000 01500 00536 [c354,0] TCP_LISTEN
000000004c149468 000.000.000.000 d1048fac d1048fab 00000000 00000000 0000
0000 00000000 00000000 01500 00536 [c07d,0] TCP_LISTEN
000000006139b8e8 127.000.000.001 11ee677a 11ee677a 00008000 00002073 11ed
7d3f 11ed7d3f 00008000 00500 04096 [13bb,d11f] TCP_ESTABLISHED

[Sridhar Bhaskarla]

Ralph,

Though it is not going to give you indepth analysis of your socket queues, you might want to install ethreal and find if the problem is in fact with your tcp servers. While it is running, duplicate your problem and follow the tcp stream. You can see each sequence and time it took to send|receive the acknowledgements. You can also see details for each packet.

You could do this with nettl and tcpdumps too. But I find it much easier with ethereal.

-Sri

[rick jones]

Indeed, that patch has been superceded.

You also want to look for a "standalone" ndd patch PHNE_26125. PLuggins some of that in to some magical mystery tools tells me that the set of patches you want to be "up to date" wrt the Transport would be:

PHCO_23651 fsck_vxfs(1M) cumulative patch
PHCO_24437 LVM commands cumulative patch
PHCO_27375 cumulative SAM/ObAM patch
PHCO_29380 user/group(add/mod/del)(1M) cumulative patch
PHKL_18543 PM/VM/UFS/async/scsi/io/DMAPI/JFS/perf patch
PHKL_20016 2nd CPU not recognized in G70/H70/I70
PHKL_23409 NFS, Large Data Space, kernel memory leak
PHKL_28150 LVM Cumulative Patch w/Performance Upgrades
PHKL_28593 VxFS 3.1 cumulative patch: CR_EIEM
PHKL_29385 IDS/9000; syscalls; eventports; dup2() race
PHKL_29434 POSIX AIO;getdirentries;MVFS;rcp;mmap/IDS;
PHKL_29648 Probe,IDDS,PM,VM,PA-8700,AIO,T600,FS,PDC,CLK
PHNE_26125 ndd(1M) cumulative patch
PHNE_26771 cumulative ARPA Transport patch
PHNE_27902 Cumulative STREAMS Patch

Also, STRMSGSIZE is a kernel tunable, not an error return value, so you would tweak that via SAM.

As for the output of ndd -get /dev/tcp tcp_status, assuming I don't fumble finger anything, the "TCP" column is a reference to the intenal TCP state location. "dst" is the destination IP address. "snxt" is the next sequence number that will be send when data is provided by the application. "suna" is the lowest sequence number of data for which we've sent, but not received an ACKnowledgement. "swnd" I _think_ is how much more data TCP will accept from the local application before it exerts local flow control. "cwnd" is the current size of the "congestion window." "rnxt" is the sequence number we next expect to receive. I think "rack" is the last sequence number we have acked. "rto" is the current value of the retransmission timeout. "mss" is the maximum segment size - the limit to the quantity of data we will send in any one TCP segment. "lport" is the local port number. "fport" is the "foreign" or "remote" port number. and finally, "state" is the TCP connection state