Netstat question

[lawrenzo_1]

Hi all,

I am running nagios as the master monitoring tool on an intel box with o/s 2.6.19-1.2895.fc6. I run an nrpe daemon on AIX5.3 and have started experiencing some issues when wait is high on the client server .....

my question is not really a nagios question however if anyone has experienced the same issue then any advise would be much appreciated.

I am troubleshooting the network issue and could do with some help translating the data from netstat -s on the linux box:

[root@prodnag01 log]# netstat -s
Ip:
28233277 total packets received
0 forwarded
0 incoming packets discarded
27526451 incoming packets delivered
28036857 requests sent out
Icmp:
4875906 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
destination unreachable: 42648
echo requests: 99409
echo replies: 4733849
149016 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 49607
echo replies: 99409
Tcp:
4104006 active connections openings
51028 passive connection openings
66508 failed connection attempts
110 connection resets received
1 connections established
22531542 segments received
22377504 segments send out
238721 segments retransmited
0 bad segments received.
48116 resets sent
Udp:
76355 packets received
42648 packets to unknown port received.
0 packet receive errors
311570 packets sent
TcpExt:
2 invalid SYN cookies received
9 resets received for embryonic SYN_RECV sockets
73422 TCP sockets finished time wait in fast timer
3201 time wait sockets recycled by time stamp
6061 delayed acks sent
83 delayed acks further delayed because of locked socket
Quick ack mode was activated 5027 times
8057467 packets directly queued to recvmsg prequeue.
35413 packets directly received from backlog
4060531394 packets directly received from prequeue
119366 packets header predicted
495198 packets header predicted and directly queued to user
9105676 acknowledgments not containing data received
1419890 predicted acknowledgments
1329 times recovered from packet loss due to fast retransmit
39 times recovered from packet loss due to SACK data
47863 congestion windows recovered after partial ack
13 TCP data loss events
9 timeouts after SACK recovery
111 timeouts in loss state
2842 fast retransmits
7 forward retransmits
36 retransmits in slow start
142127 other TCP timeouts
TCPRenoRecoveryFail: 11
3 sack retransmits failed
2 times receiver scheduled too late for direct processing
25 DSACKs sent for old packets
10 DSACKs sent for out of order packets
1 DSACKs received
2 connections reset due to unexpected data
2 connections reset due to early user close
6 connections aborted due to timeout

I'm not sure if I should be alarmed with some of the following statistics:

9105676 acknowledgments not containing data received
1419890 predicted acknowledgments
1329 times recovered from packet loss due to fast retransmit
39 times recovered from packet loss due to SACK data
47863 congestion windows recovered after partial ack
13 TCP data loss events
9 timeouts after SACK recovery
111 timeouts in loss state
2842 fast retransmits
7 forward retransmits
36 retransmits in slow start

what is the best way to trouble shoot linux networks?

Thanks

Chris

[Steven E. Protter · Accepted solution]

Shalom Chris,

Big question, troubleshooting a Linux network.

1) Its not really a Linux network, because the network is really the switches and hardware. Linux uses it as does other clients.

2) Tools
tcpdump works the same as other linuxes. You can use this or ethereal(wireshark) to monitor trafic and look for issues. At least then you may have a suspect as of now you have summary data and no proof the Linux box is the issue.

/var/log/messages
A great place to look for Linux problems.

I would be a little concerned about the last paragraph of stats. It does not show goodness as the saying goes. If you are not a network person you might want to have someone look at the switch logs to see if they point to a culprit.

Good first experiment: Turn of nagios and see if the problem goes away. I suspect it will remain.

SEP

[lawrenzo_1]

ok Thanks Steve,

will look into those tools.

I'll look at some comparison sheets for other commands diff between hpux an linux.

[Steven E. Protter]

Shalom again:


Command HP-UX Linux Issues
tcpdump x x Command line options may vary.
ethereal/wireshare x x Same GUI
landadmin ifconfig -a
netstat x x Command line options vary.
l lanscan ifconfig -a

sar x x Command line options vary.

dmesg x x No command line optoins.


Sorry if the formatting is off.

SEP

[dirk dierickx]

also the following provides loads of quick info.

ntop - http://www.ntop.org/overview.html

[lawrenzo_1]

Thanks guys,