ping -n1 -m 5 timeout occurred on HP-UX11.23

[角谷寿哉]

I do alive monitoring of HP-UX server by ping every 1 second.

One day, ping timeout occurred, but next 1 second, ping reply OK.

On LAN monitor, echo request was found, by echo reply not found.

I wonder that the reason is ip_send_source_quench on this HP-UX was disable.

If socket buufer is fulled on HP-UX and ip_send_source_quench is disable, does ping timeout occurred?

[rick jones]

ICMP Echo (aka ping) Requests do not go to a socket buffer. They are turned-around inside the stack at "layer 3ish" I doubt that the setting of ip_send_source_quench would be involved in the timeout.

Depending on where the LAN monitor sits, it is possible the echo request didn't get to the server, or that the echo reply was dropped prior to getting to the LAN monitor. No way to know without much more detailed knowledge of the infrastructure.

You might consider checking netstat and lanadmin statistics for drop stats and such.

[角谷寿哉]

Hi rick , I'm CA1487274 from Japan.

Thank you for your reply(not echo reply).

I check netstat -s and lanadmin -g PPA.

# netstat -s
tcp:
687994215 packets sent
196728506 data packets (3900502468 bytes)
21385 data packets (7247834 bytes) retransmitted
467394041 ack-only packets (8171812 delayed)
42 URG only packets
27 window probe packets
60 window update packets
326544690 control packets
987512020 packets received
3642882261 acks (for 4047236735 bytes)
7629 duplicate acks
0 acks for unsent data
3617152158 packets (2577232002 bytes) received in-sequence
0 completely duplicate packets (0 bytes)
0 packets with some dup, data (0 bytes duped)
5943 out of order packets (6713778 bytes)
1 packet (1993411124 bytes) of data after window
0 window probes
49176901 window update packets
23828886 packets received after close
0 segments discarded for bad checksum
0 bad TCP segments dropped due to state change
72204864 connection requests
72783229 connection accepts
144988093 connections established (including accepts)
145045053 connections closed (including 57139 drops)
50403 embryonic connections dropped
3497947097 segments updated rtt (of 3497947097 attempts)
39314 retransmit timeouts
7018 connections dropped by rexmit timeout
27 persist timeouts
139734 keepalive timeouts
119860 keepalive probes sent
5 connections dropped by keepalive
0 connect requests dropped due to full queue
2034 connect requests dropped due to no listener
0 suspect connect requests dropped due to aging
0 suspect connect requests dropped due to rate
udp:
0 incomplete headers
0 bad checksums
0 socket overflows
ip:
1309257704 total packets received
0 bad IP headers
48 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped after timeout
4 packets forwarded
0 packets not forwardable
icmp:
121384950 calls to generate an ICMP error message
0 ICMP messages dropped
Output histogram:
echo reply: 6648906
destination unreachable: 114736047
source quench: 0
routing redirect: 0
echo: 0
time exceeded: 0
parameter problem: 0
time stamp: 0
time stamp reply: 0
address mask request: 0
address mask reply: 0
0 bad ICMP messages
Input histogram:
echo reply: 730342
destination unreachable: 115371718
source quench: 0
routing redirect: 0
echo: 6648905
time exceeded: 0
parameter problem: 0
time stamp request: 0
time stamp reply: 0
address mask request: 0
address mask reply: 0
6648906 responses sent
igmp:
0 messages received
0 messages received with too few bytes
0 messages received with bad checksum
0 membership queries received
0 membership queries received with incorrect fields(s)
0 membership reports received
0 membership reports received with incorrect field(s)
0 membership reports received for groups to which this host belongs
0 membership reports sent
ipv6:
1251 total packets received
0 bad IPv6 headers
0 fragments received
0 fragments dropped
icmpv6:
0 calls to generate an ICMPv6 message
0 calls to generate an ICMPv6 error message
0 ICMPv6 messages dropped
0 ICMPv6 error messages dropped for rate control
Output histogram:
destination unreachable: 0
administratively prohibited: 0
time exceeded: 0
parameter problem: 0
packet too big: 0
echo: 0
echo reply: 0
router solicitation: 0
router advertisements: 0
neighbor solicitation: 0
neighbor advertisement: 0
redirect: 0
group query: 0
group response: 0
group reduction: 0
Input histogram:
destination unreachable: 0
administratively prohibited: 0
time exceeded: 0
parameter problem: 0
packet too big: 0
echo: 0
echo reply: 0
router solicitation: 0
router advertisements: 0
neighbor solicitation: 0
neighbor advertisement: 0
redirect: 0
group query: 0
group response: 0
group reduction: 0

# lanadmin -g 2

LAN INTERFACE STATUS DISPLAY
Wed, Oct 6,2010 15:09:08

PPA Number = 2
Description = lan2 HP PCI-X 1000Base-T Release B.11.23.0609
Type (value) = ethernet-csmacd(6)
MTU Size = 1500
Speed = 100000000
Station Address = 0x17a4a40516
Administration Status (value) = up(1)
Operation Status (value) = up(1)
Last Change = 790268261
Inbound Octets = 1443035332
Inbound Unicast Packets = 298908822
Inbound Non-Unicast Packets = 299828687
Inbound Discards = 0
Inbound Errors = 2953
Inbound Unknown Protocols = 597
Outbound Octets = 2349316958
Outbound Unicast Packets = 3153123240
Outbound Non-Unicast Packets = 24784021
Outbound Discards = 0
Outbound Errors = 0
Outbound Queue Length = 0
Specific = 655367

Ethernet-like Statistics Group

Index = 1
Alignment Errors = 40
FCS Errors = 3091
Single Collision Frames = 0
Multiple Collision Frames = 0
Deferred Transmissions = 0
Late Collisions = 0
Excessive Collisions = 0
Internal MAC Transmit Errors = 0
Carrier Sense Errors = 0
Frames Too Long = 0
Internal MAC Receive Errors = 0

# lanadmin -x 2
Speed = 100 Full-Duplex.
Autonegotiation = On.

I don't find any susupicious data without Speed = 100 Full-Duplex.

NIC supports 1000 Full-Duplex and L2SW set port on auto.

[Bill Hassell]

The FCS and Alignment errors in lanadmin -g look suspiscious, but they may have occurred a long time ago. Use lanadmin to clear the statistics and see if errors are counted again.

[rick jones]

If I were forced to guess, it would be that your unfortunate ICMP Echo Request was one of the 3000 or so packets dropped at layer two - the Inbound Error and FCS error stats in the lanadmin output.

Since ICMP Echo Reuqests only use IP and ICMP, you can shorten your netstat search a bit:

netstat -s -p ip
netstat -s -p icmp

[角谷寿哉]

I got lanadmin of the NIC error occurred.

As a result, Alignment Errors and FCS Errors are occurred on the NIC cards.

1 error / 500,000 packets

I wonder NIC cards or LAN cable ?

Investigation continues ...

[rick jones]

FCS translates to "Frame Check Sequence" - in essence it is a link-level checksum across the frame. So, an FCS error means that the FCS value in the frame did not match the value the NIC computed as it received the frame. My guess would be an ever so slightly marginal cable. I must confess that at the moment I cannot recall the implications of an alignment error - but I suspect some web searching could find that.

[chris huys_4]

Hi,

"Standard" fcs/alignment errors are a consequence of a mismatch between nic card speed/duplex settings and networkswitchport speed/duplex settings.

Is it certain that the networkswitchportsettings, that is "connected" to the above lancard, is set on autonegotiation ?

What sort of networkswitch is the lancard connected to ?

Greetz,
Chris

[rick jones]

Given the magnitude of traffic shown in the lanadmin stats, and the comparative low number of FCS errors, unless it was for a transient duplex-mismatch when someone was mucking with cables, my money would be on something else. I would expect a considerably higher percentage of FCS errors for a duplex mismatch.