topic Re: ping breaks in Operating System - HP-UX

ping breaks

KINGSLEY_1 — Mon, 21 Jul 2014 01:45:34 GMT

Dear All,

We installed HP-UX 11.31 DCOE on two VMs on BL890i4. After these, we installed TEMENOS T24 R12 and oracle database R12 on the VMs respectively.

Under normal circumstances you are able to ping the two VMs without any break, but as soon as the TEMENOS application services is started the ping sessions break intermittently. This has made it impossible for us to run the application.

kindly help.

kingsley

P.S. This thread has been moved from HP-UX>System Administration to HP-UX > networking. -HP Forum Moderator

Re: ping breaks

Bill Hassell — Fri, 18 Jul 2014 11:01:56 GMT

I assume when you say "ping breaks", you mean that one ping fails to return. The ICMP Echo Request message (ping) does not use normal packet handling in TCP/IP which means ping is not guarenteed to return reliably. It is a very primitie test and any application that uses ping as a health check needs a rewrite. Certainly you may have a networking problem that pushes ping to bottom of the stack but you'll need to look at netstat -s and lanadmin -g to gain some insight to the network health.

Re: ping breaks

Torsten. — Fri, 18 Jul 2014 13:02:55 GMT

I would suggest to check the network configuration again. It could be complicated, e.g. if you use virtual connect, APA, etc ... does the host have multiple addresses in the same subnet by chance? Just an example ...

Re: ping breaks

KINGSLEY_1 — Fri, 18 Jul 2014 14:34:11 GMT

Hi!

this are the output for netstat -s and lanadmin -g

[root@ghaccuet24np]/#netstat -s

tcp:
        73805207 packets sent
                71122150 data packets (50263059006 bytes)
                69457 data packets (22448487 bytes) retransmitted
                2679724 ack-only packets (65666 delayed)
                17 URG only packets
                0 window probe packets
                12777 window update packets
                3441 control packets
        71858423 packets received
                69510886 acks (for 50263070575 bytes)
                221 duplicate acks
                0 acks for unsent data
                71656948 packets (86175455255 bytes) received in-sequence
                0 completely duplicate packets (0 bytes)
                0 packets with some dup data (0 bytes duped)
                19 out of order packets (75460 bytes)
                0 packets (0 bytes) of data after window
                0 window probes
                842 window update packets
                35 packets received after close
                0 segments discarded for bad checksum
                0 bad TCP segments dropped due to state change
        672 connection requests
        763 connection accepts
        1435 connections established (including accepts)
        1500 connections closed (including 170 drops)
        116 embryonic connections dropped
        69468312 segments updated rtt (of 69468312 attempts)
        69158 retransmit timeouts
                104 connections dropped by rexmit timeout
        0 persist timeouts
        63 keepalive timeouts
                45 keepalive probes sent
                0 connections dropped by keepalive
        0 connect requests dropped due to full queue
        10 connect requests dropped due to no listener
        0 suspect connect requests dropped due to aging
        0 suspect connect requests dropped due to rate
udp:
        0 incomplete headers
        0 bad checksums
        0 socket overflows
ip:
        71920016 total packets received
        0 bad IP headers
        0 fragments received
        0 fragments dropped (dup or out of space)
        0 fragments dropped after timeout
        0 packets forwarded
        0 packets not forwardable
icmp:
        54804 calls to generate an ICMP error message
        7 ICMP messages dropped
        Output histogram:
         echo reply: 19616
         destination unreachable: 35188
         source quench: 0
         routing redirect: 0
         echo: 0
         time exceeded: 0
         parameter problem: 0
         time stamp: 0
         time stamp reply: 0
         address mask request: 0
         address mask reply: 0
        0 bad ICMP messages
        Input histogram:
         echo reply: 115
         destination unreachable: 34572
         source quench: 0
         routing redirect: 0
         echo: 19616
         time exceeded: 0
         parameter problem: 0
         time stamp request: 0
         time stamp reply: 0
         address mask request: 0
         address mask reply: 0
        19616 responses sent
igmp:
        0 messages received
        0 messages received with too few bytes
        0 messages received with bad checksum
        0 membership queries received
        0 membership queries received with incorrect field(s)
        0 membership reports received
        0 membership reports received with incorrect field(s)
        0 membership reports received for groups to which this host belongs
        0 membership reports sent

[root@ghaccuet24np]/#lanadmin -g

Error - must enter a PPA Number

Usage:   lanadmin [-a] [-A station_addr]
                  [-c]
                  [-m] [-M mtu_size]
                  [-p]
                  [-r]
                  [-R]
                  [-s] [-S speed]
                  [-x] [-X] options
                  [-g] [get_options]
                  [-V] [VLAN-commands]
                  PPA
                  [-e]
                  [-t]
Default: lanadmin

a             Display current station address corresponding to PPA Number.
A             Set new station address corresponding to PPA Number.
station_addr New station address in hex with '0x' prefix.
c             Clear LAN statistics registers for the interface corresponding to PPA Number.
on|off        Enable or disable 802.5 source routing protocol.
m             Display current MTU size corresponding to PPA Number.
M             Set new MTU size corresponding to PPA Number.
mtu_size      New MTU size in bytes.
r             Reset the local LAN interface card corresponding to the PPA Number.
R             Reset MTU size corresponding to PPA Number to default.
s             Display current speed setting corresponding to PPA Number.
S             Set new speed setting corresponding to PPA Number.
speed         New speed setting in Mbits/second.
x             Display driver specific options corresponding to PPA Number.
X             Set driver specific options corresponding to PPA Number.
options       Specific options for -x/-X.
g             Get the settings corresponding to option specified by get_options.get_options   Specific options for -g.
p             Display usage information corresponding to PPA Number.
V             Invokes VLAN specific commands.
VLAN-commands Commands specific to VLAN and their option string.
PPA           PPA Number of interface. Req'd if any options above are selected.
e             Echo commands.
t             Terse prompts.

Re: ping breaks

Bill Hassell — Fri, 18 Jul 2014 23:13:13 GMT

lanadmin -g requires the lan number being investigated. So for lan0, lanadmin -g 0, for lan4, lanadmin -g 4

In your netstat listing, the retransmit timeout value appears to be high but it is cumulative so run this regularly:

netstat -s | grep 'retransmit timeouts'

If this continues to grow, you may need some work on your network (switches, routers). Also make sure you have a recen ONC bundle installed:

# swlist | grep ONC

ONCplus B.11.31.18 ONC+ 2.3

Re: ping breaks

KINGSLEY_1 — Mon, 21 Jul 2014 08:23:34 GMT

Hi Bill!

The follwing are the output for lanadmin -g lan0 for the tow servers hosting the application and the database:

/ # lanadmin -g 0

LAN INTERFACE STATUS DISPLAY
Mon, Jul 21,2014 08:18:23

PPA Number                      = 0
Description                     = lan0 HP PCI AVIO LAN Release B.11.31.1403
Type (value)                    = ethernet-csmacd(6)
MTU Size                        = 1500
Speed                           = 10000000000
Station Address                 = 0xea05762d8196
Administration Status (value)   = up(1)
Operation Status (value)        = up(1)
Last Change                     = 65
Inbound Octets                  = 3923016212
Inbound Unicast Packets         = 119420644
Inbound Non-Unicast Packets     = 2118074
Inbound Discards                = 252506
Inbound Errors                  = 0
Inbound Unknown Protocols       = 541150
Outbound Octets                 = 2802004791
Outbound Unicast Packets        = 116591716
Outbound Non-Unicast Packets    = 1194
Outbound Discards               = 167428
Outbound Errors                 = 2
Outbound Queue Length           = 0
Specific                        = 655367

Ethernet-like Statistics Group

Index                           = 1
Alignment Errors                = 0
FCS Errors                      = 0
Single Collision Frames         = 0
Multiple Collision Frames       = 0
Deferred Transmissions          = 0
Late Collisions                 = 0
Excessive Collisions            = 0
Internal MAC Transmit Errors    = 0
Carrier Sense Errors            = 0
Frames Too Long                 = 0
Internal MAC Receive Errors     = 0

LAN INTERFACE STATUS DISPLAY
Mon, Jul 21,2014 08:16:31

PPA Number                      = 0
Description                     = lan0 HP PCI AVIO LAN Release B.11.31.1403
Type (value)                    = ethernet-csmacd(6)
MTU Size                        = 1500
Speed                           = 10000000000
Station Address                 = 0x5edd537a1735
Administration Status (value)   = up(1)
Operation Status (value)        = up(1)
Last Change                     = 78
Inbound Octets                  = 202305010
Inbound Unicast Packets         = 270837
Inbound Non-Unicast Packets     = 1813620
Inbound Discards                = 215365
Inbound Errors                  = 0
Inbound Unknown Protocols       = 465564
Outbound Octets                 = 37850195
Outbound Unicast Packets        = 279490
Outbound Non-Unicast Packets    = 195
Outbound Discards               = 0
Outbound Errors                 = 2
Outbound Queue Length           = 5
Specific                        = 655367

Ethernet-like Statistics Group

Thanks.

kingsley

Re: ping breaks

Torsten. — Mon, 21 Jul 2014 08:33:18 GMT

Could you please post the output of

# nwmgr

Re: ping breaks

KINGSLEY_1 — Mon, 21 Jul 2014 08:37:58 GMT

Name/          Interface Station          Sub-   Interface      Related
ClassInstance State     Address        system   Type           Interface
============== ========= ============== ======== ============== =========
lan0           UP        0x5EDD537A1735 igssn    1000Base-T
lan900         DOWN      0x000000000000 hp_apa   hp_apa
lan901         DOWN      0x000000000000 hp_apa   hp_apa
lan902         DOWN      0x000000000000 hp_apa   hp_apa
lan903         DOWN      0x000000000000 hp_apa   hp_apa
lan904         DOWN      0x000000000000 hp_apa   hp_apa

/ # nwmgr

Name/          Interface Station          Sub-   Interface      Related
ClassInstance State     Address        system   Type           Interface
============== ========= ============== ======== ============== =========
lan0           UP        0xEA05762D8196 igssn    1000Base-T
lan900         DOWN      0x000000000000 hp_apa   hp_apa
lan901         DOWN      0x000000000000 hp_apa   hp_apa
lan902         DOWN      0x000000000000 hp_apa   hp_apa
lan903         DOWN      0x000000000000 hp_apa   hp_apa
lan904         DOWN      0x000000000000 hp_apa   hp_apa

Re: ping breaks

Torsten. — Mon, 21 Jul 2014 08:41:15 GMT

OK, just to ensure you have a single interface only.

Re: ping breaks

Bill Hassell — Tue, 22 Jul 2014 01:45:24 GMT

From netstat -s:

    69158 retransmit timeouts
       104 connections dropped by rexmit timeout

From lanadmin -g:

    Inbound Discards                = 252506
    Outbound Discards               = 167428

These indicate significant performance problems with the network interface.
Since this is an IVM using AVIO rather than a real LAN card on a dedicated server, the data throughput may be higher than the VM can handle.

Re: ping breaks

KINGSLEY_1 — Tue, 22 Jul 2014 11:03:58 GMT

Hi Boss!

What do i do then?

regards

kingsley

Re: ping breaks

Torsten. — Tue, 22 Jul 2014 11:29:40 GMT

How is the virtual NIC configured on the host? Maybe a LAN aggregate helps (APA)?