server not reachable

[DeafFrog]

Hi Gurus ,

               We have one Rx2660 running v3 , that will not be accessiable from time to time over the network.

There is no load on the server and neither it has rebooted , netfmt o/p is attached here .

FUSERDR:/tmp#netstat -p ip
ip:
        271098 total packets received
        0 bad IP headers
        0 fragments received
        0 fragments dropped (dup or out of space)
        0 fragments dropped after timeout
        0 packets forwarded
        0 packets not forwardable

FUSERDR:/tmp#netstat -p tcp
tcp:
        1785136 packets sent
                1419570 data packets (461380726 bytes)
                578 data packets (8872 bytes) retransmitted
                357970 ack-only packets (89497 delayed)
                0 URG only packets
                0 window probe packets
                78 window update packets
                2603033 control packets
        3358662 packets received
                1662541 acks (for 461638984 bytes)
                4 duplicate acks
                0 acks for unsent data
                1410771 packets (437417202 bytes) received in-sequence
                0 completely duplicate packets (0 bytes)
                2 packets with some dup data (260 bytes duped)
                30 out of order packets (1560 bytes)
                0 packets (0 bytes) of data after window
                0 window probes
                177814 window update packets
                7 packets received after close
                0 segments discarded for bad checksum
                0 bad TCP segments dropped due to state change
        1164519 connection requests
        140279 connection accepts
        1304798 connections established (including accepts)
        2329672 connections closed (including 1032754 drops)
        1025044 embryonic connections dropped
        1397673 segments updated rtt (of 1397673 attempts)
        579 retransmit timeouts
                8 connections dropped by rexmit timeout
        0 persist timeouts
        5057 keepalive timeouts
                3707 keepalive probes sent
                30 connections dropped by keepalive
        0 connect requests dropped due to full queue
        1025045 connect requests dropped due to no listener
        0 suspect connect requests dropped due to aging
        0 suspect connect requests dropped due to rate

FUSERDR:/#lanadmin -g 1

                      LAN INTERFACE STATUS DISPLAY
                       Tue , Jul 19,2011  11:16:34

PPA Number                      = 1
Description                     = lan1 HP PCI-X 1000Base-T Release B.11.31.0809
Type (value)                    = ethernet-csmacd(6)
MTU Size                        = 1500
Speed                           = 1000000000
Station Address                 = 0x1cc4fc33a2
Administration Status (value)   = up(1)
Operation Status (value)        = down(2)
Last Change                     = 0
Inbound Octets                  = 0
Inbound Unicast Packets         = 0
Inbound Non-Unicast Packets     = 0
Inbound Discards                = 0
Inbound Errors                  = 0
Inbound Unknown Protocols       = 0
Outbound Octets                 = 0
Outbound Unicast Packets        = 0
Outbound Non-Unicast Packets    = 0
Outbound Discards               = 0
Outbound Errors                 = 0
Outbound Queue Length           = 0
Specific                        = 655367

Ethernet-like Statistics Group

Index                           = 4
Alignment Errors                = 0
FCS Errors                      = 0
Single Collision Frames         = 0
Multiple Collision Frames       = 0
Deferred Transmissions          = 0
Late Collisions                 = 0
Excessive Collisions            = 0
Internal MAC Transmit Errors    = 0
Carrier Sense Errors            = 0
Frames Too Long                 = 0
Internal MAC Receive Errors     = 0

FUSERDR:/#nwmgr -g -v -c lan0
lan0:
   Interface State =UP
   MAC Address = 0x001CC4FC33A3
   Subsystem = igelan
   Interface Type = 1000Base-T
   Hardware Path = 0/1/2/0
   NMID = 1
   Feature Capabilities = Physical Interface
                          IPV4 Recv CKO
                          IPV4 Send CKO
                          VLAN Tag Offload
                          64Bit MIB Support
                          IPV4 TCP Segmentation Offload
                          UDP Multifrag CKO
   Feature Settings = Physical Interface
                      VLAN Tag Offload
                      64Bit MIB Support
   MTU = 1500
   Speed = 1 Gbps Full Duplex (Autonegotiation : On)    ### note the difference in lanadmin command ,lanadmin is correct.

FUSERDR:/#lanadmin -x 0
Speed = 1000 Full-Duplex.
Autonegotiation = On.

Regards,

[Naj]

Hi

i have some question

1. try nslookup, got any error?
2. netstat -rn and check default GW, are you able to ping default gateway?
3. did you try to reach your server either IP or hostname? any error?

Thanks

BR
Naj

[DeafFrog]

Hi Naj ,

     1. try nslookup, got any error?

FUSERDR:/#nslookup 192.168.7.4
Using /etc/hosts on:  FUSERDR

looking up FILES
Name:    FUSERDR
Address:  192.168.7.4
Aliases:  FUSERDR.bd.com

FUSERDR:/#nslookup FUSERDR.bd.com
Using /etc/hosts on:  FUSERDR

looking up FILES
Name:    FUSERDR
Address:  192.168.7.4
Aliases:  FUSERDR.bd.com

2. netstat -rn and check default GW, are you able to ping default gateway?

FUSERDR:/#netstat -rn
Routing tables
Destination           Gateway            Flags Refs Interface  Pmtu
127.0.0.1             127.0.0.1          UH    0    lo0       32808
192.168.7.4           192.168.7.4        UH    0    lan0      32808
192.168.7.0           192.168.7.4        U     2    lan0       1500
127.0.0.0             127.0.0.1          U     0    lo0       32808
default               192.168.7.62       UG    0    lan0       1500
FUSERDR:/#ping 192.168.7.62
PING 192.168.7.62: 64 byte packets
64 bytes from 192.168.7.62: icmp_seq=0. time=0. ms
64 bytes from 192.168.7.62: icmp_seq=1. time=0. ms
64 bytes from 192.168.7.62: icmp_seq=2. time=0. ms
64 bytes from 192.168.7.62: icmp_seq=3. time=0. ms

----192.168.7.62 PING Statistics----
4 packets transmitted, 4 packets received, 0% packet loss
round-trip (ms)  min/avg/max = 0/0/0

3. did you try to reach your server either IP or hostname? any error?

No error.Server will go for an outing for half hour / 15 minutes , during this time application running on it will not be accessiable.
afetr that will return back on network.This happens 4-5 times a day.

[Naj]

Hi,

Looks fine from the output but i do not sure what application that you have running on the server.

Anyway, what kind of access did you try to reach server? ssh?ftp? or just ping the server? if you are using some tools to access into the server, did you try to restart the service?

>during this time application running on it will not be accessiable.
afetr that will return back on network.This happens 4-5 times a day.

Do you mean during that period, all other access occurred?

Thanks

[Steven E. Protter]

Shalom,

The behavior is typical of either a DNS issue (see Naj post 1 item 2) or another system coming on the network with the same IP address.

[DeafFrog]

I use putty to connect to ssh@server .No over that period , no application on the server ( or the server itself) is accessiable.When server returns , no service ( including that of application ) needs restart .

[Steven E. Protter]

Shalom,

Log on the server via its console.

Lets poke around a bit.

Examine /var/adm/syslog/syslog.log

dmesg

SEP

[DeafFrog]

I have missed the formatted netfmt o/p attachment initally , that i have reattached.

Regards,

[DeafFrog]

HI Steve ,
Here's syslog , EMS error is for power adapter that has been there since long .....all that i can find since morning ( it 's evening here ) in the syslog:

Jul 19 07:33:51 FUSERDR sshd[4649]: Bad protocol version identification 'exit' from 80.0.2.236
Jul 19 07:58:35 FUSERDR sshd[4897]: SSH: Server;Ltype: Version;Remote: 80.101.111.120-2950;Protocol: 2.0;Client: PuTTY-Release-0.54
Jul 19 07:58:48 FUSERDR sshd[4897]: Accepted keyboard-interactive/pam for root from 80.101.111.120 port 2950 ssh2
Jul 19 08:03:15 FUSERDR sshd[4991]: SSH: Server;Ltype: Version;Remote: 80.101.111.120-2959;Protocol: 2.0;Client: PuTTY-Release-0.54
Jul 19 08:03:18 FUSERDR sshd[4991]: Accepted keyboard-interactive/pam for root from 80.101.111.120 port 2959 ssh2
Jul 19 08:15:09 FUSERDR sshd[5179]: SSH: Server;Ltype: Version;Remote: 80.101.111.120-2966;Protocol: 2.0;Client: WinSCP_release_3.8.2
Jul 19 08:15:10 FUSERDR sshd[5179]: Accepted keyboard-interactive/pam for root from 80.101.111.120 port 2966 ssh2
Jul 19 08:15:10 FUSERDR sshd[5179]: subsystem request for sftp
Jul 19 09:15:10 FUSERDR sshd[5179]: SSH: Server;Ltype: Kex;Remote: 80.101.111.120-2966;Enc: aes256-cbc;MAC: hmac-sha1;Comp: none
Jul 19 10:10:33 FUSERDR sshd[6375]: SSH: Server;Ltype: Version;Remote: 80.101.111.120-3335;Protocol: 2.0;Client: PuTTY-Release-0.54
Jul 19 10:13:02 FUSERDR sshd[6405]: SSH: Server;Ltype: Version;Remote: 80.101.111.120-3343;Protocol: 2.0;Client: PuTTY-Release-0.54
Jul 19 10:13:05 FUSERDR sshd[6405]: Accepted keyboard-interactive/pam for root from 80.101.111.120 port 3343 ssh2
Jul 19 10:15:12 FUSERDR sshd[5179]: SSH: Server;Ltype: Kex;Remote: 80.101.111.120-2966;Enc: aes256-cbc;MAC: hmac-sha1;Comp: none
Jul 19 11:15:14 FUSERDR sshd[5179]: SSH: Server;Ltype: Kex;Remote: 80.101.111.120-2966;Enc: aes256-cbc;MAC: hmac-sha1;Comp: none
Jul 19 11:16:26 FUSERDR sshd[7081]: SSH: Server;Ltype: Version;Remote: 80.101.111.120-3521;Protocol: 2.0;Client: PuTTY-Release-0.54
Jul 19 11:16:29 FUSERDR sshd[7081]: Accepted keyboard-interactive/pam for root from 80.101.111.120 port 3521 ssh2
Jul 19 11:38:17 FUSERDR su: + 1 root-fuserdr
Jul 19 12:32:07 FUSERDR sshd[8102]: SSH: Server;Ltype: Version;Remote: 80.101.111.120-3765;Protocol: 2.0;Client: PuTTY-Release-0.54
Jul 19 12:32:13 FUSERDR sshd[8102]: Accepted keyboard-interactive/pam for root from 80.101.111.120 port 3765 ssh2
Jul 19 14:58:15 FUSERDR sshd[9571]: SSH: Server;Ltype: Version;Remote: 80.101.111.120-4548;Protocol: 2.0;Client: PuTTY-Release-0.54
Jul 19 14:58:18 FUSERDR sshd[9571]: Accepted keyboard-interactive/pam for root from 80.101.111.120 port 4548 ssh2
Jul 19 15:31:55 FUSERDR EMS [2904]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource: "/system/events/ia64_corehw/core_hw" (Thre
shold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 190316555 -r /system/events/ia64_corehw/core_hw -n 19
0316571 -a
Jul 19 16:36:20 FUSERDR sshd[10678]: SSH: Server;Ltype: Version;Remote: 80.101.111.120-4852;Protocol: 2.0;Client: PuTTY-Release-0.54
Jul 19 16:36:26 FUSERDR sshd[10678]: error: PAM: Authentication failed for root from 80.101.111.120
Jul 19 16:36:29 FUSERDR sshd[10678]: Accepted password for root from 80.101.111.120 port 4852 ssh2

[Arunabha Banerjee]

I think some ip conflict (duplicate ip) is happening in your environment. May be someone trying to using server ip and that time you are unable to log into the server. So in future whenever you will face this issue then try to ping the IP Address from any other node and if you are getting any response then might be someone trying to using server IP. Also that time login to the server from console and check /var/adm/syslog/syslog.log file for any error.

[DeafFrog]

Hi Arun ,

             There is no IP conflict as i am not able to ping this IP , when the server is unreachable.

any further inputs ?

Regards,

[Mel Burslan]

Best advice anyone can give you right now is to ask your network support people to put a sniffer on this segment of network and you keep a constant ping from one of the servers that you know what connects to this server without any problems while the server is up.

When you see the server failing ping connections, you can tell the network guys to stop sniffing and analyze the sniffer logs for the time when ping command started failing. If there is a duplicate IP, it will show on the sniffer. If there is a network routing issue (like a router changing routing behavior for some reason) will show up on this sniffer log as well.

If sniffer log comes clean, as in, your server stops responding to ping requests, then it is your duty to find out what is happening to that interface at the moment it stops pinging, by setting traps, maybe running a nettl tracing.

If you are absolutely sure that there is nothing happening on your system and you have no internittently bad NICs on it, more than likely there is a network routing conflict.