Lan0 occasionally drops w/no ems or broadcasts rx3600

[Doug_3]

Hi all, we have a rx3600 about 9 months old. Once or twice a week, for a second or less, it recently started to not respond to pings and telnet will freeze. If someone is concurrently preforming database access or file access telnet will drop the connections and the application will abort on the pc's.
No changes to the machine have occurred in the past month except to enable NTP sync to our internal NTP server. I disabled it 2 days ago and the error condition occured again this am.
If applications are not actively processing a database transaction they remain connected. This occurs 1-2 a week at random times, sometimes under high load and others when the machine has nothing occuring.

syslog, ems alerts, netstat, ioscan, lanscan all appear normal. lan1 is disabled.

I have not looked into ndd commands yet.

Ideas on how to problem solve this? I have not submitted a Support Case to HP yet.

Regards,
Doug
P.S., I am generous with points.

[singh sanjeev]

check the speed of the lan0...and its setting...

>>>lanadmin

[Doug_3]

Nothing unusual to my eyes:

PPA Number = 0
Description = lan0 HP PCI-X 1000Base-T Release B.11.23.0803
Type (value) = ethernet-csmacd(6)
MTU Size = 1500
Speed = 1000000000
Station Address = 0x1a4b084df2
Administration Status (value) = up(1)
Operation Status (value) = up(1)
Last Change = 462564875
Inbound Octets = 360589554
Inbound Unicast Packets = 571871454
Inbound Non-Unicast Packets = 214567220
Inbound Discards = 8658
Inbound Errors = 0
Inbound Unknown Protocols = 158802939
Outbound Octets = 3973584327
Outbound Unicast Packets = 793318207
Outbound Non-Unicast Packets = 284
Outbound Discards = 0
Outbound Errors = 0
Outbound Queue Length = 0

[singh sanjeev]

lanadmin -x 0
lanadmin -s 0

[BUPA IS]

please also post the output of
lanadmin -g mibstats_ext 0
Mike

[VK2COT]

Hello,

Two things come to my mind straight away
(I actually helped a major customer with
almost identical symptoms two weeks ago):

a) Check buffer cache settings:

# kctune | egrep "dbc_min_pct|dbc_max_pct"

They are very commonly "abused" kernel
parameters that generate problems like yours.

b) Is ICMP disabled on your LAN?
For example, if you cannot ping default
gateway, then make sure that
Dead Gateway Detect kernel parameter
is set correctly.

Cheers,

VK2COT

[Steven E. Protter]

Shalom Doug,

EMS uses the LAN to report problems.

If the lan goes away, its not going to let you know, because it can't.

What I like to do in such circumstances is have reliable, outside system monitor my production systems LAN, and they notify me via SMS or email if a production system is offline.

No system is fool proof.

As to the problem, I suspect the following areas:
* Bad NIC (use cstm or xstm to test)
* Bad Cat-5 cable or medium (replace)
* Bad network switch port (check router, try a different port)

It sounds to me like a bad NIC card, but that is just based on 11 years of systems administration experience.

SEP

[Basheer_2]

since your NIC is 1GB
did anybody change the speed on this using the lanadmin

physically check the NIC is positioned correctly.

remove the NIC cable at the server side, putback do the same thing at the swithc/hub side.

what does /var/adm/syslog/syslog.log show

[Doug_3]

Thanks everyone! This is now occuring regularly at 5:50 am most days and randomly once in a while. This suggests to me a process is throwing the NIC non-responsive. We have not added any scripts to cron. What processes could cause the NIC to become non-responsive?

Here is the output requested:
Speed = 1000 Full-Duplex.
Autonegotiation = On.
Speed = 1000000000

lanadmin -g mibstats_ext 0

LAN INTERFACE EXTENDED MIB STATISTICS DISPLAY
Mon, May 4,2009 09:15:25

Interface Name = lan0
PPA Number = 0
Description = lan0 HP PCI-X 1000Base-T Release B.11.23.0803
Interface Type(value) = ethernet-csmacd(6)
MTU Size = 1500
Speed = 1000 Mbits/Sec
Station Address = 0x001a4b084df2
Administration Status = up
Operation Status = up
Last Change = Thu Feb 12 11:08:34 2009
Inbound Octets = 352285531798
Inbound Unicast Packets = 605022212
Inbound Multicast Packets = 2327
Inbound Broadcast Packets = 218773096
Inbound Discards = 8658
Inbound Errors = 0
Inbound Unknown Protocols = 161560625
Outbound Octets = 238049967472
Outbound Unicast Packets = 813962291
Outbound Multicast Packets = 7
Outbound Broadcast Packets = 283
Outbound Discards = 0
Outbound Errors = 0
Counter Discontinuity Time = Sat Dec 20 22:14:27 2008
Physical Promiscuous Mode = false
Physical Connector Present = true
Interface Alias =
Link Up/Down Trap Enable = enabled

Ethernet Specific Extended Statistics Display

Index = 1
Alignment Errors = 0
FCS Errors = 0
Internal MAC Transmit Errors = 0
Frame Too Long Errors = 0
Internal MAC Receive Errors = 0
Symbol Errors = 0
Single Collision Frames = 0
Multiple Collision Frames = 0
SQE Test Errors = 0
Deferred Transmissions = 0
Late Collisions = 0
Excessive Collisions = 0
Carrier Sense Errors = 0
Control Field Errors = 0
Multicasts Accepted = 0
Duplex Status = fullDuplex
Rate Control Ability = false
Rate Control Status = rateControlOff
Collision Count = 0
Collision Frequency = 0

[Bill Hassell]

Check this value:

ndd -get /dev/ip ip_ire_gw_probe

If it reports 1, icmp (ping) response may have be turned off in your gateway (routers). When the network stack detects no ping, it assumes the gateway is dead and silently disables it. The goal is to tell the OS to use a different router in case link fails. However, this is quite a complex setup and seldom implemented. Turn it off with:

ndd -set /dev/ip ip_ire_gw_probe 0

and then to make the change permanent, edit the file /etc/rc.config.d/nddconf:

# Remove dead gateway detection
TRANSPORT_NAME[1]=ip
NDD_NAME[1]=ip_ire_gw_probe
NDD_VALUE[1]=0