Intermittent network adapter communications stoppage

[Russell Price]

Hi All,

We are running 2*HP C3000 workstations, both
with HP-UX 11i. At apparently random intervals
one or both of the machines will stop
communicating over the network. Terminal
sessions freeze, etc. The blackout only lasts
from between 10-30 seconds, and then
communication continues as if nothing had
happened. One machine is connected to a Cisco
Catalyst 3500XL. Both the workstation and the
switch have their ports locked at 100BASE-TX
FD ("Manual" for the HP). The other is
connected to a 10Mb passive hub, and its port
is set to 10BASE-T HD Manual. Neither machine
shows excessive network error stats (lanadmin).
There are no IP address conflicts. I have not
messed with the MAC addresses (they are set
to the factory default). Anybody have any
idea what might be going on here?

Thanks in advance.

R

[Wouter Jagers]

Never easy to find problems which are not continuous or reproduceable :-/

A couple of 'wonderings':
- When this happens, how does the system behave besides networking ? Are local shells reacting any slower ? Have you been able to run top or glance while this happened ?
- Does it sometimes happen that both machines get it at the -exact- same time ? (of course, should these be the only 2 machines you have you can't really know this)
- Do you have a machine besides those two ? (You might use it to monitor the others a bit, network wise)

greets
Wout

[Steven E. Protter]

You may have a problem with speed and duplex.

HP-9000 cards 100 BT and slower will not tolerate auto negotiate on most switches, especially cisco.

You may wish to set those speed setting manually in /etc/rc.config.d/hpbtlanconf

Also if there is another machine with the same ip address or virtual ip address, this will cause your symptoms. Check that with a ping while the workstations are shut down.

SEP

[Berlene Herren]

You can also check for duplicate IP and/or MAC address by looking at the nettl.log

#netfmt -f /var/adm/nettl.LOG001 > /tmp/log.out

Berlene

[Ron Kinner]

Do you have patch PHNE_23645? PHKL_23250? PHKL_23297? PHNE_28799? PHNE_28923? PHNE_26939?

Can you still ping local hosts when this problem happens?

Are you sure you have enough memory? Could the delay be caused by stuff being stored on the harddrive because of a lack of memory?

What NIC and driver are you using?

Ron

[Russell Price]

Hi All,

Thanks to everybody who took time to reply to my request for assistance.
Responses to each message:

Wout:
Yeah, I really dislike these intermittent problems. I'm not normally at
the console of the machines when the problem occurs, so I've only gotten
a chance to interact with one of them once as the problem was in progress.
It comes and goes so fast I don't have much time to run diagnostics.
However I did notice that while the network problem was happening the
machine seemed to be running normally. So long as a command doesn't
involve network access system response time is not such that one would
say "What's going on here, this machine is bogged down?" or similar.
The interruptions do not seem to occur on each machine at the same time,
but I shall get an independent check on this. Thanks for the idea. I have
no other operational HP machines besides these two, though I have lots of
other machines (SGI, Sun, Linux of various pedigree). I'll task one of
the others to keep track of both the HPs.

SEP:
As I mentioned in the original post, I have already manually set the HP
network interface speeds (in /etc/rc.config.d/hpbdlanconf), and also the
Cisco switch port. Autonegotiation should no longer be an issue. Also,
as per the original post, there are no other machines using the same IP
or MAC addresses. Duplicate network addresses are >not< the problem.

Berlene:
There are no current diagnostics from the
netfmt -f /var/adm/nettl.LOG000 command that shows any kind of IP
or MAC address problems/duplication. Just a few complaints about not
being connected to a network when I had disconnected the network cable
(at least I know logging is still working).

Ron:
We've installed patch PHKL_23297. Patches PHNE_23645, PHKL_23250,
PHNE_28799, PHNE_28923 and PHNE_26939 have not been installed.
Should I install the others as well? Once the problem has manifested itself,
the workstation will not respond to a ping until the network interface comes
back up (usually within 20-30 seconds). Each machine has enough
memory for what it is doing. One has 512MB and the other has 1024MB.
Neither machine is using any swap space. One is at 76% memory usage
(the one with 512MB), and the other is at 83%. Good thought though. The
NICs on the machines are built onto the system boards. An ioscan yeilds:
"HP PCI 10/100Base-TX Core". The system docs were of no help in trying
to find out exactly what they are. The driver is whatever the HP-UX install
loads by default for the built-in NIC.

R

[Berlene Herren]

dead gateway?

ndd -get /dev/ip ip_ire_status | grep -e IRE_GATEWAY -e flag

and see if the output for the default route says "IRE_GATEWAY DEAD". If so, you've got the dead gateway problem. This typically means that the default gateway is a firewall that refuses to answer ICMP echo requests.

Berlene

[Russell Price]

Berlene,

Thanks. Interesting, but no dead gateways to be seen. Output was:

0000000042886388 0000000000000000 0000000000000000 000.000.000.000
00000000 xxx.yyy.003.103 xxx.yyy.000.001 00000 00000 000 IRE_GATEWAY

Argh, I zigged when I should have zagged replying to this thread and
ended up posting a new message rather than a reply. Sorry.

R

[Ron Kinner]

I think I may have typed in 11.11 instead of 11 for the patches so some of them do not apply to your box. What I'm looking for are patches that replace the NIC drivers or the tcp/ip stack. I don't know exactly which NICs you have but a search on 11.0 for 100bt brings up the following possibilities:

s700_800 11.00 HP-PB 100Base-T cumulative patch PHNE_23982
notes PHNE_23982
notes
s700_800 11.00 HSC 100BT lan cumulative patch PHNE_25579
notes PHNE_25579
notes
s700_800 11.00 PCI 100BT lan cumulative patch PHNE_25580
notes PHNE_25580
notes
s700_800 11.00 Built-in PCI 100BASE-T patch PHNE_25907
notes PHNE_25907
notes
s700_800 11.00 4 port PCI 100BASE-T cumulative patch PHNE_26271
notes PHNE_26271
notes
s700_800 11.00 PCI 100B(Tx/Fx) cumulative patch PHNE_26293
notes PHNE_26293
notes
s700_800 11.00 cumulative ARPA Transport patch PHNE_21767
notes PHNE_26771
notes
s700_800 11.00 Cumulative STREAMS Patch PHNE_27902
notes PHNE_27902
notes
s700_800 11.00 LAN product cumulative patch PHNE_28143
notes PHNE_28143
notes
s700_800 11.00 EISA FDDI B.11.00.0[0-7] cumulative patch PHNE_28588
notes PHNE_28588
notes
s700_800 11.00 EISA 100BT cumulative patch PHNE_28636
notes PHNE_28636
notes
s700_800 11.00.[04-16] APA cumulative patch PHNE_26773
notes PHNE_28777
notes


You would have to read the release notes and decide if they apply to your box.

Ron

[Ron Kinner]

Oops I forgot to add: Search the patch database at:

http://www1.itrc.hp.com/service/patch/search.do?BC=patch.breadcrumb.main|&pageContextName=hpux:::

Ron

[Russell Price]

Ron,

Thanks for the info. We are running 11.11
(Marketing-jargon has it as 11i), so the
original patch list you gave was what I
used. One machine has subsequently settled
down after patching. The other (on a 10Mbit
Netgear hub), is still experiencing a few
brief interface interruptions each day, but
is much better than it was. I suspect now
that its problem may be that it dislikes the
passive hub we have it on. I'll try swapping
out the hub and see what happens. Thanks once
again for your kind assistance.

Russell