How to resolve networking freeze issue

[Andreas D. Skjervold]

Hi

Have been here with similar question earlier about a L2000 randomly freezing for approx 15-30 sec then resolving the situation and operation continues as normal.

I believed this was a HW issue as I first got a disk error and then some error reported in STM about the Core I/O card. These errors are now corrected by HP tech.

But the freeze continues to happen.

How should I monitor the system to see whats happening at the freeze moment?

This is a 11i (11.11 upgrade from 11.0) with sept 2001 QualityPack.
Using NIS and lots of NFS filesystems.
No errors in syslog.

Andreas

Andreas

[Roger Baptiste]

hi,

The freezing could be due to many reasons, not neccesarily network. Even a disk failure or intermittent disk timouts can cause a system to seemingly hang up. Two places
to check this would be in the dmesg/console output. Set up dmesg in cron to send output to a file at say every five minute intervals.
/usr/sbin/dmesg -> /var/adm/messages. That will catch some errors. For details on what was exactly running on the system during the time of freeze, look at Measureware/perfview logs. It has lots of information, including the processes being run at that time.

HTH
raj

[harry d brown jr]

Are you runing EMS or any other kind of monitoring stuff? Also, have you checked with you networking people to see if they notice anything unusual?

live free or die
harry

[Andreas D. Skjervold]

Hi

EMS reports nothing. It did earlier, but that was due to bad disk. This is now replaced, but the freeze remains.

Suspect that this might be a networking issue but I'm not able to sort out what.
Unfortunately I do not have any networking people at the moment, so what should I look for in the network?

Andreas

[Roger Baptiste]

<>

glance, option l and see
whether there are collisions or errors in the stats.

do a ping to the DNS server and see the icmp response time.

Check your dns settings and reverseloop settings.
(/etc/nsswitch.conf ; /etc/resolv.conf ) and make sure the setting is as it is supposed to be.

HTH
raj

[Darrell Allen]

Hi Andreas,

It would probably be helpful to know how you notice the freeze. Is it certain commands? Is everyone or just a few people affected? Affecting people on the same subnet or over a WAN? Is the console affected?

DNS name resolution could be a problem. The physical network could also. Are ping response times steady or erratic?

Darrell

[harry d brown jr]

Andreas,

You really need the help of your network people to assist you to determine if it is a router or hub issue.

As asked earlier, how do you detect these network pauses?

live free or die
harry

[Andreas D. Skjervold]

Hi Raj

I notice the freeze by simply working on the server, typing in vi or just commands, or using sam. Simply doing anything, then suddenly, everything freezes, and at the same time so does it for all the users on this system. Other systems at same subnet are not affected (as far as I can see)

Have not been able to check if this also includes the console, as I've had someproblems with an broken WebConsole, but will check this on monday.

One thing I have found on this server that differs from my other systems are when checking the routing tables, netstat -rn gives a quick report with the IP numbers, but when typing in netstat -r to get only the hostnames, the output have a 3-5 sec delay while looking up a hostname...
So maybe this is a name resolution problem.
The most disturbing part of this routing issue is that the host that show up delayed is a system that shouldnt be there. Its a server from a completely different subnet from which I have suspected no traffic at all.
And; the haostname on this system, isn't in /etc/hosts, nor in NIS (ypcat hosts), and I'm not using DNS. But still the name comes popping up...?

Using Glance, I find nothing except the fact that when the freeze occures, the server relaxes; 100% idle...

Andreas

[Darrell Allen]

Hi again,

Seems there definitely is a name resolution problem since netstat -r has a delay on the host that is not in /etc/hosts or NIS. So you could add that host. The route itself could be a dynamic route picked up by gated if you are running that. Check /etc/rc.config.d/netconf to see if the route is a static route and / or if gated is enabled.

Darrell

[Roger Baptiste]

<>

Check the /etc/rc.config.d/netconf file and see whether you have a route for this system in there?

Also look at lanscan/ifconfig output
and recheck the configuration
of the lan ports.

If you are using NIS, then make sure there is no problem at the NIS end.

To make sure you are (not)using NIS or DNS: do
nslookup
>policy


<>

Well, in that case i can think
of only two suspects, either
a bad/flaky disk connection
or a name/IP resolution problem. If it is the disk problem, it should get recorded in the syslog file as timeout or lbolt errors.

HTH
raj

[Andreas D. Skjervold]

Hi

Have checked:
- Switch setting matches LAN card (Autoneg on on both sides)

- NIS is configured. Uses DNS on NIS server end if host isn't in NIS.

-No static route or gated configured in netconf.

-Mystical hosts in netstat is possibly HP-OpenView server elsewhere in organisation. Belive this does timely pollings of system status, and that explains the on and off presense of this hosts in netstat.

More hints are welcome.

Andreas

[Darrell Allen]

Just for clarification, when you said the host (presuming the Openview server) shows up and it shouldn't, is that with netstat -r? If it is simply with netstat or netstat -a then that's different. It is normal for netstat or netstat -a to show hosts that have opened a socket to your server.

Darrell

[Anthony deRito]

It is normal for netstat -r to take several minutes at times depending on how much name resolution is required. Don't let this throw you off the track of finding the problem. I don't think it has anything to do with networking. You may want to attempt working at the console when the problem occurs and see how responsive the system is. Try to do on the console what your users do from a virtual terminal if you can. Remove all connections to your LAN if possible while doing this and try to get the system as close to standalone as possible. I would also have HP (or yourself if you have the diagnostic passwords) to go into the SP and run some low level diagnostics on your hardware especially on your I/O cards. It may take running diagnostics on several components and trying to duplicate the problem.

If this all works well then you can start narrowing down the problem to specific applications by bring the system up a level at a time and at the same time trying to shutdown/restart applications one at a time. This does not sound easy to solve but it would be easier to approach the problem systematically.

Good luck.

Tony

[Andreas D. Skjervold]

Hi again

Darren; the OpenView hosts shows up with netstat -r.

Anthony; I've been trhu the steps of diagnostic with HP tech, and found errors, but they are now resolved.
Seems like I have to do it by getting the server nearly standalone, and painstakinly starting different applications. The problem is that the freeze be abcent for long times before suddenly showing up several times.

Are at the moment working on the console to check if it freezes also.

Will be back with info in a week, since I will be away.

Andreas