topic Re: Vax system freezing up in Operating System - OpenVMS

Vax system freezing up

vmsserbo — Thu, 08 Feb 2007 14:52:52 GMT

My offices have been experiencing a problem with our Vax system 4100 the last couple days. It will freeze up for about 5 minutes at a time. When it freezes up users cannot navigate from screen to screen, some users get kicked off, and some users’ screens just freeze. If people get kicked off and try to get back on the username prompt does not come up and the cursor just blinks on the screen. This happened once last Friday and it has happened a couple times this morning. This is happening to all of my offices across the state at the same time, so I know it’s not a network problem in one office. After about five minutes, it goes back to normal and everyone is able to login.

Any suggestions on what may be cause would be appreciated!

Re: Vax system freezing up

melvyn burnard — Thu, 08 Feb 2007 15:11:22 GMT

posted in completely the wrong forum.
moved to more appropriate forum

Re: Vax system freezing up

Steven Schweda — Thu, 08 Feb 2007 15:20:27 GMT

The first step might be asking in a
VMS-related forum, where more people are more
likely to know what a VAX is.

"SHOW ERRORS"?

Anything in "SYS$MANAGER:OPERATOR.LOG"?

There are many possibilities, including a
high-priority process eating the whole CPU,
a bad disk drive, bad network hardware, and
so on.

"navigate from screen to screen" in what?
"kicked off" how? By whom?

Is anyone connected to a local serial port,
or is everyone connected through the
network?

Re: Vax system freezing up

Jan van den Ende — Thu, 08 Feb 2007 15:31:57 GMT

Miles,

_WAY_ too little info to solve this, but (wild shot in the dark) can you do a
$ SHOW DEV/FUL for every disk on that system?
Look for the HIGHWATER setting. If they are all disabled, forget about my hunch.
If you find some (if only one) then do a
SET VOLUME /NOHIGH for each.
Doing some decent-sized allocations on /highwater drives CAN cause just these symptoms.
Then again, I might be completely wrong here. Not enough info....

Proost.

Have one on me.

jpe

Re: Vax system freezing up

vmsserbo — Thu, 08 Feb 2007 15:41:00 GMT

All disabled!

Re: Vax system freezing up

labadie_1 — Thu, 08 Feb 2007 16:31:14 GMT

Check the pagefile utilization and the non-paged pool.

Can you do
$ mc agen$feedback
(this just gives back the prompt, but creates a new file sys$system:agenfeedback.dat, with peak values)
and post the result of the following command
$ sea sys$system:agen$feedback.dat pagef,fail
$ sh mem/fil

Ideally, try to have a process (batch, detached, interactive...) doing something like

$ loop
$ sh sys/out=a.tmp
$ wait 00:01:00
$ goto loop

to see if you have processes in "interesting states" (rwxxx, pfg...)

Good hunt.

Re: Vax system freezing up

Martin Hughes — Thu, 08 Feb 2007 18:42:42 GMT

When you experience freezes, is it just with application users? or does it also affect your DCL login?.

Assuming that all remote connections freeze, I would be trying to connect to the local console to troubleshoot when the problem happens again. This will either give you a login that works to troubleshoot, or if it also freezes, at least this will eliminate some possible causes.

As Steven suggested, check your operator log, and also check your binary errorlog.

Re: Vax system freezing up

Robert Gezelter — Thu, 08 Feb 2007 19:36:28 GMT

Miles,

As others have pointed out, there are any number of hardware problems, resource shortages, and other behaviors that can give rise to what appears to be a "hang".

Whether the system is actually hung, or merely VERY busy is another possibility.

This problem is one of the class of problems where troubleshooting in the forum is very difficult.

For example, while a communications problem at one office is somewhat ruled out, a problem somewhere in the network backbone is still a distinct possibility.

Are there any messages coming out on the console when the system appears to be hung?

While it is self serving (my firm has done, and does research this type of problem for clients), if this is a recurring problem, retaining experienced assistance is likely the shortest path to a solution.

- Bob Gezelter, http://www.rlgsc.com

Re: Vax system freezing up

Hoff — Thu, 08 Feb 2007 21:41:22 GMT

Robert G. is correct; this is going to be difficult and problematic to diagnose in a tiny little window. The process is going to be interative.

Get a connection into the system console port on this VAX 4000 model 100 series box, via a redundant path; via remote (secured) modem dial-in or otherwise. The point here is to figure out if the system is hung, or the network is hung.

Software parameters are certainly a possibility, but I'd be aiming first at the hardware and the local network backbone. At the system error log, and at communications errors that can get logged.

Oh, and just because it hits all offices in the state doesn't mean that all the connections don't go through a bad router in, say, Hawaii -- network wires don't always run where you'd expect. (Ok, so I've been hoping for an on-site call for a bad router in Hawaii.) Even multiple vendors and supposedly redundant communications vendors can run all the connections through some rather odd paths, as the occasional backhoe operator has found.

But in any case, the connection into the console will give you some clue as to the particular source of the problem.

If the system is wedged at the console, then the next step can involve recording a series of PC addresses and then initiating a forced crash at the console, and the creation of a crashdump. From a crashdump, some idea of the trigger for the problem can become visible.

Disclosure: as is Robert G., HoffmanLabs is also in this troubleshooting business.

Stephen Hoffman
HoffmanLabs

Re: Vax system freezing up

Volker Halle — Fri, 09 Feb 2007 01:56:02 GMT

Miles,

as others have said, system hangs - especially when they are temporary - are extremely hard to diagnose. You need to collect as much information as possible and carefully observe the symptoms.

Try to connect a 'console manager' type application to the console port (OPA0:) - a laptop with a terminal emulator and an activated session log file will do. Make sure OPA0: is SET TERM/BROADCAST/PERM.

Consider to use AMDS (RMDRIVER) data collector on the OpenVMS VAX system and run Availabilty Manager on a separate node to capture any events automatically reported by AMDS.

Check errorlog and OPERATOR.LOG for any related entries at the internval of the system hang.

Try to think of a couple of specific tests to be executed when the system hangs next time, to rule out or confirm certain possible scenarios.

- ping the node
- have 'hung' users type CTRL-T (make sure it's enabled)
- try to issue commands on OPA0: or another local terminal

Volker.

Re: Vax system freezing up

Barry Alford — Fri, 09 Feb 2007 05:38:08 GMT

My Â£0.02:

There may be insufficient blocks on the system disk for the Audit Server to write to its log.

If the Audit Server can't log it, then it will not let anyone do it!

Log in on OPA0 and SHOW SYSTEM; if everything is in a SUS(pended) state except AUDIT$SERVER (?), and SHOW DEV D says there is little space on the disk, then this is the likely culprit.

Re: Vax system freezing up

John Travell — Fri, 09 Feb 2007 11:27:38 GMT

An exceedingly simple check to make is to discover whether the machine responds to Control_O. This is handled, and echoed by the driver, and unlike (almost?) all other commands or messages does NOT depend on user level code.
You SHOULD see successive *OUTPUT OFF* and *OUTPUT ON* messages. If you do NOT see them, the hang is either hardware or very high priority software. If you DO see them, while the machine is otherwise hung, it will rule out a number of potential problem areas. It will NOT, however, provide a complete diagnosis...
JT:

Re: Vax system freezing up

Andy Bustamante — Fri, 09 Feb 2007 12:54:11 GMT

As Jan points out anything here is going to be a menu could be, might be and check these.

First question, what version of VMS are you running? Next, is this a VAX or Alphaserver, the 4100 was a popular Alphaserver and many customers refer to any VMS system as a VAX. What else is running, are your backsups running into the work day now when they used to be done before 8:00 am?

Load Availablity Manager and the client monitor on another local system.

Try leaving a terminal logged on with "$ reply/enable" to check for possible messages. "$ type sys$manager:operator.log" may have captured pool capacity messages, although the absence won't rule out this as a potential issue.

Check the network, bandwidth or line errors. Is anyone moving large files over the network?

Andy