topic Re: login process stalls in Operating System - OpenVMS

login process stalls

Bjay — Mon, 09 Feb 2009 14:58:17 GMT

we have one VAX OpenVMS V6.2-1H3 it was working till Friday today when I tried to log in to the system for a daily checkup my log-in process do not proceeds,it do tells me about my last successful log -in information here is the message

Username: trimuse
Password:
Welcome to OpenVMS Alpha Operating System, Version V6.2-1H3 on node vaxapp
Last interactive login on Friday, 6-FEB-2009 14:46:50.95

it is waiting here forever,then I checked with different users, everyone is facing same issue even the system user could not check in.

I checked with my colleges what relevant changes were made, one of my co-worker enabled the auditing for everything
$ set audit /enable=all
we were suspecting that can be the reason as it was working fine before enabling audit for everything.
we have a way to execute commands from one of the other node which communicate with "vaxapp" via decnet, so we stop the audit server
$ set audit /server=stop
then tried log in, this did not help
so we revert and started the audit server again $ set audit /server=start.

we are still in the same hanger
___
|
0
/|\
|
/ \

thanks
Bjay

Re: login process stalls

Robert Gezelter — Mon, 09 Feb 2009 15:02:48 GMT

Bjay,

SET AUDIT/ENABLE=ALL can produce a very large amount of data.

My first guess is that the system disk is full. If you have some access to the machine, then freeing some disk space may very well resolve the problem.

Of course, deleting the voluminous audit information is a possibility, but I would recommend moving it to another disk rather than deleting it. It can be deleted later if it is truly not needed.

- Bob Gezelter, http://www.rlgsc.com

Re: login process stalls

Hein van den Heuvel — Mon, 09 Feb 2009 15:04:51 GMT

What is the STATE of the waiting processes, as seen from a logged in process? SUSPEND?

The AUDIT_SERVER is infamous for suspending processes. For example, check out $HELP SET AUDIT/BACKLOG

Are you per chance running out of disk space for the volume with the audit log (sys$sysdevice?)

This me a good time to check with the HP Availability Manager, if you have the collector going for this older OpenVMS version.

hth,
Hein.

Re: login process stalls

Ian Miller. — Mon, 09 Feb 2009 15:05:11 GMT

If you login /NOCOMMAND then does that help?

Is there some processing in SYLOGIN that is causing a problem?
Perhaps you can edit the SYLOGIN to simplify it and see if this helps?

Check the status of the disks especially the system disk and the disk on which the audit log is place (if different).

Re: login process stalls

Bjay — Mon, 09 Feb 2009 15:12:09 GMT

we do have enough space
Device Device Error Volume Free Trans Mnt
Name Status Count Label Blocks Count Cnt
VAXAPP$DKA500: Online wrtlck 0
VAXAPP$DKB0: Mounted 0 appsys 2557413 961 1
VAXAPP$DKB100: Online 0
VAXAPP$DKB200: Mounted 0 dsk200 1777662 1 1
VAXAPP$DRA0: Mounted 0 dskrd1 4123647 114 1
VAXAPP$DRB0: Mounted 0 dskrd20 25053405 1 1
VAXAPP$DVA0: Online 0

or am I interpreting wrong
DKB0 is our system disk

Re: login process stalls

Bjay — Mon, 09 Feb 2009 15:19:46 GMT

Username: trimuse /NOCOMMAND
Password:
Welcome to OpenVMS Alpha Operating System, Version V6.2-1H3 on node VAXAPP
Last interactive login on Friday, 6-FEB-2009 14:46:50.95

============================================
$ sho audit /all
List of audit journals:
Journal name: SECURITY
Journal owner: (system audit journal)
Destination: SYS$COMMON:[SYSMGR]SECURITY.AUDIT$JOURNAL
Monitoring: enabled
Warning thresholds, Block count: 100 Duration: 2 00:00:00.0
Action thresholds, Block count: 25 Duration: 0 00:30:00.0

Security auditing server characteristics:
Database version: 4.4
Backlog (total): 100, 200, 300
Backlog (process): 5, 2
Server processing intervals:
Archive flush: 0 00:01:00.00
Journal flush: 0 00:05:00.00
Resource scan: 0 00:05:00.00
Final resource action: purge oldest audit events

Security archiving information:
Archiving events: none
Archive destination:

System security alarms currently enabled for:
Authorization
Breakin: dialup,local,remote,network,detached
Logfailure: dialup,network,detached

System security audits currently disabled

%SHOW-W-NOAUDITING, security auditing disabled; no events will be logged

does that help

Re: login process stalls

Wim Van den Wyngaert — Mon, 09 Feb 2009 15:37:25 GMT

Do you have something in the sylogin.com file that is doing logging or lookup or so ?

If the file it uses is locked it may wait for ever.

Is batch still working ?

Wim

Re: login process stalls

Hein van den Heuvel — Mon, 09 Feb 2009 15:50:03 GMT

At this point I suspect a bad lock on RIGHTSLIST or such common file.

Is this a cluster?

>> Username: trimuse /NOCOMMAND
:
>> Last interactive login on Friday, 6-FEB-2009 14:46:50.95
:
>> does that help

Not much... It looks like it still hangs, but that was not explicit in the reply.

The last_login time (SYSUAF.DAT) has not been updated.
This suggest to me that the login has gotten 'nowhere'.

The /NOCOM is a handy tool, when a user LOGIN.COM is mucked up, but you indicaed multiple users had the problem.
/NOCOM does not bypass SYLOGIN.COM, but I suspect you do not get that for.

You may want to edit SYLOGIN.COM to leave some breadcrumbs while debugging this, may write a record to a (shared?) file indicating 'entered SYLOGIN for as '
and 'exit ... '. If it gets there at all!

So back to my first question.
While it is 'waiting forever', can you do a SHOW SYSTEM and identify the process. What is the state?

Can you use ANALYZE/SYSTEN and go to the waiting process, and do a SHWO PROC/CHAN looking for 'busy' and/or SHOW PROC/LOCK looking for 'waiting'.

Did you have availability manager ready to go?

finally... as much as it pains me to say this... the quickest way to get the users going again is probably to reboot.

Cheers,
Hein.

Re: login process stalls

Bjay — Mon, 09 Feb 2009 16:00:37 GMT

plz check the attachment

Re: login process stalls

Bjay — Mon, 09 Feb 2009 16:02:51 GMT

I can only execute command but do not get shell so could not do $ ana /sys

Re: login process stalls

Robert Gezelter — Mon, 09 Feb 2009 16:37:16 GMT

Bjay,

Can you do a SHOW MEMORY command?

Extremely slow login is also a symptom of a badly over-allocated page file.

In that case, the system appears to be doing nothing. However, that perception is incorrect. It is doing something, albeit VERY VERY slowly.

- Bob Gezelter, http://www.rlgsc.com

Re: login process stalls

Craig A — Mon, 09 Feb 2009 16:46:53 GMT

I'm not sure if you have access to the $ prompt. (If not, then you need to login to an account that has)

Please execute the following command and post the output.

$ SHOW DEVICE SYS$SYSDEVICE: /FULL
$ SHOW SYS/CLUSTER
$ MCR AUTHORIZE SHOW TRIMUSE
$ DIR/FULL SYSUAF

Re: login process stalls

Bjay — Mon, 09 Feb 2009 16:52:00 GMT

$ show memory
System Memory Resources on 9-FEB-2009 11:48:26.40

Physical Memory Usage (pages): Total Free In Use Modified
Main Memory (640.00Mb) 81920 64063 16920 937

Virtual I/O Cache (Kbytes): Total Free In Use
Cache Memory 3200 104 3096

Granularity Hint Regions (pages): Total Free In Use Released
Execlet code region 512 0 415 97
Execlet data region 88 2 86 0
VMS exec data region 968 0 968 0
Resident image code region 1024 0 818 206

Slot Usage (slots): Total Free Resident Swapped
Process Entry Slots 265 140 125 0
Balance Set Slots 263 140 123 0

Dynamic Memory Usage (bytes): Total Free In Use Largest
Nonpaged Dynamic Memory 5226496 2856064 2370432 2364992
Paged Dynamic Memory 2834432 1464976 1369456 1443680

Paging File Usage (blocks): Free Reservable Total
DISK$AXPVMSSYS:[SYS0.SYSEXE]SWAPFILE.SYS 38272 38272 38272
DISK$AXPVMSSYS:[SYS0.SYSEXE]PAGEFILE.SYS 1312800 989520 1318784

Of the physical pages in use, 3655 pages are permanently allocated to OpenVMS.

===============
Wim there are 4 to 5 batch jobs running

Bjay

Re: login process stalls

Bjay — Mon, 09 Feb 2009 17:00:02 GMT

sho mem output in attachment

Re: login process stalls

Jess Goodman — Mon, 09 Feb 2009 17:52:05 GMT

After logging in by using USERNAME/NOCOMMAND then do
$ SET VERIFY
$ @LOGIN !or whatever login command file is pointed to by SYSUAF

Then you should see what command is hanging.

Re: login process stalls

John Gillings — Mon, 09 Feb 2009 20:50:32 GMT

Bjay
Do the processes eventually get in if you wait long enough? (could be a very long time). Are there CPU intensive processes running?

Could you please post the output of:

$ SHOW SYSTEM
$ MCR SYSGEN SHOW DORMANTWAIT
$ MCR SYSGEN SHOW PIXSCAN

Re: login process stalls

Bjay — Tue, 10 Feb 2009 02:18:53 GMT

plz find req info in attachment

Re: login process stalls

Robert Gezelter — Tue, 10 Feb 2009 02:44:14 GMT

Bjay,

According to the attachment to your most recent posting, several processes are in RWPAG.

Is it known what those processes are trying to do?

Working without details, it is difficult to assess, but RWPAG implies that their may be a pool fragmentation issue.

Has this problem ever happened before? If so, when did it happen? The system has been running for 56 days.

I am loath to suggest increasing the sizes of the non-paged (NPAGEDYN) and paged (PAGEDYN) regions without specific evidence, but dynamic storage fragmentation (also referred to as "pool fragmentation") would appear to be a possibility.

- Bob Gezelter, http://www.rlgsc.com

Re: login process stalls

Hakan Zanderau ( Anders — Tue, 10 Feb 2009 06:18:32 GMT

Before doing anything else, I would have investigated the batchjob with processid 00000278. It has used over 1 hour of CPU-time. Is it the known behavior for this batchjob ?

Hakan

Re: login process stalls

Wim Van den Wyngaert — Tue, 10 Feb 2009 07:08:43 GMT

Please post sh mem/pool/fu. I suspect that the largest block is too small due to fragmentation. I normally reboot the system if possible. Or restart some memory eationg stuff. Or incrage the NPAGDYN / PAGDYN).

WIm