Operating System - OpenVMS
cancel
Showing results for 
Search instead for 
Did you mean: 

Process in RWAST, how to handle it?

 
smsc_1
Regular Advisor

Process in RWAST, how to handle it?

 

Ok, first to all, I know kill a RWAST process is not the way.

But I'm gonna crazy with that process that block also the Oracle DataBase on OpenVMS IA64 Version V8.3-1H1

 

SQL> attach 'f DB_LOG';
%SQL-F-ERRATTDEC, Error attaching to database DB_LOG
-RDB-F-SYS_REQUEST, error from system services request
-RDMS-F-DBSHUTDOWN, database shutdown is in progress

 

 

Well, please, could someone help me to handle the RWAST process? I cannot reboot the machine since it's in production.

 


SDA> SHOW SUMMARY
.....
 206000B4 0034 VIQ_NTP_V404    SYSTEM       HIB      15 890CD800 B3B4A000    694
 206461B7 0037 LOG_CONV848     USER         RWAST     5 898F6F80 B3B82000   6889
.....



SDA> SHOW PROCESS/INDEX=0037 Process index: 0037 Name: LOG_CONV848 Extended PID: 206461B7 -------------------------------------------------------------------- Process status: 20044023 RES,DELPEN,BATCH,PHDRES,ERDACT status2: 00000001 QUANTUM_RESCHED PCB address 898F6F80 JIB address 89DBC400 PHD address B3B82000 Swapfile disk address 00000000 KTB vector address 898F72A8 HWPCB address FFFFFFFF.B3B82080 Callback vector address 00000000 Termination mailbox 0000 Master internal PID 08C30037 Subprocess count 0 Creator extended PID 00000000 Creator internal PID 00000000 Previous CPU Id 00000003 Current CPU Id 00000003 Previous ASNSEQ 0000000000000018 Previous ASN 00000000005CFD37 Initial process priority 3 # open files remaining 290/300 Delete pending count 1 Direct I/O count/limit 149/150 UIC [00100,000001] Buffered I/O count/limit 148/150 Abs time of last event 0A1049D9 BUFIO byte count/limit 124608/125696 # of threads 1 ASTs remaining 3994/4000 Swapped copy of LEFC0 00000000 Timer entries remaining 4000/4000 Swapped copy of LEFC1 00000000 Active page table count 0 Global cluster 2 pointer 00000000 Process WS page count 5715 Global cluster 3 pointer 00000000 Global WS page count 1174 PCB Specific Spinlock 89805980 Subprocesses in job 0 Press RETURN for more. SDA> Process index: 0037 Name: LOG_CONV848 Extended PID: 206461B7 -------------------------------------------------------------------- Thread index: 0000 ------------------ Current capabilities: System: 000C QUORUM,RUN User: 0000.00000000 Permanent capabilities: System: 000C QUORUM,RUN User: 0000.00000000 Current affinities: 00000000.00000000 Permanent affinities: 00000000.00000000 Thread status: 20044023 RESPEN status2: 00000001 KTB address 898F6F80 HWPCB address FFFFFFFF.B3B82080 PKTA address 7FFEFF98 Callback vector address 00000000 Internal PID 08C30037 Callback error 00000000 Extended PID 206461B7 Current CPU id 00000003 State RWAST Flags 00000000 Base priority 3 Current priority 5 Waiting EF cluster 4 Event flag wait mask 00000001 CPU since last quantum 00000292 Mutex count 0 ASTs active NONE

 

 

Many thanks!

 

./ Lucas
7 REPLIES
smsc_1
Regular Advisor

Re: Process in RWAST, how to handle it?

 

Adding more info:

 

SDA> SET PROCESS/INDEX=0037
%SDA-W-LOSTPROCESS, cleaning up: SDA's current process no longer exists; context now set to process running SDA
SDA> show call

Memory Stack Frame at 000007FD.BFFD4680 (User)
----------------------------------------------
Flags:  (none)
        Current PC   = 00000000.0009FE70        SDA$SHARE+9FE70
        Return PC    = FFFFFFFF.842094A0        LIBRTL+001DB4A0

        Current SP   = 00000000.7AC1AFF0
        Previous SP  = 00000000.7AC1BB20
        Current BSP  = 000007FD.BFFD4680
        Previous BSP = 000007FD.BFFD45B8


Valid GPRs for current routine: [() = NaT bit set]
--------------------------------------------------
R0           00000000.00000000         
R1           00000000.006B8000         
R4           00000000.00000000         
R5           00000000.00000000         
R6           00000000.00000001         
R7           00000000.008E5BB0         
R12          00000000.7AC1AFF0         
R13          00000000.7B5081C8         CMA$TIS_SHR+161C8
R32          00000000.008E5BB0         
R33          00000000.0045FC3A         SDA$SHARE+004A7C3A
R34          00000000.0045DA40         SDA$SHARE+004A5A40
R35          00000000.000091F8         CWLNM$K_UPDATE_SIZE+00010

    Press RETURN for more.
SDA> 

Memory Stack Frame at 000007FD.BFFD4680 (User)
----------------------------------------------
R36          00000000.00000001         
R37          FFFFFFFF.842094A0         LIBRTL+001DB4A0
R38          00000000.7AC9F730         
R39          00000000.7FF43EB0         
R40          00000000.0004F12C         SDA$SHARE+2D12C
R41          00000000.0003AFD0         SDA$SHARE+18FD0
R42          00000000.0003B910         SDA$SHARE+19910
R43          00000000.00044B18         SDA$SHARE+22B18
R44          00000000.0003B810         SDA$SHARE+19810
R45          00000000.0003B7F4         SDA$SHARE+197F4
R46          FFFFFFFF.800E73C0         SWIS$INTSTK_RESTORE+00180
R47          00000000.000000FF         
R48          00000000.0000000F         
R49          00000000.0000000F         
R50          FFFFFFFF.B3020CC8         SGN$GL_SYSTEM_CHECK
R51          00000000.00000000         
R52          00000000.00000010         
R53          00000000.00000000         
R54          00000000.0000000F         
R55          00001013.080A6010         
R56          FFFFFFFF.842094A0         LIBRTL+001DB4A0
R57          C0000000.00000C1D         
R58          40000000.00052D13         
R59          00000000.006B8000         

    Press RETURN for more.
SDA> 

Memory Stack Frame at 000007FD.BFFD4680 (User)
----------------------------------------------
Output GPRs for current routine: [() = NaT bit set]
---------------------------------------------------
R60          00000000.0003AFD0         SDA$SHARE+18FD0
R61          00000000.00000840         MCASAV$Q_F122
R62          00000000.0003AFE0         SDA$SHARE+18FE0
R63          00000000.0003AFE0         SDA$SHARE+18FE0
R64          02000000.00000840         

Valid FPRs for current routine:
-------------------------------
F2           00000.00000000.00000000
F3           00000.00000000.00000000
F4           00000.00000000.00000000
F5           00000.00000000.00000000
F16          00000.00000000.00000000
F17          00000.00000000.00000000
F18          00000.00000000.00000000
F19          00000.00000000.00000000
F20          00000.00000000.00000000
F21          00000.00000000.00000000
F22          00000.00000000.00000000
F23          00000.00000000.00000000
F24          00000.00000000.00000000
F25          00000.00000000.00000000

    Press RETURN for more.
SDA> 

Memory Stack Frame at 000007FD.BFFD4680 (User)
----------------------------------------------
F26          00000.00000000.00000000
F27          00000.00000000.00000000
F28          00000.00000000.00000000
F29          00000.00000000.00000000
F30          00000.00000000.00000000
F31          00000.00000000.00000000

Branch and Application registers for current routine:
-----------------------------------------------------
B0           00000000.001A1330         SDA$SHARE+001A1330
B1           00000000.00000000         
B2           00000000.00000000         
B3           00000000.00000000         
B4           00000000.00000000         
B5           00000000.00000000         
AR.FPSR      0009804C.0270033F
AR.LC        00000000.00000000
AR.EC        00000000.00000000
AR.BSP       000007FD.BFFD4680
AR.BSPSTORE  000007FD.BFFD4810
AR.RNAT      00000000.00000000
AR.UNAT      00000000.00000000
AR.RSC       00000000.0000000F
AR.PFS       C0000000.00000C1D

    Press RETURN for more.
SDA> 

Memory Stack Frame at 000007FD.BFFD4680 (User)
----------------------------------------------
PRs for current routine: [X = Not Preserved]
--------------------------------------------
0        8        16       24       32       40       48       56
11001010 00101011 10100000 00000000 00000000 00000000 00000000 00000010 








SDA> SET PROCESS/INDEX=0037
SDA> SHOW PROCESS/CHANNEL

Process index: 0037   Name: LOG_CONV848       Extended PID: 206461B7
--------------------------------------------------------------------


                            Process active channels
                            -----------------------

Channel    CCB     Window     Status    Device/file accessed
-------    ---     ------     ------    --------------------
  0010  7FEDA000  00000000  Busy        SMWI71$DKA100:
  0020  7FEDA020  8995DB00              $1$DGA100:[SMSC.SCRIPTS]FRODO.COM;2
  0030  7FEDA040  890B8CC0              $1$DGA100:[VMS$COMMON.SYSEXE]SQL$72.EXE;2 (section file)
  0040  7FEDA060  88D99480              $1$DGA100:[VMS$COMMON.SYSLIB]PTHREAD$RTL.EXE;1 (section file)
  0050  7FEDA080  88DA44C0              $1$DGA100:[VMS$COMMON.SYSEXE]DCL.EXE;1 (section file)
  0060  7FEDA0A0  88D94E00              $1$DGA100:[VMS$COMMON.SYSLIB]DCLTABLES.EXE;264 (section file)
  0070  7FEDA0C0  89BE2840              $1$DGA100:[SMSC.LOG]LOG_CONVERT.LOG;2652
  0080  7FEDA0E0  8911A400              $1$DGA100:[SMSC.SCRIPTS]LOG_CONVERT.COM;6
  0090  7FEDA100  88D95100              $1$DGA100:[VMS$COMMON.SYSLIB]LIBOTS.EXE;1 (section file)
  00A0  7FEDA120  88D95080              $1$DGA100:[VMS$COMMON.SYSLIB]LIBRTL.EXE;1 (section file)
  00B0  7FEDA140  88D97900              $1$DGA100:[VMS$COMMON.SYSLIB]CMA$TIS_SHR.EXE;1 (section file)
  00C0  7FEDA160  88D97480              $1$DGA100:[VMS$COMMON.SYSLIB]SECURESHR.EXE;1 (section file)
  00D0  7FEDA180  88D93E00              $1$DGA100:[VMS$COMMON.SYSLIB]SECURESHRP.EXE;1 (section file)
  00E0  7FEDA1A0  88D96B80              $1$DGA100:[VMS$COMMON.SYSLIB]SMGSHR.EXE;1 (section file)
  00F0  7FEDA1C0  88D99080              $1$DGA100:[VMS$COMMON.SYSLIB]NCSSHR.EXE;1 (section file)
  0100  7FEDA1E0  88D96280              $1$DGA100:[VMS$COMMON.SYSLIB]LBRSHR.EXE;1 (section file)
  0110  7FEDA200  88D95800              $1$DGA100:[VMS$COMMON.SYSLIB]DCXSHR.EXE;1 (section file)

    Press RETURN for more.
SDA>

Process index: 0037   Name: LOG_CONV848       Extended PID: 206461B7
--------------------------------------------------------------------

Channel    CCB     Window     Status    Device/file accessed
-------    ---     ------     ------    --------------------
  0120  7FEDA220  890B8940              $1$DGA100:[VMS$COMMON.SYSLIB]SQL$UTL_SHARE72.EXE;2 (section file)
  0130  7FEDA240  88D99800              $1$DGA100:[VMS$COMMON.SYSLIB]DECC$SHR.EXE;1 (section file)
  0140  7FEDA260  88D99180              $1$DGA100:[VMS$COMMON.SYSLIB]DPML$SHR.EXE;1 (section file)
  0150  7FEDA280  890B73C0              $1$DGA100:[VMS$COMMON.SYSLIB]RDBSHR.EXE;2 (section file)
  0160  7FEDA2A0  88D9B280              $1$DGA100:[VMS$COMMON.SYSLIB]CXXL$011_SHR.EXE;1 (section file)
  0170  7FEDA2C0  88D9A300              $1$DGA100:[VMS$COMMON.SYSLIB]CXXL$LANGRTL.EXE;1 (section file)
  0180  7FEDA2E0  890B7FC0              $1$DGA100:[VMS$COMMON.SYSLIB]RDB$SHARE72.EXE;2 (section file)
  0190  7FEDA300  890B7F40              $1$DGA100:[VMS$COMMON.SYSLIB]RDB$COSIP.EXE;2 (section file)
  01A0  7FEDA320  890B84C0              $1$DGA100:[VMS$COMMON.SYSMSG]SQL$MSG72.EXE;2 (section file)
  01B0  7FEDA340  890B7040              $1$DGA100:[VMS$COMMON.SYSMSG]RDBMSGS.EXE;2 (section file)
  01C0  7FEDA360  890B5300              $1$DGA100:[VMS$COMMON.SYSMSG]RDMSMSG72.EXE;2 (section file)
  01D0  7FEDA380  890B6E40              $1$DGA100:[VMS$COMMON.SYSMSG]COSI$MSG.EXE;2 (section file)
  01E0  7FEDA3A0  88DA97C0              $1$DGA100:[VMS$COMMON.SYSMSG]SHRIMGMSG.EXE;1 (section file)
  01F0  7FEDA3C0  88DA8240              $1$DGA100:[VMS$COMMON.SYSMSG]DECC$MSG.EXE;1 (section file)
  0200  7FEDA3E0  88DA89C0              $1$DGA100:[VMS$COMMON.SYSMSG]CXXL$MSG_SHR.EXE;1 (section file)
  0210  7FEDA400  00000000              $1$DGA100:
  0220  7FEDA420  00000000  Busy        $1$DGA100:
  0230  7FEDA440  890B3880              $1$DGA100:[VMS$COMMON.SYSLIB]RDMSHR72.EXE;2 (section file)
  0240  7FEDA460  88D94880              $1$DGA100:[VMS$COMMON.SYSLIB]EPC$SHR.EXE;1 (section file)
  0250  7FEDA480  890B3980              $1$DGA100:[VMS$COMMON.SYSLIB]RDMSHRP72.EXE;2 (section file)
  0260  7FEDA4A0  890B3200              $1$DGA100:[VMS$COMMON.SYSLIB]RDMPRV72.EXE;2 (section file)

    Press RETURN for more.
SDA>

Process index: 0037   Name: LOG_CONV848       Extended PID: 206461B7
--------------------------------------------------------------------

Channel    CCB     Window     Status    Device/file accessed
-------    ---     ------     ------    --------------------
  0270  7FEDA4C0  891A2C80              $1$DGA100:[VMS$COMMON.SYSMSG]EPC$MSG.EXE;1
  0280  7FEDA4E0  899E1980              $1$DGA100:[VMS$COMMON.SYSEXE]RIGHTSLIST.DAT;1
  0290  7FEDA500  8975FD80              $1$DGA400:[SMSC_DB.LOG]SMSC_LOG_RDB.RDB;1
  02A0  7FEDA520  00000000              MBA3:
  02B0  7FEDA540  89217DC0              $1$DGA400:[SMSC_DB.LOG]MAIN_AREA.RDA;1
  02C0  7FEDA560  89D99040              $1$DGA400:[SMSC_DB.LOG]MAIN_AREA.SNP;1
  02D0  7FEDA580  8999F0C0              $1$DGA100:[SMSC.TMP]frodo.txt;3
  02E0  7FEDA5A0  00000000  Busy        SMWI71$DKA100:

  Total number of open channels : 46.

 

./ Lucas

Re: Process in RWAST, how to handle it?

Hello,

 

You might also use

 

$ rmu/show lock/mode=blocking

 

to see if any other process is blocking yours.

 

Rudy

 

 

smsc_1
Regular Advisor

Re: Process in RWAST, how to handle it?

 

Hello Rudy, this is the result:

 

rmu/show lock/mode=blocking
%RMU-I-NOLOCKSOUT, no locks on this node with the specified qualifiers

 

Instead, if I use:

 

RMU/SHOW LOCKS/PROCESS=206461B7

 

I see this is only a little output part):

 

Resource: page 24276

          ProcessID Process Name        Lock ID   System ID Requested Granted  
          --------- ---------------     --------- --------- --------- -------
Owner:    206461B7  LOG_CONV848....     1C03A3BD  00010003            EX

--------------------------------------------------------------------------------
Resource: TSN block 6

          ProcessID Process Name        Lock ID   System ID Requested Granted  
          --------- ---------------     --------- --------- --------- -------
Owner:    206461B7  LOG_CONV848....     0103A695  00010003            CR

--------------------------------------------------------------------------------
Resource: page 22671

          ProcessID Process Name        Lock ID   System ID Requested Granted  
          --------- ---------------     --------- --------- --------- -------
Owner:    206461B7  LOG_CONV848....     2103A798  00010003            EX

--------------------------------------------------------------------------------
Resource: AIJ journal info 0

          ProcessID Process Name        Lock ID   System ID Requested Granted  
          --------- ---------------     --------- --------- --------- -------
Owner:    206461B7  LOG_CONV848....     3C03AA13  00010003            CR

--------------------------------------------------------------------------------
Resource: page 24141

          ProcessID Process Name        Lock ID   System ID Requested Granted  
          --------- ---------------     --------- --------- --------- -------
Owner:    206461B7  LOG_CONV848....     0B03ABA7  00010003            EX

--------------------------------------------------------------------------------
Resource: page 24171

          ProcessID Process Name        Lock ID   System ID Requested Granted  
          --------- ---------------     --------- --------- --------- -------
Owner:    206461B7  LOG_CONV848....     5503AC4E  00010003            EX

--------------------------------------------------------------------------------
Resource: page 22266

          ProcessID Process Name        Lock ID   System ID Requested Granted  
          --------- ---------------     --------- --------- --------- -------
Owner:    206461B7  LOG_CONV848....     4703ADCC  00010003            EX

 

./ Lucas
GuentherF
Trusted Contributor

Re: Process in RWAST, how to handle it?

RWAST means VMS forced the process into this wait state because it has outstanding resources (like non-paged pool memory) which first needs to be returned via a kernel mode AST.

 

I see that this process has a bunch of busy channels to a disk drive. If an I/O to the disk drive has not returned yet  the process remains stuck in RWAST.

 

You have to fix that drive or, more likely the connection (FC?) to the drive . If that cannot be fixed you do have to reboot.

 

/Guenther

smsc_1
Regular Advisor

Re: Process in RWAST, how to handle it?

 

I found a very interesting article about RWAST:

 

http://www3.sympatico.ca/n.rieck/docs/openvms_notes_RWAST.htm

 

Plus I finally found the root cause of mine RWAST... There was an insternal BLade disk broken and the scripts hangs on I/O operation from this disk!!!

  

./ Lucas
GuentherF
Trusted Contributor

Re: Process in RWAST, how to handle it?

The article has good information. However, the procedure to get rid of a user process in RWAST may cause a later system crash. The reason is that with outstanding I/Os there maybe internal OS threads (called fork process) which would become active if for some reason that device becomes "operable" again. In that case a fork process might try to attach to the user process using stale data in kernel mode...kaboom.

 

I rather prefer a scheduled reboot.

 

/Guenther

smsc_1
Regular Advisor

Re: Process in RWAST, how to handle it?

 

You're absolutely right, Guenther,

the reboot will solve the issue in a "clean way", infact I already did it, but after some hours I have again some process/script in RWAST. So, first to reboot, I think the right way is found the root cause. ;)

  

./ Lucas