topic Re: Process in RWAST, how to handle it? in Operating System - OpenVMS

Process in RWAST, how to handle it?

smsc_1 — Thu, 21 Jun 2012 06:32:09 GMT

Ok, first to all, I know kill a RWAST process is not the way.

But I'm gonna crazy with that process that block also the Oracle DataBase on OpenVMS IA64 Version V8.3-1H1

SQL> attach 'f DB_LOG';
%SQL-F-ERRATTDEC, Error attaching to database DB_LOG
-RDB-F-SYS_REQUEST, error from system services request
-RDMS-F-DBSHUTDOWN, database shutdown is in progress

Well, please, could someone help me to handle the RWAST process? I cannot reboot the machine since it's in production.


SDA> SHOW SUMMARY
.....
 206000B4 0034 VIQ_NTP_V404    SYSTEM       HIB      15 890CD800 B3B4A000    694
 206461B7 0037 LOG_CONV848     USER         RWAST     5 898F6F80 B3B82000   6889
.....



SDA> SHOW PROCESS/INDEX=0037

Process index: 0037   Name: LOG_CONV848       Extended PID: 206461B7
--------------------------------------------------------------------
Process status:          20044023  RES,DELPEN,BATCH,PHDRES,ERDACT
        status2:         00000001  QUANTUM_RESCHED

PCB address              898F6F80    JIB address              89DBC400
PHD address              B3B82000    Swapfile disk address    00000000
KTB vector address       898F72A8    HWPCB address   FFFFFFFF.B3B82080
Callback vector address  00000000    Termination mailbox          0000
Master internal PID      08C30037    Subprocess count                0
Creator extended PID     00000000    Creator internal PID     00000000
Previous CPU Id          00000003    Current CPU Id           00000003
Previous ASNSEQ  0000000000000018    Previous ASN     00000000005CFD37
Initial process priority        3    # open files remaining        290/300
Delete pending count            1    Direct I/O count/limit        149/150
UIC                [00100,000001]    Buffered I/O count/limit      148/150
Abs time of last event   0A1049D9    BUFIO byte count/limit     124608/125696
# of threads                    1    ASTs remaining               3994/4000
Swapped copy of LEFC0    00000000    Timer entries remaining      4000/4000
Swapped copy of LEFC1    00000000    Active page table count         0
Global cluster 2 pointer 00000000    Process WS page count        5715
Global cluster 3 pointer 00000000    Global WS page count         1174
PCB Specific Spinlock    89805980    Subprocesses in job             0

    Press RETURN for more.
SDA> 

Process index: 0037   Name: LOG_CONV848       Extended PID: 206461B7
--------------------------------------------------------------------

Thread index: 0000
------------------
Current capabilities:    System:     000C               QUORUM,RUN
                         User:       0000.00000000
Permanent capabilities:  System:     000C               QUORUM,RUN
                         User:       0000.00000000

Current affinities:                  00000000.00000000
Permanent affinities:                00000000.00000000

Thread status:                       20044023  RESPEN
       status2:                      00000001  

KTB address              898F6F80    HWPCB address   FFFFFFFF.B3B82080
PKTA address             7FFEFF98    Callback vector address  00000000
Internal PID             08C30037    Callback error           00000000
Extended PID             206461B7    Current CPU id           00000003
State                     RWAST      Flags                    00000000
Base priority                   3    Current priority                5
Waiting EF cluster              4    Event flag wait mask     00000001
CPU since last quantum   00000292    Mutex count                     0
ASTs active                  NONE

Many thanks!

Re: Process in RWAST, how to handle it?

smsc_1 — Thu, 21 Jun 2012 06:55:15 GMT

Adding more info:

SDA> SET PROCESS/INDEX=0037
%SDA-W-LOSTPROCESS, cleaning up: SDA's current process no longer exists; context now set to process running SDA
SDA> show call

Memory Stack Frame at 000007FD.BFFD4680 (User)
----------------------------------------------
Flags:  (none)
        Current PC   = 00000000.0009FE70        SDA$SHARE+9FE70
        Return PC    = FFFFFFFF.842094A0        LIBRTL+001DB4A0

        Current SP   = 00000000.7AC1AFF0
        Previous SP  = 00000000.7AC1BB20
        Current BSP  = 000007FD.BFFD4680
        Previous BSP = 000007FD.BFFD45B8


Valid GPRs for current routine: [() = NaT bit set]
--------------------------------------------------
R0           00000000.00000000         
R1           00000000.006B8000         
R4           00000000.00000000         
R5           00000000.00000000         
R6           00000000.00000001         
R7           00000000.008E5BB0         
R12          00000000.7AC1AFF0         
R13          00000000.7B5081C8         CMA$TIS_SHR+161C8
R32          00000000.008E5BB0         
R33          00000000.0045FC3A         SDA$SHARE+004A7C3A
R34          00000000.0045DA40         SDA$SHARE+004A5A40
R35          00000000.000091F8         CWLNM$K_UPDATE_SIZE+00010

    Press RETURN for more.
SDA> 

Memory Stack Frame at 000007FD.BFFD4680 (User)
----------------------------------------------
R36          00000000.00000001         
R37          FFFFFFFF.842094A0         LIBRTL+001DB4A0
R38          00000000.7AC9F730         
R39          00000000.7FF43EB0         
R40          00000000.0004F12C         SDA$SHARE+2D12C
R41          00000000.0003AFD0         SDA$SHARE+18FD0
R42          00000000.0003B910         SDA$SHARE+19910
R43          00000000.00044B18         SDA$SHARE+22B18
R44          00000000.0003B810         SDA$SHARE+19810
R45          00000000.0003B7F4         SDA$SHARE+197F4
R46          FFFFFFFF.800E73C0         SWIS$INTSTK_RESTORE+00180
R47          00000000.000000FF         
R48          00000000.0000000F         
R49          00000000.0000000F         
R50          FFFFFFFF.B3020CC8         SGN$GL_SYSTEM_CHECK
R51          00000000.00000000         
R52          00000000.00000010         
R53          00000000.00000000         
R54          00000000.0000000F         
R55          00001013.080A6010         
R56          FFFFFFFF.842094A0         LIBRTL+001DB4A0
R57          C0000000.00000C1D         
R58          40000000.00052D13         
R59          00000000.006B8000         

    Press RETURN for more.
SDA> 

Memory Stack Frame at 000007FD.BFFD4680 (User)
----------------------------------------------
Output GPRs for current routine: [() = NaT bit set]
---------------------------------------------------
R60          00000000.0003AFD0         SDA$SHARE+18FD0
R61          00000000.00000840         MCASAV$Q_F122
R62          00000000.0003AFE0         SDA$SHARE+18FE0
R63          00000000.0003AFE0         SDA$SHARE+18FE0
R64          02000000.00000840         

Valid FPRs for current routine:
-------------------------------
F2           00000.00000000.00000000
F3           00000.00000000.00000000
F4           00000.00000000.00000000
F5           00000.00000000.00000000
F16          00000.00000000.00000000
F17          00000.00000000.00000000
F18          00000.00000000.00000000
F19          00000.00000000.00000000
F20          00000.00000000.00000000
F21          00000.00000000.00000000
F22          00000.00000000.00000000
F23          00000.00000000.00000000
F24          00000.00000000.00000000
F25          00000.00000000.00000000

    Press RETURN for more.
SDA> 

Memory Stack Frame at 000007FD.BFFD4680 (User)
----------------------------------------------
F26          00000.00000000.00000000
F27          00000.00000000.00000000
F28          00000.00000000.00000000
F29          00000.00000000.00000000
F30          00000.00000000.00000000
F31          00000.00000000.00000000

Branch and Application registers for current routine:
-----------------------------------------------------
B0           00000000.001A1330         SDA$SHARE+001A1330
B1           00000000.00000000         
B2           00000000.00000000         
B3           00000000.00000000         
B4           00000000.00000000         
B5           00000000.00000000         
AR.FPSR      0009804C.0270033F
AR.LC        00000000.00000000
AR.EC        00000000.00000000
AR.BSP       000007FD.BFFD4680
AR.BSPSTORE  000007FD.BFFD4810
AR.RNAT      00000000.00000000
AR.UNAT      00000000.00000000
AR.RSC       00000000.0000000F
AR.PFS       C0000000.00000C1D

    Press RETURN for more.
SDA> 

Memory Stack Frame at 000007FD.BFFD4680 (User)
----------------------------------------------
PRs for current routine: [X = Not Preserved]
--------------------------------------------
0        8        16       24       32       40       48       56
11001010 00101011 10100000 00000000 00000000 00000000 00000000 00000010 








SDA> SET PROCESS/INDEX=0037
SDA> SHOW PROCESS/CHANNEL 

Process index: 0037   Name: LOG_CONV848       Extended PID: 206461B7
--------------------------------------------------------------------


                            Process active channels
                            -----------------------

Channel    CCB     Window     Status    Device/file accessed
-------    ---     ------     ------    --------------------
  0010  7FEDA000  00000000  Busy        SMWI71$DKA100:
  0020  7FEDA020  8995DB00              $1$DGA100:[SMSC.SCRIPTS]FRODO.COM;2
  0030  7FEDA040  890B8CC0              $1$DGA100:[VMS$COMMON.SYSEXE]SQL$72.EXE;2 (section file)
  0040  7FEDA060  88D99480              $1$DGA100:[VMS$COMMON.SYSLIB]PTHREAD$RTL.EXE;1 (section file)
  0050  7FEDA080  88DA44C0              $1$DGA100:[VMS$COMMON.SYSEXE]DCL.EXE;1 (section file)
  0060  7FEDA0A0  88D94E00              $1$DGA100:[VMS$COMMON.SYSLIB]DCLTABLES.EXE;264 (section file)
  0070  7FEDA0C0  89BE2840              $1$DGA100:[SMSC.LOG]LOG_CONVERT.LOG;2652
  0080  7FEDA0E0  8911A400              $1$DGA100:[SMSC.SCRIPTS]LOG_CONVERT.COM;6
  0090  7FEDA100  88D95100              $1$DGA100:[VMS$COMMON.SYSLIB]LIBOTS.EXE;1 (section file)
  00A0  7FEDA120  88D95080              $1$DGA100:[VMS$COMMON.SYSLIB]LIBRTL.EXE;1 (section file)
  00B0  7FEDA140  88D97900              $1$DGA100:[VMS$COMMON.SYSLIB]CMA$TIS_SHR.EXE;1 (section file)
  00C0  7FEDA160  88D97480              $1$DGA100:[VMS$COMMON.SYSLIB]SECURESHR.EXE;1 (section file)
  00D0  7FEDA180  88D93E00              $1$DGA100:[VMS$COMMON.SYSLIB]SECURESHRP.EXE;1 (section file)
  00E0  7FEDA1A0  88D96B80              $1$DGA100:[VMS$COMMON.SYSLIB]SMGSHR.EXE;1 (section file)
  00F0  7FEDA1C0  88D99080              $1$DGA100:[VMS$COMMON.SYSLIB]NCSSHR.EXE;1 (section file)
  0100  7FEDA1E0  88D96280              $1$DGA100:[VMS$COMMON.SYSLIB]LBRSHR.EXE;1 (section file)
  0110  7FEDA200  88D95800              $1$DGA100:[VMS$COMMON.SYSLIB]DCXSHR.EXE;1 (section file)

    Press RETURN for more.
SDA> 

Process index: 0037   Name: LOG_CONV848       Extended PID: 206461B7
--------------------------------------------------------------------

Channel    CCB     Window     Status    Device/file accessed
-------    ---     ------     ------    --------------------
  0120  7FEDA220  890B8940              $1$DGA100:[VMS$COMMON.SYSLIB]SQL$UTL_SHARE72.EXE;2 (section file)
  0130  7FEDA240  88D99800              $1$DGA100:[VMS$COMMON.SYSLIB]DECC$SHR.EXE;1 (section file)
  0140  7FEDA260  88D99180              $1$DGA100:[VMS$COMMON.SYSLIB]DPML$SHR.EXE;1 (section file)
  0150  7FEDA280  890B73C0              $1$DGA100:[VMS$COMMON.SYSLIB]RDBSHR.EXE;2 (section file)
  0160  7FEDA2A0  88D9B280              $1$DGA100:[VMS$COMMON.SYSLIB]CXXL$011_SHR.EXE;1 (section file)
  0170  7FEDA2C0  88D9A300              $1$DGA100:[VMS$COMMON.SYSLIB]CXXL$LANGRTL.EXE;1 (section file)
  0180  7FEDA2E0  890B7FC0              $1$DGA100:[VMS$COMMON.SYSLIB]RDB$SHARE72.EXE;2 (section file)
  0190  7FEDA300  890B7F40              $1$DGA100:[VMS$COMMON.SYSLIB]RDB$COSIP.EXE;2 (section file)
  01A0  7FEDA320  890B84C0              $1$DGA100:[VMS$COMMON.SYSMSG]SQL$MSG72.EXE;2 (section file)
  01B0  7FEDA340  890B7040              $1$DGA100:[VMS$COMMON.SYSMSG]RDBMSGS.EXE;2 (section file)
  01C0  7FEDA360  890B5300              $1$DGA100:[VMS$COMMON.SYSMSG]RDMSMSG72.EXE;2 (section file)
  01D0  7FEDA380  890B6E40              $1$DGA100:[VMS$COMMON.SYSMSG]COSI$MSG.EXE;2 (section file)
  01E0  7FEDA3A0  88DA97C0              $1$DGA100:[VMS$COMMON.SYSMSG]SHRIMGMSG.EXE;1 (section file)
  01F0  7FEDA3C0  88DA8240              $1$DGA100:[VMS$COMMON.SYSMSG]DECC$MSG.EXE;1 (section file)
  0200  7FEDA3E0  88DA89C0              $1$DGA100:[VMS$COMMON.SYSMSG]CXXL$MSG_SHR.EXE;1 (section file)
  0210  7FEDA400  00000000              $1$DGA100:
  0220  7FEDA420  00000000  Busy        $1$DGA100:
  0230  7FEDA440  890B3880              $1$DGA100:[VMS$COMMON.SYSLIB]RDMSHR72.EXE;2 (section file)
  0240  7FEDA460  88D94880              $1$DGA100:[VMS$COMMON.SYSLIB]EPC$SHR.EXE;1 (section file)
  0250  7FEDA480  890B3980              $1$DGA100:[VMS$COMMON.SYSLIB]RDMSHRP72.EXE;2 (section file)
  0260  7FEDA4A0  890B3200              $1$DGA100:[VMS$COMMON.SYSLIB]RDMPRV72.EXE;2 (section file)

    Press RETURN for more.
SDA> 

Process index: 0037   Name: LOG_CONV848       Extended PID: 206461B7
--------------------------------------------------------------------

Channel    CCB     Window     Status    Device/file accessed
-------    ---     ------     ------    --------------------
  0270  7FEDA4C0  891A2C80              $1$DGA100:[VMS$COMMON.SYSMSG]EPC$MSG.EXE;1
  0280  7FEDA4E0  899E1980              $1$DGA100:[VMS$COMMON.SYSEXE]RIGHTSLIST.DAT;1
  0290  7FEDA500  8975FD80              $1$DGA400:[SMSC_DB.LOG]SMSC_LOG_RDB.RDB;1
  02A0  7FEDA520  00000000              MBA3:
  02B0  7FEDA540  89217DC0              $1$DGA400:[SMSC_DB.LOG]MAIN_AREA.RDA;1
  02C0  7FEDA560  89D99040              $1$DGA400:[SMSC_DB.LOG]MAIN_AREA.SNP;1
  02D0  7FEDA580  8999F0C0              $1$DGA100:[SMSC.TMP]frodo.txt;3
  02E0  7FEDA5A0  00000000  Busy        SMWI71$DKA100:

  Total number of open channels : 46.

Re: Process in RWAST, how to handle it?

Rudy_De_Keyser — Thu, 21 Jun 2012 08:33:03 GMT

Hello,

You might also use

$ rmu/show lock/mode=blocking

to see if any other process is blocking yours.

Rudy

Re: Process in RWAST, how to handle it?

smsc_1 — Thu, 21 Jun 2012 09:56:16 GMT

Hello Rudy, this is the result:

rmu/show lock/mode=blocking
%RMU-I-NOLOCKSOUT, no locks on this node with the specified qualifiers

Instead, if I use:

RMU/SHOW LOCKS/PROCESS=206461B7

I see this is only a little output part):

Resource: page 24276

          ProcessID Process Name        Lock ID   System ID Requested Granted  
          --------- ---------------     --------- --------- --------- -------
Owner:    206461B7  LOG_CONV848....     1C03A3BD  00010003            EX

--------------------------------------------------------------------------------
Resource: TSN block 6

          ProcessID Process Name        Lock ID   System ID Requested Granted  
          --------- ---------------     --------- --------- --------- -------
Owner:    206461B7  LOG_CONV848....     0103A695  00010003            CR

--------------------------------------------------------------------------------
Resource: page 22671

          ProcessID Process Name        Lock ID   System ID Requested Granted  
          --------- ---------------     --------- --------- --------- -------
Owner:    206461B7  LOG_CONV848....     2103A798  00010003            EX

--------------------------------------------------------------------------------
Resource: AIJ journal info 0

          ProcessID Process Name        Lock ID   System ID Requested Granted  
          --------- ---------------     --------- --------- --------- -------
Owner:    206461B7  LOG_CONV848....     3C03AA13  00010003            CR

--------------------------------------------------------------------------------
Resource: page 24141

          ProcessID Process Name        Lock ID   System ID Requested Granted  
          --------- ---------------     --------- --------- --------- -------
Owner:    206461B7  LOG_CONV848....     0B03ABA7  00010003            EX

--------------------------------------------------------------------------------
Resource: page 24171

          ProcessID Process Name        Lock ID   System ID Requested Granted  
          --------- ---------------     --------- --------- --------- -------
Owner:    206461B7  LOG_CONV848....     5503AC4E  00010003            EX

--------------------------------------------------------------------------------
Resource: page 22266

          ProcessID Process Name        Lock ID   System ID Requested Granted  
          --------- ---------------     --------- --------- --------- -------
Owner:    206461B7  LOG_CONV848....     4703ADCC  00010003            EX

Re: Process in RWAST, how to handle it?

GuentherF — Thu, 21 Jun 2012 21:05:13 GMT

RWAST means VMS forced the process into this wait state because it has outstanding resources (like non-paged pool memory) which first needs to be returned via a kernel mode AST.

I see that this process has a bunch of busy channels to a disk drive. If an I/O to the disk drive has not returned yet the process remains stuck in RWAST.

You have to fix that drive or, more likely the connection (FC?) to the drive . If that cannot be fixed you do have to reboot.

/Guenther

Re: Process in RWAST, how to handle it?

smsc_1 — Tue, 26 Jun 2012 21:38:35 GMT

I found a very interesting article about RWAST:

http://www3.sympatico.ca/n.rieck/docs/openvms_notes_RWAST.htm

Plus I finally found the root cause of mine RWAST... There was an insternal BLade disk broken and the scripts hangs on I/O operation from this disk!!!

Re: Process in RWAST, how to handle it?

GuentherF — Tue, 26 Jun 2012 22:10:05 GMT

The article has good information. However, the procedure to get rid of a user process in RWAST may cause a later system crash. The reason is that with outstanding I/Os there maybe internal OS threads (called fork process) which would become active if for some reason that device becomes "operable" again. In that case a fork process might try to attach to the user process using stale data in kernel mode...kaboom.

I rather prefer a scheduled reboot.

/Guenther

Re: Process in RWAST, how to handle it?

smsc_1 — Wed, 27 Jun 2012 05:58:19 GMT

You're absolutely right, Guenther,

the reboot will solve the issue in a "clean way", infact I already did it, but after some hours I have again some process/script in RWAST. So, first to reboot, I think the right way is found the root cause. ;)