regarding system performance

[#1]

hi please find the attachment




regards
ajaydec

[Steven Schweda]

I found it. Now, can you attach a copy of
Microsoft Word for VMS Alpha so I can read
it? Or, you could just attach a plain-text
version, which everyone could read easily.

[#3]

Hi,

We have a process by name DECEDI$CSF_2 which has to do some kind of process but right now its not performing and it is in hanging postion as if it is waiting for some resource.
Below are the output of SDA

1) SDA> show process decedi$csf_2 /locks /brief

Process index: 0177 Name: DECEDI$CSF_2 Extended PID: 20835F77
--------------------------------------------------------------------
LKB Address Lockid ParentId PID BLKAST SubLocks RQ GR Queue RSB Address Resource Name Mode
----------------- -------- -------- -------- -------- -------- -- -- ------- ----------------- ------------------------------- ----
FFFFFFFE.FB5ACF80 370B301C 4C0AB147 01AF0177 01573550 0 CR Granted FFFFFFFE.EC5AE1C0 G..... Exec
FFFFFFFE.F33C6A80 7D08FE6A 4C0AB147 01AF0177 01573550 0 EX Granted FFFFFFFE.FD04A800 G..... Exec
FFFFFFFE.F86E7AC0 1A098B1D 4C0AB147 01AF0177 01573550 0 EX Granted FFFFFFFE.EC4BABC0 R..... Exec
FFFFFFFE.F73A86C0 67018746 4C0AB147 01AF0177 00000000 0 NL Granted FFFFFFFE.EC2BDE80 R..... Exec
FFFFFFFE.F6486BC0 660855B0 4C0AB147 01AF0177 00000000 0 PR Waiting FFFFFFFE.EC8E75C0 I..... Exec
FFFFFFFE.F0709980 130CB475 4C0AB147 01AF0177 00000000 0 PR Waiting FFFFFFFE.FB162F80 I..... Exec
FFFFFFFE.F795F340 200142BC 4C0AB147 01AF0177 00000000 0 NL Granted FFFFFFFE.FA164580 H... Exec
FFFFFFFE.EE835D40 4D0DF0B7 00000000 01AF0177 00000000 0 EX Granted FFFFFFFE.F9C3C1C0 BG2477 User
FFFFFFFE.FD4ACF80 50047E03 00000000 01AF0177 00000000 5 NL Granted FFFFFFFE.FB8D9340 RMS$.......SYS_DISK ... Exec
FFFFFFFE.EEFAB5C0 3E0C0965 50047E03 01AF0177 00000000 0 NL Granted FFFFFFFE.FB8D9480 .... Exec
FFFFFFFE.F2B13C00 6305AA9C 50047E03 01AF0177 00000000 0 NL Granted FFFFFFFE.FD620080 .... Exec
FFFFFFFE.F1528800 560BB1A2 50047E03 01AF0177 00000000 0 NL Granted FFFFFFFE.F2036F80 .... Exec

2) SDA> show process decedi$csf_2

Process index: 0177 Name: DECEDI$CSF_2 Extended PID: 20835F77
--------------------------------------------------------------------
Process status: 00140001 RES,PHDRES,LOGIN
status2: 00000001 QUANTUM_RESCHED

PCB address 83720940 JIB address 82544D80
PHD address 91BD2000 Swapfile disk address 00000000
KTB vector address 83720C30 HWPCB address FFFFFFFF.91BD2080
Callback vector address 00000000 Termination mailbox 1DDB
Master internal PID 01AF0177 Subprocess count 0
Creator extended PID 00000000 Creator internal PID 00000000
Previous CPU Id 00000001 Current CPU Id 00000001
Previous ASNSEQ 0000000000150D39 Previous ASN 0000000000000020
Initial process priority 4 # open files remaining 1986/2000
Delete pending count 0 Direct I/O count/limit 5000/5000
UIC [00300,000010] Buffered I/O count/limit 5000/5000
Abs time of last event 1268C64F BUFIO byte count/limit 7994944/7994944
# of threads 1 ASTs remaining 4999/5000
Swapped copy of LEFC0 00000000 Timer entries remaining 999/1000
Swapped copy of LEFC1 00000000 Active page table count 0
Global cluster 2 pointer 84456C80 Process WS page count 901
Global cluster 3 pointer 00000000 Global WS page count 512
PCB Specific Spinlock 83718F00 Subprocesses in job 0

Process index: 0177 Name: DECEDI$CSF_2 Extended PID: 20835F77
--------------------------------------------------------------------

Thread index: 0000
------------------
Current capabilities: System: 0000000C QUORUM,RUN
User: 00000000
Permanent capabilities: System: 0000000C QUORUM,RUN
User: 00000000
Current affinities: 00000000
Permanent affinities: 00000000
Thread status: 00140001
status2: 00000001

KTB address 83720940 HWPCB address FFFFFFFF.91BD2080
PKTA address 7FFEFF98 Callback vector address 00000000
Internal PID 01AF0177 Callback error 00000000
Extended PID 20835F77 Current CPU id 00000001
State LEF Flags 00000000
Base priority 4 Current priority 6
Waiting EF cluster 0 Event flag wait mask FDFFFFFF
CPU since last quantum 0001 Mutex count 0
ASTs active NONE

3) show sys/process=decedi$csf_2

$ show sys/process=decedi$csf_2
OpenVMS V8.2 on node EDICL1 16-AUG-2007 10:31:01.22 Uptime 35 17:56:56
Pid Process Name State Pri I/O CPU Page flts Pages
20835F77 DECEDI$CSF_2 LEF 6 21097954 0 10:12:53.41 1935 1413

From above information can we know why decedi$csf_2 process hangs


Regards,
ajaydec

[#4]

Hi,

We have a process by name DECEDI$CSF_2 which has to do some kind of process but right now its not performing and it is in hanging postion as if it is waiting for some resource.
Below are the output of SDA

1) SDA> show process decedi$csf_2 /locks /brief

Process index: 0177 Name: DECEDI$CSF_2 Extended PID: 20835F77
--------------------------------------------------------------------
LKB Address Lockid ParentId PID BLKAST SubLocks RQ GR Queue RSB Address Resource Name Mode
----------------- -------- -------- -------- -------- -------- -- -- ------- ----------------- ------------------------------- ----
FFFFFFFE.FB5ACF80 370B301C 4C0AB147 01AF0177 01573550 0 CR Granted FFFFFFFE.EC5AE1C0 G..... Exec
FFFFFFFE.F33C6A80 7D08FE6A 4C0AB147 01AF0177 01573550 0 EX Granted FFFFFFFE.FD04A800 G..... Exec
FFFFFFFE.F86E7AC0 1A098B1D 4C0AB147 01AF0177 01573550 0 EX Granted FFFFFFFE.EC4BABC0 R..... Exec
FFFFFFFE.F73A86C0 67018746 4C0AB147 01AF0177 00000000 0 NL Granted FFFFFFFE.EC2BDE80 R..... Exec
FFFFFFFE.F6486BC0 660855B0 4C0AB147 01AF0177 00000000 0 PR Waiting FFFFFFFE.EC8E75C0 I..... Exec
FFFFFFFE.F0709980 130CB475 4C0AB147 01AF0177 00000000 0 PR Waiting FFFFFFFE.FB162F80 I..... Exec
FFFFFFFE.F795F340 200142BC 4C0AB147 01AF0177 00000000 0 NL Granted FFFFFFFE.FA164580 H... Exec
FFFFFFFE.EE835D40 4D0DF0B7 00000000 01AF0177 00000000 0 EX Granted FFFFFFFE.F9C3C1C0 BG2477 User
FFFFFFFE.FD4ACF80 50047E03 00000000 01AF0177 00000000 5 NL Granted FFFFFFFE.FB8D9340 RMS$.......SYS_DISK ... Exec
FFFFFFFE.EEFAB5C0 3E0C0965 50047E03 01AF0177 00000000 0 NL Granted FFFFFFFE.FB8D9480 .... Exec
FFFFFFFE.F2B13C00 6305AA9C 50047E03 01AF0177 00000000 0 NL Granted FFFFFFFE.FD620080 .... Exec
FFFFFFFE.F1528800 560BB1A2 50047E03 01AF0177 00000000 0 NL Granted FFFFFFFE.F2036F80 .... Exec

2) SDA> show process decedi$csf_2

Process index: 0177 Name: DECEDI$CSF_2 Extended PID: 20835F77
--------------------------------------------------------------------
Process status: 00140001 RES,PHDRES,LOGIN
status2: 00000001 QUANTUM_RESCHED

PCB address 83720940 JIB address 82544D80
PHD address 91BD2000 Swapfile disk address 00000000
KTB vector address 83720C30 HWPCB address FFFFFFFF.91BD2080
Callback vector address 00000000 Termination mailbox 1DDB
Master internal PID 01AF0177 Subprocess count 0
Creator extended PID 00000000 Creator internal PID 00000000
Previous CPU Id 00000001 Current CPU Id 00000001
Previous ASNSEQ 0000000000150D39 Previous ASN 0000000000000020
Initial process priority 4 # open files remaining 1986/2000
Delete pending count 0 Direct I/O count/limit 5000/5000
UIC [00300,000010] Buffered I/O count/limit 5000/5000
Abs time of last event 1268C64F BUFIO byte count/limit 7994944/7994944
# of threads 1 ASTs remaining 4999/5000
Swapped copy of LEFC0 00000000 Timer entries remaining 999/1000
Swapped copy of LEFC1 00000000 Active page table count 0
Global cluster 2 pointer 84456C80 Process WS page count 901
Global cluster 3 pointer 00000000 Global WS page count 512
PCB Specific Spinlock 83718F00 Subprocesses in job 0

Process index: 0177 Name: DECEDI$CSF_2 Extended PID: 20835F77
--------------------------------------------------------------------

Thread index: 0000
------------------
Current capabilities: System: 0000000C QUORUM,RUN
User: 00000000
Permanent capabilities: System: 0000000C QUORUM,RUN
User: 00000000
Current affinities: 00000000
Permanent affinities: 00000000
Thread status: 00140001
status2: 00000001

KTB address 83720940 HWPCB address FFFFFFFF.91BD2080
PKTA address 7FFEFF98 Callback vector address 00000000
Internal PID 01AF0177 Callback error 00000000
Extended PID 20835F77 Current CPU id 00000001
State LEF Flags 00000000
Base priority 4 Current priority 6
Waiting EF cluster 0 Event flag wait mask FDFFFFFF
CPU since last quantum 0001 Mutex count 0
ASTs active NONE

3) show sys/process=decedi$csf_2

$ show sys/process=decedi$csf_2
OpenVMS V8.2 on node EDICL1 16-AUG-2007 10:31:01.22 Uptime 35 17:56:56
Pid Process Name State Pri I/O CPU Page flts Pages
20835F77 DECEDI$CSF_2 LEF 6 21097954 0 10:12:53.41 1935 1413

From above information can we know why decedi$csf_2 process hangs


Regards,
ajaydec

[Volker Halle]

ajaydec,

start with SDA> SHOW PROC/LOCK and look for locks in the 'waiting' state. There seem to be 2 exec mode locks in your case.

Then try SDA> SHOW PROC/CHAN and look for 'busy' channels.

Why do you consider this to be a 'system performance' issue ?

Watch that process with SHOW PROC/CONT/ID=xxx

Does it do anything ? like DIRIOs ? Then it might just be busy reading/write files.

Volker.

[#6]

Hi,

The output of show process /id=xxx /cont is:


Process DECEDI$CSF_2 10:53:16



State LEF Working set 1413

Cur/base priority 9/4 Virtual pages 16487

Current PC 80155454 CPU time 0 16:00:31.39

Current PSL 0000001B Direct I/O 33054205

Current user SP 7AE371F0 Buffered I/O 333

PID 20835F77 Page faults 1935

UIC [DECEDI] Event flags C1000081
80000000


$1$DKA0:[SYS1.SYSCOMMON.][SYSEXE]DECEDI$CSF.EXE


Its almost more than 48 hr. then also decedi$csf_2 process is not performing. The state of decedi$csf_2 is still in LEF state.

OpenVMS V8.2 on node EDICL1 17-AUG-2007 10:54:24.95 Uptime 36 18:20:16
Pid Process Name State Pri I/O CPU Page flts Pages
20835F77 DECEDI$CSF_2 LEF 6 33062990 0 16:00:46.20 1935 1413


What should I look to know why decedi$csf_2 process is in hanging postion. Is decedi$csf_2 is waiting for some kind of resource then how to know about that.
And there is no other process running on the system.
Output of monitor system is:

Process States
+ CPU Busy (34) â + LEF: 15 LEFO: 0
| HIB: 58 HIBO: 0
CPU 0 400 COM: 0 COMO: 0
|* | PFW: 0 CUR: 1
MWAIT: 0 Other: 0
Cur Top: DECEDI$CSF_2 (30) Total: 74

+ Page Fault Rate (0) â + + Free List Size (173307) +
|| | | 512K
MEMORY 0 500 0
| | | | 128K
+ Mod List Size (4965) +
Cur Top: NITIN (0)

+ Direct I/O Rate (173) â + + Buffered I/O Rate (3) â +
| | |
I/O 0 500 0 500
|
Cur Top: DECEDI$CSF_2 (172) Cur Top: NITIN (1)



Out of 34% of busy CPU 30% is because of decedi$csf_2 and out of 173 Direct I/O Rate 172 is because of DECEDI$CSF_2.

Can I know what can we infer from this.

Regards,
Ajaydec

[Volker Halle]

Ajaydec,

the SHOW PROC/CONT and MONITOR data is mostly unreadable...

This process seems to be consuming lots of CPU time and seems to be doing lots of DIRIOs (probably disk-IOs). It is NOT hung ! The fact that you always see it in LEF state is because it's waiting for it's IOs to finish.

If it does not seem to do 'any work' application-wise, it could very well be in a loop caused by some error condition.

You could find out with SDA, which files this process is reading or writing:

$ ANA/SYS
SDA> SET PROC/ID=xxx
SDA> SHOW PROC/CHAN
SDA> SHOW PROC/RMS=WCB
...
SDA> EXIT

Look at the read and write counts in the Window Control Block for very high numbers:

READS: 0000000x WRITES: 0000000x

Then correlate the WCB address shown with the address shown in the Window column of the SHOW PROC/CHAN display to find the related file name.

Maybe you should consider re-starting that application...

Volker.

[#8]

Hi,

Please find the output of monitor system in the attachment.

As written before, decedi$csf_2 process is doing lots of direct I/O, out of total 172 I/O of system 170 is because of decedi$csf_2 but how to find what the process is reading and where it is writing.

I followed the above steps:

$ ANA/SYS
SDA> SET PROC/ID=xxx
SDA> SHOW PROC/CHAN
SDA> SHOW PROC/RMS=WCB
...
SDA> EXIT


But in SDA> show proc /rms=wcb I found only one file which was being read by decedi$csf_2 process:


WLFL: 84200F80 SIZE: 00B0
WLBL: 82420280 TYPE: 12
ACCESS: 01 READ
PID: 01AF0177 ORGUCB: 8212BE40
ACON: 00000000
NMAP: 00000002 FCB: 82294900
RVT: 00000000 LINK: 00000000
READS: 00000010 WRITES: 00000000
STVBN: 00000001 P1_COUNT: 00000020
P1_RVN: 00000000
P1_LBN: 00E60770 P2_COUNT: 00000010
P2_RVN: 00000000
P2_LBN: 00E78DC0

But what can I infer from this.

Regards,
ajaydec

[#9]

hi volker
as u said that ur nopt able to read the previous update properly please find the same as atatchment

[Volker Halle]

ajaydec,

so the process does not seem to use RMS or it's closing and re-openening the file all the time.

SDA> SHOW PROC/CHAN will show all open files. If you are lucky, one of them may show as 'busy', i.e. an IO is currently outstanding.

$ MONI DISK - do you see disk-IOs, which add up to about 170 IO/sec - which disk(s) ?

$ MONI FCP will show, if there is a high file lookup/open rate.

Volker.