regarding system performance

[#1]

hi please find the attachment




regards
ajaydec

[Steven Schweda]

I found it. Now, can you attach a copy of
Microsoft Word for VMS Alpha so I can read
it? Or, you could just attach a plain-text
version, which everyone could read easily.

[#3]

Hi,

We have a process by name DECEDI$CSF_2 which has to do some kind of process but right now its not performing and it is in hanging postion as if it is waiting for some resource.
Below are the output of SDA

1) SDA> show process decedi$csf_2 /locks /brief

Process index: 0177 Name: DECEDI$CSF_2 Extended PID: 20835F77
--------------------------------------------------------------------
LKB Address Lockid ParentId PID BLKAST SubLocks RQ GR Queue RSB Address Resource Name Mode
----------------- -------- -------- -------- -------- -------- -- -- ------- ----------------- ------------------------------- ----
FFFFFFFE.FB5ACF80 370B301C 4C0AB147 01AF0177 01573550 0 CR Granted FFFFFFFE.EC5AE1C0 G..... Exec
FFFFFFFE.F33C6A80 7D08FE6A 4C0AB147 01AF0177 01573550 0 EX Granted FFFFFFFE.FD04A800 G..... Exec
FFFFFFFE.F86E7AC0 1A098B1D 4C0AB147 01AF0177 01573550 0 EX Granted FFFFFFFE.EC4BABC0 R..... Exec
FFFFFFFE.F73A86C0 67018746 4C0AB147 01AF0177 00000000 0 NL Granted FFFFFFFE.EC2BDE80 R..... Exec
FFFFFFFE.F6486BC0 660855B0 4C0AB147 01AF0177 00000000 0 PR Waiting FFFFFFFE.EC8E75C0 I..... Exec
FFFFFFFE.F0709980 130CB475 4C0AB147 01AF0177 00000000 0 PR Waiting FFFFFFFE.FB162F80 I..... Exec
FFFFFFFE.F795F340 200142BC 4C0AB147 01AF0177 00000000 0 NL Granted FFFFFFFE.FA164580 H... Exec
FFFFFFFE.EE835D40 4D0DF0B7 00000000 01AF0177 00000000 0 EX Granted FFFFFFFE.F9C3C1C0 BG2477 User
FFFFFFFE.FD4ACF80 50047E03 00000000 01AF0177 00000000 5 NL Granted FFFFFFFE.FB8D9340 RMS$.......SYS_DISK ... Exec
FFFFFFFE.EEFAB5C0 3E0C0965 50047E03 01AF0177 00000000 0 NL Granted FFFFFFFE.FB8D9480 .... Exec
FFFFFFFE.F2B13C00 6305AA9C 50047E03 01AF0177 00000000 0 NL Granted FFFFFFFE.FD620080 .... Exec
FFFFFFFE.F1528800 560BB1A2 50047E03 01AF0177 00000000 0 NL Granted FFFFFFFE.F2036F80 .... Exec

2) SDA> show process decedi$csf_2

Process index: 0177 Name: DECEDI$CSF_2 Extended PID: 20835F77
--------------------------------------------------------------------
Process status: 00140001 RES,PHDRES,LOGIN
status2: 00000001 QUANTUM_RESCHED

PCB address 83720940 JIB address 82544D80
PHD address 91BD2000 Swapfile disk address 00000000
KTB vector address 83720C30 HWPCB address FFFFFFFF.91BD2080
Callback vector address 00000000 Termination mailbox 1DDB
Master internal PID 01AF0177 Subprocess count 0
Creator extended PID 00000000 Creator internal PID 00000000
Previous CPU Id 00000001 Current CPU Id 00000001
Previous ASNSEQ 0000000000150D39 Previous ASN 0000000000000020
Initial process priority 4 # open files remaining 1986/2000
Delete pending count 0 Direct I/O count/limit 5000/5000
UIC [00300,000010] Buffered I/O count/limit 5000/5000
Abs time of last event 1268C64F BUFIO byte count/limit 7994944/7994944
# of threads 1 ASTs remaining 4999/5000
Swapped copy of LEFC0 00000000 Timer entries remaining 999/1000
Swapped copy of LEFC1 00000000 Active page table count 0
Global cluster 2 pointer 84456C80 Process WS page count 901
Global cluster 3 pointer 00000000 Global WS page count 512
PCB Specific Spinlock 83718F00 Subprocesses in job 0

Process index: 0177 Name: DECEDI$CSF_2 Extended PID: 20835F77
--------------------------------------------------------------------

Thread index: 0000
------------------
Current capabilities: System: 0000000C QUORUM,RUN
User: 00000000
Permanent capabilities: System: 0000000C QUORUM,RUN
User: 00000000
Current affinities: 00000000
Permanent affinities: 00000000
Thread status: 00140001
status2: 00000001

KTB address 83720940 HWPCB address FFFFFFFF.91BD2080
PKTA address 7FFEFF98 Callback vector address 00000000
Internal PID 01AF0177 Callback error 00000000
Extended PID 20835F77 Current CPU id 00000001
State LEF Flags 00000000
Base priority 4 Current priority 6
Waiting EF cluster 0 Event flag wait mask FDFFFFFF
CPU since last quantum 0001 Mutex count 0
ASTs active NONE

3) show sys/process=decedi$csf_2

$ show sys/process=decedi$csf_2
OpenVMS V8.2 on node EDICL1 16-AUG-2007 10:31:01.22 Uptime 35 17:56:56
Pid Process Name State Pri I/O CPU Page flts Pages
20835F77 DECEDI$CSF_2 LEF 6 21097954 0 10:12:53.41 1935 1413

From above information can we know why decedi$csf_2 process hangs


Regards,
ajaydec

[#4]

Hi,

We have a process by name DECEDI$CSF_2 which has to do some kind of process but right now its not performing and it is in hanging postion as if it is waiting for some resource.
Below are the output of SDA

1) SDA> show process decedi$csf_2 /locks /brief

Process index: 0177 Name: DECEDI$CSF_2 Extended PID: 20835F77
--------------------------------------------------------------------
LKB Address Lockid ParentId PID BLKAST SubLocks RQ GR Queue RSB Address Resource Name Mode
----------------- -------- -------- -------- -------- -------- -- -- ------- ----------------- ------------------------------- ----
FFFFFFFE.FB5ACF80 370B301C 4C0AB147 01AF0177 01573550 0 CR Granted FFFFFFFE.EC5AE1C0 G..... Exec
FFFFFFFE.F33C6A80 7D08FE6A 4C0AB147 01AF0177 01573550 0 EX Granted FFFFFFFE.FD04A800 G..... Exec
FFFFFFFE.F86E7AC0 1A098B1D 4C0AB147 01AF0177 01573550 0 EX Granted FFFFFFFE.EC4BABC0 R..... Exec
FFFFFFFE.F73A86C0 67018746 4C0AB147 01AF0177 00000000 0 NL Granted FFFFFFFE.EC2BDE80 R..... Exec
FFFFFFFE.F6486BC0 660855B0 4C0AB147 01AF0177 00000000 0 PR Waiting FFFFFFFE.EC8E75C0 I..... Exec
FFFFFFFE.F0709980 130CB475 4C0AB147 01AF0177 00000000 0 PR Waiting FFFFFFFE.FB162F80 I..... Exec
FFFFFFFE.F795F340 200142BC 4C0AB147 01AF0177 00000000 0 NL Granted FFFFFFFE.FA164580 H... Exec
FFFFFFFE.EE835D40 4D0DF0B7 00000000 01AF0177 00000000 0 EX Granted FFFFFFFE.F9C3C1C0 BG2477 User
FFFFFFFE.FD4ACF80 50047E03 00000000 01AF0177 00000000 5 NL Granted FFFFFFFE.FB8D9340 RMS$.......SYS_DISK ... Exec
FFFFFFFE.EEFAB5C0 3E0C0965 50047E03 01AF0177 00000000 0 NL Granted FFFFFFFE.FB8D9480 .... Exec
FFFFFFFE.F2B13C00 6305AA9C 50047E03 01AF0177 00000000 0 NL Granted FFFFFFFE.FD620080 .... Exec
FFFFFFFE.F1528800 560BB1A2 50047E03 01AF0177 00000000 0 NL Granted FFFFFFFE.F2036F80 .... Exec

2) SDA> show process decedi$csf_2

Process index: 0177 Name: DECEDI$CSF_2 Extended PID: 20835F77
--------------------------------------------------------------------
Process status: 00140001 RES,PHDRES,LOGIN
status2: 00000001 QUANTUM_RESCHED

PCB address 83720940 JIB address 82544D80
PHD address 91BD2000 Swapfile disk address 00000000
KTB vector address 83720C30 HWPCB address FFFFFFFF.91BD2080
Callback vector address 00000000 Termination mailbox 1DDB
Master internal PID 01AF0177 Subprocess count 0
Creator extended PID 00000000 Creator internal PID 00000000
Previous CPU Id 00000001 Current CPU Id 00000001
Previous ASNSEQ 0000000000150D39 Previous ASN 0000000000000020
Initial process priority 4 # open files remaining 1986/2000
Delete pending count 0 Direct I/O count/limit 5000/5000
UIC [00300,000010] Buffered I/O count/limit 5000/5000
Abs time of last event 1268C64F BUFIO byte count/limit 7994944/7994944
# of threads 1 ASTs remaining 4999/5000
Swapped copy of LEFC0 00000000 Timer entries remaining 999/1000
Swapped copy of LEFC1 00000000 Active page table count 0
Global cluster 2 pointer 84456C80 Process WS page count 901
Global cluster 3 pointer 00000000 Global WS page count 512
PCB Specific Spinlock 83718F00 Subprocesses in job 0

Process index: 0177 Name: DECEDI$CSF_2 Extended PID: 20835F77
--------------------------------------------------------------------

Thread index: 0000
------------------
Current capabilities: System: 0000000C QUORUM,RUN
User: 00000000
Permanent capabilities: System: 0000000C QUORUM,RUN
User: 00000000
Current affinities: 00000000
Permanent affinities: 00000000
Thread status: 00140001
status2: 00000001

KTB address 83720940 HWPCB address FFFFFFFF.91BD2080
PKTA address 7FFEFF98 Callback vector address 00000000
Internal PID 01AF0177 Callback error 00000000
Extended PID 20835F77 Current CPU id 00000001
State LEF Flags 00000000
Base priority 4 Current priority 6
Waiting EF cluster 0 Event flag wait mask FDFFFFFF
CPU since last quantum 0001 Mutex count 0
ASTs active NONE

3) show sys/process=decedi$csf_2

$ show sys/process=decedi$csf_2
OpenVMS V8.2 on node EDICL1 16-AUG-2007 10:31:01.22 Uptime 35 17:56:56
Pid Process Name State Pri I/O CPU Page flts Pages
20835F77 DECEDI$CSF_2 LEF 6 21097954 0 10:12:53.41 1935 1413

From above information can we know why decedi$csf_2 process hangs


Regards,
ajaydec

[Volker Halle]

ajaydec,

start with SDA> SHOW PROC/LOCK and look for locks in the 'waiting' state. There seem to be 2 exec mode locks in your case.

Then try SDA> SHOW PROC/CHAN and look for 'busy' channels.

Why do you consider this to be a 'system performance' issue ?

Watch that process with SHOW PROC/CONT/ID=xxx

Does it do anything ? like DIRIOs ? Then it might just be busy reading/write files.

Volker.

[#6]

Hi,

The output of show process /id=xxx /cont is:


Process DECEDI$CSF_2 10:53:16



State LEF Working set 1413

Cur/base priority 9/4 Virtual pages 16487

Current PC 80155454 CPU time 0 16:00:31.39

Current PSL 0000001B Direct I/O 33054205

Current user SP 7AE371F0 Buffered I/O 333

PID 20835F77 Page faults 1935

UIC [DECEDI] Event flags C1000081
80000000


$1$DKA0:[SYS1.SYSCOMMON.][SYSEXE]DECEDI$CSF.EXE


Its almost more than 48 hr. then also decedi$csf_2 process is not performing. The state of decedi$csf_2 is still in LEF state.

OpenVMS V8.2 on node EDICL1 17-AUG-2007 10:54:24.95 Uptime 36 18:20:16
Pid Process Name State Pri I/O CPU Page flts Pages
20835F77 DECEDI$CSF_2 LEF 6 33062990 0 16:00:46.20 1935 1413


What should I look to know why decedi$csf_2 process is in hanging postion. Is decedi$csf_2 is waiting for some kind of resource then how to know about that.
And there is no other process running on the system.
Output of monitor system is:

Process States
+ CPU Busy (34) â + LEF: 15 LEFO: 0
| HIB: 58 HIBO: 0
CPU 0 400 COM: 0 COMO: 0
|* | PFW: 0 CUR: 1
MWAIT: 0 Other: 0
Cur Top: DECEDI$CSF_2 (30) Total: 74

+ Page Fault Rate (0) â + + Free List Size (173307) +
|| | | 512K
MEMORY 0 500 0
| | | | 128K
+ Mod List Size (4965) +
Cur Top: NITIN (0)

+ Direct I/O Rate (173) â + + Buffered I/O Rate (3) â +
| | |
I/O 0 500 0 500
|
Cur Top: DECEDI$CSF_2 (172) Cur Top: NITIN (1)



Out of 34% of busy CPU 30% is because of decedi$csf_2 and out of 173 Direct I/O Rate 172 is because of DECEDI$CSF_2.

Can I know what can we infer from this.

Regards,
Ajaydec

[Volker Halle]

Ajaydec,

the SHOW PROC/CONT and MONITOR data is mostly unreadable...

This process seems to be consuming lots of CPU time and seems to be doing lots of DIRIOs (probably disk-IOs). It is NOT hung ! The fact that you always see it in LEF state is because it's waiting for it's IOs to finish.

If it does not seem to do 'any work' application-wise, it could very well be in a loop caused by some error condition.

You could find out with SDA, which files this process is reading or writing:

$ ANA/SYS
SDA> SET PROC/ID=xxx
SDA> SHOW PROC/CHAN
SDA> SHOW PROC/RMS=WCB
...
SDA> EXIT

Look at the read and write counts in the Window Control Block for very high numbers:

READS: 0000000x WRITES: 0000000x

Then correlate the WCB address shown with the address shown in the Window column of the SHOW PROC/CHAN display to find the related file name.

Maybe you should consider re-starting that application...

Volker.

[#8]

Hi,

Please find the output of monitor system in the attachment.

As written before, decedi$csf_2 process is doing lots of direct I/O, out of total 172 I/O of system 170 is because of decedi$csf_2 but how to find what the process is reading and where it is writing.

I followed the above steps:

$ ANA/SYS
SDA> SET PROC/ID=xxx
SDA> SHOW PROC/CHAN
SDA> SHOW PROC/RMS=WCB
...
SDA> EXIT


But in SDA> show proc /rms=wcb I found only one file which was being read by decedi$csf_2 process:


WLFL: 84200F80 SIZE: 00B0
WLBL: 82420280 TYPE: 12
ACCESS: 01 READ
PID: 01AF0177 ORGUCB: 8212BE40
ACON: 00000000
NMAP: 00000002 FCB: 82294900
RVT: 00000000 LINK: 00000000
READS: 00000010 WRITES: 00000000
STVBN: 00000001 P1_COUNT: 00000020
P1_RVN: 00000000
P1_LBN: 00E60770 P2_COUNT: 00000010
P2_RVN: 00000000
P2_LBN: 00E78DC0

But what can I infer from this.

Regards,
ajaydec

[#9]

hi volker
as u said that ur nopt able to read the previous update properly please find the same as atatchment

[Volker Halle]

ajaydec,

so the process does not seem to use RMS or it's closing and re-openening the file all the time.

SDA> SHOW PROC/CHAN will show all open files. If you are lucky, one of them may show as 'busy', i.e. an IO is currently outstanding.

$ MONI DISK - do you see disk-IOs, which add up to about 170 IO/sec - which disk(s) ?

$ MONI FCP will show, if there is a high file lookup/open rate.

Volker.

[Hein van den Heuvel]

>> READS: 00000010 WRITES: 00000000
:
>> P1_COUNT: 00000020 P1_LBN: 00E60770
>> P2_COUNT: 00000010 P2_LBN: 00E78DC0

>> But what can I infer from this.

First of all we notice that you are dealing with a relatively fragmented file. Tiny chunks for a large allocation. :-(.
Secondly we see that the actually IO to this file might not be a problem (low read/write count).
But the most important information would come from the next time you issue the same command: Does it change!? Is the application making progress in the file?

You might also want to use a wonderful SDA extention Volker wrote: PROCIO. See:
http://eisner.encompasserve.org/~halle/

You may also want to use SHOW MEM/CACH=(VOLU=xxx,TOPQIO) to see cached IO activity counts for the suspect file.

I don't know EDI... does is work with RMS sequential files? Indexed? What picture does MONI MODE paint.

And... did you consider that it all might be working just fine? Do you have comparison material to suggest what you see is wrong?

Does the EDI tool have an internal/application level performance/throughput monitor to help show what it is doing?

Steven wrote>> found it. Now, can you attach a copy of Microsoft Word for VMS Alpha

I agree with that sentiment Steven... Ajaydec, the data which you attached is simple text. KISS. Attach it as a simple text file and the retrieval and formatting will be so much easier for teh folks who are trying to help.

Good luck!
Hein.

[#12]

Hi,

1) decedi$csf_2 process works on rms files. It is used to fetch the file from data base and then process it and then save it again in different data base and also on hard disk. Right now there are lots of files to be fetched by decedi$csf_2 process but decedi$csf_2 is not processing those files and now its almost more then 60 hr, since decedi$csf_2 has fetched last file.

2) I have attached output of following commands with the attachment in *.doc format to increase the readability.
a) Monitor system
b) show sys/process=decedi$csf_2 /full
c) show process decedi$csf_2 /locks /brief
d) SDA> show process decedi$csf_2
e) show process /id=xxx /cont
f) monitor mode
g) SDA> show process /chan
h) monitor disk
i) monitor fcp
j) SDA> SHOW PROC/RMS=WCB

Is there any more information I can provide to know why decedi$csf_2 process is not performing.

Regards,
ajaydec

[Steven Schweda]

> 2) I have attached output of following
> commands with the attachment in *.doc
> format to increase the readability.

This must be some new meaning for "increase".

[Volker Halle]

ajaydec,

why do you think you need to put simple text data into a proprietory file format ? A simple .TXT file would have been sufficient ...

EDI seems to be using a RDB database ! The files on $1$DKA300: are .RDA and .RDB files.

You need to troubleshoot this with RDB database utilities.

Volker.

[Hein van den Heuvel]

That does NOT look like an RMS application, but rather an RDB application. Your next tools of choice would be RDB tools.

For example (if memory serves me right) :

RMU/SHOW STATISTICS/SCREEN=STALL


Did you check (operator) log files for simple things as 'disk full' or quota problems?

After so long not processing records, I'd be mighty tempted to re-start the application and/or the database and hope for the best.

At the very least a restart is an opportunity for a clear error/warning message.

Good luck!
Hein.

[Volker Halle · Accepted solution]

ajaydec,

you might need to specify the root db file:

$ rmu/show stat $1$dka300:[audit_db]decedi$audit_db.rdb

To further troubleshoot this, you will probably need to find someone, who understands and is able to support DECEDI !

Volker.

[#17]

I am sorry, I just want to help you to get better picture of the process and then help me.
Now I got something relevant that it is something to do with the database.
Thanks to all for their time and support.

Now can anyone help me how to troubleshoot for the database.

Regards,
ajaydec

[Doug Phillips]

What does this EDI module do? (I don't know DECEDI) Since it has TCPIP open, is it the send or receive or format conversion module?

If your RDB tools don't show that the container is corrupted or full, suspect the EDI data. Maybe your trading partner has sent a new format version? There could be a new record set that is causing a tight logic-loop. The LEF state could also indicate that the program might be trying to send a message to and/or get a response from a device that isn't available.

[#19]

REGARDS
AJAYDEC