Disk IO without doing IO's

[Wim Van den Wyngaert]

I have found a cluster on which a program is doing strange things.

Normal situation : program is not having significant thruput to any disk.

Current situation : program is doing top thruput to some disks (not always the same disk).

In VPA for 21:00 - 23:59
1) bufio almost 0
2) direct IO rate : 0.004 (continuously)
3) disk IO about 100 per sec, thruput about 4 MB per sec (item DISKTP, to disk dsa2)
4) IO size about 80 pages
5) no top hot files

My questions.
1) how to anylyze what is causing this thruput ?
2) how can there be disk io without direct IO ?

VMS 7.3. Also seen on 6.2 but difficult to confirm.

Wim

[Willem Grooters]

Wim,

could it be the program bypasses RMS to access the disk (Logical or eve physical IO)? It might be scanning the disk for some reason, reading chunks of data this way (fast access). Or accessing particular locations?
I guess yuo'd need to check the system to locate the program that does these strange things and analyze it's device access.

[Wim Van den Wyngaert]

Willem,

If I do sda show proc/chan I see :

channel to dsa10
some exe stuff
std in/out
4 bg devices of which 3 are busy
1 has 3 operations, 1 has 433K op, 1 has 744K and the non-busy has 0 operation. This after 23 days uptime. So the BG device can't be blamed ?

How can I check the physical and logical IO ?

Wim

[Andy Bustamante]

What do your paging figures look like?

Does the the program use global sections backed by files? In this case VMS will report I/O as a hard page fault.

[Richard White_5]

Good Morning Wim...

Very interesting problem... Both Andy and Willem have suggested excellent avenues to pursue. You mentioned that the I/O is not always to the same disk-volume, so Andy's suggestion of possible Paging-I/O may very well be valid, especially if there are Page-Files on the same disk(s) that DISKTP was showing the I/O activity.

As for Willem's suggestion of the QIO's possibly bypassing RMS, you might be able to use "SDA> sho proc/rms/ind=xxxx "... If the particular process is using RMS to execute RMS_Services (Open, Get, Put, etc.) to a particular file; then the 6th and 7th screens from the SDA> command will show you some of the pertinent info.

For example the sixth screen will display the Channel-Control-Block Address, the Unit-Control_Block Address of the Disk-Volume, and the Window-Control-Block Address that contains the retrieval pointers to the file.

The seventh screen of "SDA>sho proc/rms" will display the File-Control-Block with File-Header info, and the total number of Reads and Writes (per RMS) are also displayed on this seventh screen. Some of the prelim screens will display the FAB/RAB and XAB RMS-Info. If RMS_Services are not being utilized for the QIO, then a message indicating No_Image/RMS structures are loaded, will be displayed back to SDA>.

Still trying to pursue a method for tracking Phys_IO vs. Log_IO vs. Virt_IO...

Thanx,
whynot3k

[Robert Gezelter]

Wlm,

To separate the paging traffic from the regular IO, consider using a virtual disk (LD). Then, the non-paging traffic can be traced and the details analyzed.

- Bob Gezelter, http://www.rlgsc.com

[Wim Van den Wyngaert]

The process is an interface between a program and another process on the machine. Or on another node of the same cluster. It is using TCP/IP. Has nothing on disk. Details unknown.

Andy : no paging activity.

Richard : I have currently no channel visible to anything on dsa2. So, show proc/rms can't show me something. And VPA shows no file activity.

Wim

[Wim Van den Wyngaert]

Rather strange news :

Yesterday between 18 and 19h there was activity by the process going to dsa4. And dsa4 contains only Sybase database files. And the process has nothing to do with that disk but still there are IO's registered on that process ...

A bug in the program that confuses VPA ???

Wim

[Willem Grooters]

Wim,

Given the process uses BG-devices - obviousy since it uses TCPIP - you can find out which ports it uses. Therefore a suggestion, to get at least a clue what's going on: set up TCPTRACE to listen to the busy ports and dumping what's found. That would only be feasable if these ports are fixed, of course.
Perhaps it may give a clue what's going on within that proces.

[Richard White_5]

Good Morning Wim...

Becoming more interesting by the hour... Is it possible that this process/program interface/communicates with an Application Server/Control Process, possibly via mailboxes or global_sections? Or is this a spawned-sub-process or multi-threaded process?

I realize that you are being requested to try multiple/different suggestions, but if you are able to find the time during the next episode, I would like to request an ouptut-log-file or screen-capture if possible.

If you could capture and forward the output of the following SDA> commands, (executed 3 or 4 times,) it may jog someone's memory (Andy, Willem, yourself, myself, etc.)

SDA> show proc/ind=xxxx
SDA> show proc/sema
SDA> show proc/proc (or /pst)
SDA> show proc/reg
SDA> show proc/imag
SDA> show proc/chan
SDA> show proc/lock
SDA> sho proc/rms

I refrain from suggesting SDA>show proc/all since I don't think that we need the WSLE's or PTE's at this point...

Thanx,
whynot3k