Process introspection, resource consumption

[Ralph Grothe]

Hi,

to obtain a clearer picture of real memory consumption of oracle (or any other) processes I delved into the struct definitions of the pstat() API, hoping to find enough explanation from the short line trailing comments therein
(manpages of ps, pstat are too vague and neglectant)

However, due to my poor C coding and the absence of a troublefree ANSI C compiler I rely on CPAN's Proc::ProcessTable,
whose HP-UX implementation gives access to almost all struct members of __pst_status.
(I reckon the ps command is using this very struct although the leading underscores convey some sort of privacy)


$ perl -MProc::ProcessTable -e 'print "@{[Proc::ProcessTable->new->fields]}\n"'
uid pid ppid dsize tsize ssize nice ttynum pgrp pri addr cpu utime stime start flag state wchan procnum cmndline fname time cpticks cptickstotal fss pctcpu rssize suid ucomm shmsize mmsize usize iosize vtsize vdsize vssize vshmsize vmmsize vusize viosize minorfaults majorfaults nswap nsignals msgrcv msgsnd maxrss sid schedpolicy ticksleft rdir cdir text highestfd euid egid ioch usercycles systemcycles interruptcycles gid lwpid


Since I'm especially interested in memory consumption I go for the various sizes.
The so-called real d,s,t sizes are plausible to me.
But how can I interpret the so-called virtual entities?
Do they relate to the similar notion of virtual memory (viz. physical and swap memory combined)?

These are the struct members I assume to be of interest to me.

$ grep size /usr/include/sys/pstat/pm_pstat_body.h
_T_LONG_T pst_dsize; /* # real pages used for data */
_T_LONG_T pst_tsize; /* # real pages used for text */
_T_LONG_T pst_ssize; /* # real pages used for stack */
_T_LONG_T pst_rssize; /* resident set size for process (private pages) */
_T_LONG_T pst_shmsize; /* # real pages used for shared memory */
_T_LONG_T pst_mmsize; /* # real pages used for memory mapped files */
_T_LONG_T pst_usize; /* # real pages used for U-Area & K-Stack */
_T_LONG_T pst_iosize; /* # real pages used for I/O device mapping */
_T_LONG_T pst_vtsize; /* # virtual pages used for text */
_T_LONG_T pst_vdsize; /* # virtual pages used for data */
_T_LONG_T pst_vssize; /* # virtual pages used for stack */
_T_LONG_T pst_vshmsize; /* # virtual pages used for shared memory */
_T_LONG_T pst_vmmsize; /* # virtual pages used for mem-mapped files */
_T_LONG_T pst_vusize; /* # virtual pages used for U-Area & K-Stack */
_T_LONG_T pst_viosize; /* # virtual pages used for I/O dev mapping */
_T_LONG_T pst_maxrss; /* highwater mark for proc resident set size */
_T_LONG_T pst_text_size; /* Page size used for text objects. */
_T_LONG_T pst_data_size; /* Page size used for data objects. */

[Ralph Grothe]

I also seek explanation of the flags member.

To cite from ps's manpage


flags Flags (octal and additive) associated with the
process:

0 Swapped
1 In core
2 System process
4 Locked in core (e.g., for physical I/O)
10 Being traced by another process
20 Another tracing flag

The default heading for this column is F.


Here an excerpt from oracle's processes


# UNIX95= ps -u oracle -o comm,sz,flags|head -5
COMMAND SZ F
oracleLOLA 14789 1001
oracleLOLA 14878 1001
oracleLOLA 14750 1001
oracleLOLA 0 1001


Doing base 10 conversion the 01001 values from the F column equal 513.
But if I sum up the possibile values from the above manpage comment I at the most arrive at 37 (decimal) or 31 (octal).
What does 513 or 01001 mean?

Little comment in the struct member definiton of __pst_stat

_T_LONG_T pst_flag; /* flags associated with process */

and a little farther down they even changed to base 16 when defining these macros

/*
* Process flag bits for pst_flag
*/
#define PS_INCORE 0x1 /* this process is in memory */
#define PS_SYS 0x2 /* this process is a system process */
#define PS_LOCKED 0x4 /* this process is locked in memory */
#define PS_TRACE 0x8 /* this process is being traced */
#define PS_TRACE2 0x10 /* this traced process has been waited for */
#define PS_TXTLOCKED 0x20 /* this process' text is locked in memory*/
#define PS_DATLOCKED 0x40 /* this process' data is locked in memory*/
#define PS_SYSCALLTRACE 0x80 /* per-process syscall tracing enabled */
#define PS_SWLAZY 0x100 /* process has associated Lazy Swap region(s) */
#define PS_64ASL 0x200 /* process has 64-bit address space layout */
#define PS_CHANGEDPRIV 0x400 /* process was or is privileged */


But anyway, maybe it's sufficient for me to check if the flag is 0 in order to find out if the process has been swapped out.

[Ralph Grothe]

Here a summation done in the Perl debugger with the scriplet attached.
I wonder how to interpret the numbers for the 811 procs during the __pst_stat call.

DB<1> c 28
main::(pstat.pl:28): 1;
DB<2> x \%pst_stat_mem
0 HASH(0x8000000100309608)
'dsize' => 311839
'mmsize' => 1074415
'rssize' => 887354
'shmsize' => 1320968586
'ssize' => 81911
'tsize' => 11497780
'vdsize' => 427424
'vmmsize' => 3420918
'vshmsize' => 1337469570
'vssize' => 114352
'vtsize' => 13271497
'vusize' => 6488
DB<3> p scalar @$user_procs
811

[James Murtagh]

Hi Ralph,

When a process gets created and as it maps other areas into its address space it creates pregions or contiguous virtual address maps. There will be pregions for its text, data, stack segments etc. As the pregions must contain contiguous virtual addresses the full size of the memory oject must be mapped initially. The "real" pages are those in its pregion that are in-core or active in memory, the "virtual" pages is the total of the virtual address space allocated for that region. As the kernel brings pages in on demand and has to write back pages either to a backing store (swap) or front strore (the file itself in the case of mmap'd files) there may be some difference in the totals. The memory management white paper explains all this in glorious detail.

For the ps flags output you need to do your base 10 conversion to base 16 :

# echo 0o1001=X|adb
201

So this will map quite nicely into the flag definitions from the pstat output. I'm really hoping your version of Oracle is 64 bit! :-)

Cheers,

James.

[Ralph Grothe]

James,

thanks for the explanation, but I still fail on basic page arithmetics.

E.g. from Perl debugger session, where I summed up all sizes in a hashref $consumption over all processes run by oracle.

DB<9> !5
x $consumption
0 HASH(0x80000001004c06c0)
'dsize' => 340313
'mmsize' => 1148734
'rssize' => 940969
'shmsize' => 1434618003
'ssize' => 86951
'tsize' => 12486964
'vdsize' => 458656
'vmmsize' => 3660307
'vshmsize' => 1438192587
'vssize' => 121248
'vtsize' => 14270921
'vusize' => 6976
DB<10> p 1434618003/(940969-340313-86951-12486964)
-119.818505805312

For the supposed units (either mem pages or KB) I refer to cooments in pm_stat_body.h
There they only talk about pages.

When I read the metric explanation for the so called Resident Set Size (RSS) from glance's help menu it ist explained as such:

RSS = Stack + Data + Text + Shared/Refs

As you can see from command line # 10 in debugger above, I tried to verify this calculation by isolating the Nos. of references to shared memory segments.

So expanding for Refs would yield

Refs = Shared / (RSS - Stack - Data - Text)

As you can see above this yields negative refs which is crap.

So what the hack is the damned __pst_status struct holding?

[James Murtagh]

The RSS is the size of all private regions of memory in that if the process was to terminate this would be the size of memory freed, hence termed the resident set size of the process. This includes the data, stack, uarea, private mmap and possibly private text. Unfortunately you can only get the total mmap data from pstat, you need to look at the process memory regions in glance and sum up all the private areas. So your formula would be more like :

RSS = data + stack + uarea + private-mmap

So although only the virtual uarea size if listed it works out that approx 506749 pages are used for private mmap.

I can't think offhand of a programmatic way to find if the mmap regions are private or not....you could use q4 but this would be a very long task unless you could code a perl script for it. I'll have a look and see what I can find out.

- jm.