Calculating Memory Use with Awk

[Vassilios]

I'm having a problem calculating memory usuage in my HP server.

If you to a ps -efl it outputs everything including the "SZ" column which is the size of the memory being used by that process. Do you agree?

First, I'd like to know when i do a ps -efl, what are the units of the SZ column? Is it in KB, MB?? it doesn't say. Remember at school, whenever we display information, we have to give the units for all information. We go into industry and produce products, and we forget these very simple rules. And then the man pages on the ps command don't mention them either.

Well. If i do a:

ps -ef | awk '{totalsize+=$10};END{print totalsize}'

it prints the total size it added up from column 10 (the SZ column).

But...
when i do a top command, the total memory being used HAS NOTHING in common with what my awk script produced.

Is it a problem with units? Am I not comparing oranges with oranges??

[Don Morris_1]

Roughly -- resident set size (SZ) has some approximations to try to account for shared objects across the affiliated processes instead of inflating the figures. But close enough, yes.

man ps tells you: "sz - The size in physical pages of the core image of the process, including text, data, and stack space. Physical page size is defined by _SC_PAGE_SIZE in the header file (see sysconf(2) and unistd(5))."

Considering I see that in the docs.hp.com version going back to 10.20 -- I don't think this is a recent addition.

So... your awk script is adding up pages. Top by default (and again, the man page tells you besides the units in the output) outputs in Kb. So you need to multiply the result of your awk by the result of getconf _SC_PAGE_SIZE [to get bytes] / 1024 [to get Kb].

[Vassilios]

Hello, thanks for your answer.

So, what do I need to add in my awk script to compute the fact that it is the number of pages being given. I want to transfer the answer to Megabytes.

Thanks

[Dennis Handly]

>what do I need to add in my awk script to compute the fact that it is the number of pages being given. I want to transfer the answer to Megabytes.

print totalsize * 4 * 1024 / (1024 * 1024)

[TTr]

The awk program is using 32-bit arithmetic so the suming in awk overflows at 2147483647. (2^31 - 1). It does not error out, it keeps adding but the sum does not change once you reach the above number. At least it did it on my 11.11 workstation. I did not check if there are any patches for awk.

An alternative to adding with awk would be to use bc or dc. I did it using a "while read" loop, I don't know how to do it with inline bc commands. Maybe a bc expert can offer a quicker solution.

I also added a comment to your other posting about "ls -l | awk"

[James R. Ferguson]

Hi Vassilos:

As TTr wrote, "The awk program is using 32-bit arithmetic so the suming in awk overflows at 2147483647. (2^31 - 1)."

...and for that reason, you could use Perl instead of 'awk':

# awk 'BEGIN{printf "%d %d\n",(2**31-1),((2**31-1)+10)}'
2147483647 2147483647
# perl -e 'BEGIN{printf "%d %d\n",(2**31-1),((2**31-1)+10)}'
2147483647 2147483657

For this posting:

# ps -ef | perl -nale '$totalsize+=$F[9];END{printf "%10d\n",($totalsize*4*1024/(1024*1024))}'

Notice that Perl counts from zero whereas 'awk' counts from one. Hence the 9th field in Perl is the tenth in 'awk'.

Regards!

...JRF...

[James R. Ferguson]

Hi (again):

By the way. I think you meant to add the 'l' switch to your 'ps':

ps -elf | perl -nale '$totalsize+=$F[9];END{printf "%10d\n",($totalsize*4*1024/(1024*1024))}'

Regards!

...JRF...

[Dennis Handly]

>TTr: The awk program is using 32-bit arithmetic so the summing in awk overflows at 2147483647.

Where did you get this idea? It uses a C double.
But you could just use:
print totalsize * 4 / 1024

Or you may have to use printf with a %f format.

[Ralph Grothe]

Hi Vassilios,

only a negligible "improvement" which could ease your parsing and arithmetics.
As you are only interested in the SZ column why not simplifying the ps part of the pipe like

$ UNIX95= ps -e -o sz= | awk ...

or even

$ UNIX95= ps -e -o vsz= | awk ...

which already outputs memory in KB rather than pages (see man ps).

[TTr]

> Dennis: Where did you get this idea?

It was not an idea. I used the awk statement from above code and it stopped counting at the above number. Maybe I should have said "It appears that awk is using 32-bit arithmetic...". As I said in my posting it was on 11.11. I just did a "what" on awk and I have PHCO_33916 on it.

I also found this in PHCO_36161 which is exactly what I experienced and is caused by INT_MAX.

<--snip-->
( SR:8606454418 CR:JAGag11066 )
According to UNIX95 specification, awk(1) should use %d as the internal conversion format for a numeric value that is exactly equal to the value of an integer. Any other numeric value should be converted to a string with the value of the variable CONVFMT as the fmt argument. The defect can be seen as:
$ print 2147483648 | UNIX95=1 awk '{ print $1-0 }'
O/p : 2147483647 (This is wrong o/p)
Here %d is used as the format specifier even if the number to be converted is non-integer (greater than INT_MAX).

Resolution: awk(1) is modified to comply with the UNIX95 standards. The above example gives the following output with fix:
$ print 2147483648 | UNIX95=1 awk '{ print $1-0 }'
2.14748e+09

<---end snip--->

[Don Morris_1]

Note that VSZ and SZ aren't at all the same thing. One (VSZ) is the virtual object size, the other is actual physical memory consumption. [If you malloc 500 pages but only touch 1, VSZ would be 500 * page size / 1024, SZ would be 1 (allowing for variable page size heuristic hints, etc.)].

Again -- I don't know enough awk (sorry, I think in C not shell scripts) to know exactly how to work it in... but I can't recommend hard-coding the expectation that the system page size is 4kb in scripts. The page size is tunable with 11.31.0809 and I'd expect it to remain so in the future. Your scripts doubtless work fine now -- but when and if you encounter a system with 16kb or 64kb pages, you will get the wrong answer. "getconf _SC_PAGE_SIZE" will always give you the right factor. (Divide that by 1024 first if you want Kb and are worried about overflows -- but you'd still run into problems on a 2Tb or beyond system, right? Assuming signed 32-bit math and all).

[Dennis Handly]

>TTr: I used the awk statement from above code and it stopped counting at the above number. Maybe I should have said "It appears that awk is using 32-bit arithmetic...".

Ah, I used a giant number and had no problems:
awk 'BEGIN {print 2000000000 + 2000000000 + 2000000000 }' /dev/null
6e+09
Unfortunately this format gives a bogus answer:
printf "%lld\n"
4612175268889493503

So you would need: printf "%.0f\n"
6000000000