Simpler Navigation for Servers and Operating Systems
Completed: a much simpler Servers and Operating Systems section of the Community. We combined many of the older boards, so you won't have to click through so many levels to get at the information you need. Check the consolidated boards here as many sub-forums are now single boards.
Operating System - Tru64 Unix
cancel
Showing results for 
Search instead for 
Did you mean: 

sys_exec / memtest / mixed memory models / excessive page-in

Diederik de Groot
Occasional Advisor

sys_exec / memtest / mixed memory models / excessive page-in

Hi Guys,

We bought a secondhand ES-40 Model 2, with 8 Gb of memory and 4 CPU's and installed V5.1B-4. After the complete installation and setting sysconfigtab to swap_lazy=1. Sys_check mentioned that we were having "excessive page-in's". I started a little investigation and found that we had two different models of memory in the FRU list:

P00_mfgpro2:show fru
FRUname E Part# Serial# Model/Other Alias/Misc

SMB0.MMB0 00 54-25582-01.B04 AY92660910
SMB0.MMB0.DIM1 02 20-00ESA-08 006a1311 400 98
SMB0.MMB0.DIM2 02 20-00ESA-08 00551311 400 98
SMB0.MMB0.DIM3 00 20-00ESA-08 013f061c 200 98
SMB0.MMB0.DIM4 00 20-00ESA-08 01830616 200 98
SMB0.MMB1 00 54-25582-01.B04 AY92501881
SMB0.MMB1.DIM1 00 20-00ESA-08 007f1311 400 98
SMB0.MMB1.DIM2 00 20-00ESA-08 007e1311 400 98
SMB0.MMB1.DIM3 02 20-00ESA-08 014a061c 200 98
SMB0.MMB1.DIM4 02 20-00ESA-08 0120061c 200 98
SMB0.MMB2 00 54-25582-01.B04 AY92699910
SMB0.MMB2.DIM1 00 20-00ESA-08 00d80616 200 98
SMB0.MMB2.DIM2 00 20-00ESA-08 0160061c 200 98
SMB0.MMB2.DIM3 00 20-00ESA-08 00a7061c 200 98
SMB0.MMB2.DIM4 00 20-00ESA-08 00370616 200 98
SMB0.MMB3 00 54-25582-01.B04 AY92601020
SMB0.MMB3.DIM1 00 20-00ESA-08 01750616 200 98
SMB0.MMB3.DIM2 00 20-00ESA-08 00770616 200 98
SMB0.MMB3.DIM3 00 20-00ESA-08 00a2061c 200 98
SMB0.MMB3.DIM4 00 20-00ESA-08 00710616 200 98
<-- snip -->

To be honest i can't find anything on the internet about the difference between the models 200 and 400 and ran a "sys_exer &", with the following result:

P00_mfgpro2:show_status
ID Program Device Pass Hard/Soft Bytes Written Bytes Read
-------- ------------ ------------ ------ --------- ------------- -------------
00000001 idle system 0 0 0 0 0
00000061 memtest memory 26 0 0 22632464384 22632464384
00000064 memtest memory 26 0 0 22615687168 22615687168
00000066 memtest memory 26 0 0 22615687168 22615687168
000000bc memtest memory 6 0 0 21474836480 21474836480
000000c3 memtest memory 27 0 0 22615687168 22615687168
000000c4 memtest memory 8 0 0 22547775488 22547775488
00000109 exer_kid dra0.0.0.4.1 0 0 0 0 31856640
00000112 exer_kid dqa0.0.0.15. 0 0 0 0 8294400
0000027f nettest ewa0.0.0.1.0 758 0 0 1067264 1067264

Please see attached file for full log.

Now i started to worry a bit. While some of the memory regions were fast (00000061, 00000064, 00000066 and 000000c3) two memory regions (000000bc and 000000c4) where really slow. My Questions:
- I guess my page-ins could be a result of this. How can i correlate between memory regions and the actual DIMM's on the MMB's?
- Is the a simple method to find out which memory modules are giving me this poor performance?
- Have i inserted the memory modules in the correct locations (Memory array wise) ?
- Should i loose the Model 400 DIMM's and replace them with Model 200 ?

I hope someone out there can help me out.

Thanks in advance,

Diederik de Groot
dkgroot [a][t] talon [.] nl
15 REPLIES
Ivan Ferreira
Honored Contributor

Re: sys_exec / memtest / mixed memory models / excessive page-in

Kernel parameter should be swap_eager.

We have a lot of page-ins in our servers, we don't worry about that if we don't page out. You can try the suggestion of the Tuning Guide to try to minimize paging, but this did not help to use.


>> - I guess my page-ins could be a result of this.

I really doubt it.

Memory problems could be reported by dia or webes if you have installed.

Por que hacerlo dificil si es posible hacerlo facil? - Why do it the hard way, when you can do it the easy way?
Vladimir Fabecic
Honored Contributor

Re: sys_exec / memtest / mixed memory models / excessive page-in

There is no kernel parameter "swap_lazy".
It should be "swap_eager" like Ivan said.

I see no problem in your case.

Lot of page-ins?
This may be normal for some kinds of applications.

Can you post output of:
# sysconfig -q vm
In vino veritas, in VMS cluster
Diederik de Groot
Occasional Advisor

Re: sys_exec / memtest / mixed memory models / excessive page-in

Hi Ivan Ferreira,
Thanks for your reply. I should go back to swap eager once i have a decent enough drive to swap to. Currently swapping on the system-drive and that's going to be a bit of a problem. Plus that swap-partition is too small. That aside.

I still think it strange to say the least that a portion of my memory is so much slower that the rest. Can you think of any reason for this. The only possible conclusion I could come to is a broken or incompatible memory. The test don't show broken but they do show very high lag, maybe caused by incompatibility.

Do you concur.

Do you know of any way to correlate the memory test addresses to the actual memory banks ?

Regards and thanks for any help,

Diederik
Hein van den Heuvel
Honored Contributor

Re: sys_exec / memtest / mixed memory models / excessive page-in

Diederik,

Whoah! There seems to be an awful lot of confusion here!

[Klok horen luiden, maar weet niet waar de klepel hangt? ;-]

First of all there is NO relation between the 'flavor' of (hardware) memory and (software) swap activities. The only thing that defines swapping system usage and the quantity of memory, not the 'quality'.

Second, those multiple tests do not tests different regions, they are just different tests which take different times. Ig nore that!

Third, they only thing which does define memory speed (other than the clock/cpu-speed) is the interleaving level. The log porvided (thanks!) indicates this at level 4. That's great. That proves the memory is similar enough and should remove any concern about some FRU detail difference.

Fourth.. were you really going to consider spending money looks alone with no solid foundation back to the application and usage? Boy they must love you at the garage!
Are you sure you are dutch?
Ok... sorry... that was too much teasing perhaps.


Fifth, I like lazy swapping (but je must pick the right parameter name to get it).
I do not believe in overcommiting and requiring swap space in general. I control my applications to not use more, or only a little more, than the actuall memory available. Thust I have successfully run Tru64 setups with 64GB physical memory and 2Gb of swap 'just in case'.

Finally, do follow up on the "excessive page-in's", quantify that! Find out what rule triggered. Do your own observations with "vmstat 10 10" or similar. It could eb the normal and expected behaviour depending on the application. It coudl be a sign that the application is (erroneously?) fires up too many processes or some such.

Page-in is just a way to get stuff into memory to work on it. Page outs, as already indicated, are scary and suggest there is too little memory available.

Met vriendelijk groetjes,

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting


Diederik de Groot
Occasional Advisor

Re: sys_exec / memtest / mixed memory models / excessive page-in

Hi Vladimir,

thanks for your reply.

I know there is no such thing as swap_lazy and i didn't say that in my mail. I have swap_eager = 0 and vm-aggressive-swap = 0.

Here is my shortend "sysconfigdb -l".

generic:
version_banner = HP Tru64 UNIX
version_avendor = HP
version_vendor = Hewlett-Packard Company
memberid = 0
new_vers_high = 1445673072572143232
new_vers_low = 52560
version_product = Tru64 UNIX
login_name_max = 64

advfs:
AdvfsAccessMaxPercent = 35
AdvfsSyncMmapPages = 0

inet:
ipport_userreserved = 65535
tcbhashsize = 16384
tcbhashnum = 16
tcp_keepalive_default = 1
tcp_sendspace = 65535
tcp_recvspace = 65535

proc:
round_robin_switch_rate = 40
max-per-proc-address-space = 8589934592
max-per-proc-data-size = 8053063680
max-per-proc-stack-size = 536870912
per-proc-address-space = 8589934592
per-proc-data-size = 8053063680
per-proc-stack-size = 536870912
max-proc-per-user = 2048
max-threads-per-user = 4096
maxusers = 4096
sched-min-idle = 60
round_robin_switch_rate = 0
exec_disable_arg_limit = 1

rt:
aio_task_max_num = 8193

socket:
somaxconn = 65535
sominconn = 65535
sbcompress_threshold = 600

vfs:
bufcache = 1

sec:
acl_mode = enable

ipc:
shm-max = 4278190080
shm-min = 1
shm-mni = 2048
shm-seg = 512
sem-mni = 1364
sem-msl = 512
sem-opm = 100
sem-vmx = 32767
msg-tql = 1024
msg-max = 16384
ssm-threshold = 8388608

vm:
swapdevice = /dev/disk/dsk0b
new-wire-method = 1
vm-page-free-reserved = 20
vm-page-free-min = 20
vm-page-free-target = 2048
vm_page_prewrite_target = 64
vm_syncswapbuffers = 1024
vm-swap-eager = 0
vm-bigpg-enabled = 0
vm-bigpg-seg = 0
vm-bigpg-shm = 256
vm-bigpg-ssm = 256
vm-bigpg-stack = 64
vm-aggressive-swap = 0
vm-ubcseqstartpercent = 15
vm-ubcdirtypercent = 30
ubc-maxpercent = 20
ubc-minpercent = 5
ubc-borrowpercent = 10
vm_max_wrpgio_kluster = 65536
vm_max_rdpgio_kluster = 65536

Thanks,

Diederik
Rob Leadbeater
Honored Contributor

Re: sys_exec / memtest / mixed memory models / excessive page-in

> I know there is no such thing as swap_lazy
> and i didn't say that in my mail.

Err, yes you did...! Read your original post again !!

Cheers,

Rob



Diederik de Groot
Occasional Advisor

Re: sys_exec / memtest / mixed memory models / excessive page-in

Woops you guys are right...

I Mean't set to swapping lazily in the original post.

But still that's not the issue. My memory seems to be out of sync when i run memory tests... Can anyone verify that this is normal behaviour ?

Thanks,

Diederik
Rob Leadbeater
Honored Contributor

Re: sys_exec / memtest / mixed memory models / excessive page-in

Yes it is normal. I saw the same thing when testing memory on an ES40 and ES45 recently.

Cheers,

Rob
Hein van den Heuvel
Honored Contributor

Re: sys_exec / memtest / mixed memory models / excessive page-in

>> My memory seems to be out of sync when i run memory tests... Can anyone verify that this is normal behaviour ?

Why do you say that?

How do you conclude that?

Just the model# difference doesn't mean much.

The 'id' column in the show_status is the number of the test, not an indication of a particular chunk of memory.

You would have to do pretty advanced (silly!) stuff to actually try to test memory response time differences for specific pieces of memory.
You can only expect differences in RAD/cell based system, which the ES40 is not.

And if performance is really critical, then why bother with the old & slow ES40 ? Try to get an ES45 at least.

Btw... if I wanted to test individual memory asa simple c-hacker, then I would probably allocate a bunch of GH memory and map that in chunks.

Cheers,
Hein.
Diederik de Groot
Occasional Advisor

Re: sys_exec / memtest / mixed memory models / excessive page-in

Hi Rob en Hein,

Thanks for your answers. I feel a bit better now about the machine we bought. Thanks a lot. I just thought that output was something to worry about and might maybe explain the high numbers of page-in's i was getting. Compared to our other ES-40 (running OSF 4.0f).

Now i will have to hunt for a nice drive to put my swapfile on and switch back to swap_lazy=1 i guess.

If i know we have enough memory to run all the applications in, why would i want to reserve space in the swap file and have disk I/O spent on the reservation (I know it's a-synchronous). And how would this solve my page-in's ,i don't understand quite yet.

Regards,

Diederik de Groot
Diederik de Groot
Occasional Advisor

Re: sys_exec / memtest / mixed memory models / excessive page-in

Hi Hein,

Read your full post again, thanks for the excellent debunking of my errors.

No more worries about my memory modelnumbers and sys_exer memtests.

I like lazy swapping as well and don't really see the need to switch over to eager either.

Funny thing is, at the moment i'm not running any non HP Tru64 OS Applications. Just the OS... and still i get hi numbers of In-Pages with the machine really doing anything. sys_check told me to check it out and i did. I know sys_check doesn't know everything, but just in case i compared it to the other ES-40 we have and it our current production machine generating less page-in under full strain.

We would like to switch over to this machine and i'm trying to be rigorous in testing and checking before taking it into production.

Having 128000 page-in after rebooting the machine 20 minutes ago was worrying to me.

At the moment things have quieted down a bit on this machine and i see less page-in's at the moment.

Running advfs defragment really put's the page faults into overdrive though... Still a little miffed about that one... I'll debunk that later i guess...

It was just something i wanted to check out and run against you guys.

It also gave me the assurance that when i need any help in the future that posting a message here will actually create a response.

Next time i'll try not to ask such "stupid" questions again. Pretty hard to find out though if your own question is "stupid" until you actually ask it. Nastiest problem with asking" stupid" questions, is that this thread will remain here on the internet until the end of time of course.

Thanks you guys,

Regards,

Diederik de Groot
Ivan Ferreira
Honored Contributor

Re: sys_exec / memtest / mixed memory models / excessive page-in

Hi Diederik, please support the community by assigning points. Please see:

http://forums1.itrc.hp.com/service/forums/helptips.do?#28
Por que hacerlo dificil si es posible hacerlo facil? - Why do it the hard way, when you can do it the easy way?
jim owens_1
Valued Contributor

Re: sys_exec / memtest / mixed memory models / excessive page-in

two additional things to keep in mind:

- you can not compare V4.0 and V5.1B statistics. Even comparing different PK levels on V5.1B can be misleading because there are accounting differences (i.e. counts that were wrong are now tracked correctly).

- sys_check is good for some things but notorious for being wrong about kernel tuning. It has gotten progressively more wrong since V5.0 and no effort will ever be made to fix sys_check.
Pieter 't Hart
Honored Contributor

Re: sys_exec / memtest / mixed memory models / excessive page-in

the system knows about memory in terms of dimms, banks(multiple dimms) and arrays (multiple banks).
I regrouped your memory info in some other way:

SMB0.MMB0.DIM1 02 20-00ESA-08 006a1311 400 98
SMB0.MMB1.DIM1 00 20-00ESA-08 007f1311 400 98
SMB0.MMB2.DIM1 00 20-00ESA-08 00d80616 200 98
SMB0.MMB3.DIM1 00 20-00ESA-08 01750616 200 98

SMB0.MMB0.DIM4 00 20-00ESA-08 01830616 200 98
SMB0.MMB1.DIM4 02 20-00ESA-08 0120061c 200 98
SMB0.MMB2.DIM4 00 20-00ESA-08 00370616 200 98
SMB0.MMB3.DIM4 00 20-00ESA-08 00710616 200 98

as i read your info about your memory i would suggest it may form an array with all 200's (dimm3+4) but not with mixed 200/400 (dimm1+2)
this could explain some performance effect.

as for the page-in's : sys_check itself does an enormous amount of page-in's so i question if this is really a problem (same as for the number of files in /tmp)
as long as you dont exessive page-out (as in Ivan's post)
Diederik de Groot
Occasional Advisor

Re: sys_exec / memtest / mixed memory models / excessive page-in

Hi Pieter,

I already moved the memory models around to make one array of model 400 modules just to make sure, they were mixed before. I checked out the documentation to figure out which module to please where.

Thanks a lot, i'll check again to see if i did it right.

According to the others it should not matter as i found out after the moving around.

Regards,

Diederik