life, the universe, and vmstat

[Ken Scharpell]

vmstat is monitoring virtual (logical) memory which fluctuates very frequently and as its logical it doesnt necessarily mean its affecting your physical memory levels - so dont look too much into it. Only look for key indicators;

Yes, if the W column is non-zero you have processess completely swapped out which means you have considerable memory pressure - this is confirmed by pi and po also being non zero. In an ideal world pi and po and w should all be zero which means physical memory is sufficient to run all current processes without paging/swapping.

A better command to use to see how physical memory is being used is swapinfo -mt

Yes, your sar stats seem to show c0t6d0 (boot disk?) running flat out - but run sar for sometime and over multiple days to confirm this is not a one-off. Get some better averaged stats over a longer time before confirming that disk is overloaded and you need to move some files/disks around. In your case striping over multple disks will help to balance the %busy load more evenly over all your available disks.

[John Palmer]

Hi,

My first thought on this system is that you've not got enough RAM and it's spending most of its time paging. Disk c0t6d0 is indeed a bottleneck but only because it's where all the virtual memory is being paged to/from.

It looks like a classic example of a disk appearing to be a bottleneck whereas in actual fact the cause is a lack of memory.

Your best options are to:-
add memory or
reduce the workload.

You could also spread the paging I/O by adding swap volumes on the other disks but the benefits are likely to be minimal.

Hope this is of some assistance.

Regards,
John

[John Palmer]

A quick indicator of how much memory pressure a system is under is to check the amount of cpu time that the process 'vhand' is using.

[Andrew Brain or Yong Ji]

Thanks for the quick answers guys but I must clarify that I'm also after some hard global definitions of what vmstat's stats really mean (not just my results).

Also I should clarify that this system isn't always like this. The problem observed seems to happen randomly about once or twice a week. However some slight paging activity does occur normally which of course increases as demand increases.

So I do agree that the system is under some memory pressure in any case but the problem observed here is rather severe as the system basically hangs when it happens.

There's really only two applications we run on this server being "Netscape Proxy" and a library application called "dbc". Both are rather heavy on the memory resident side. As stated earlier Netscape Proxy may use memory mapping which may have something to do with this problem. Frankly I intend to move the Proxy off this server which I think has a good chance of fixing the problem for good.

Total memory is 160MB of which 130MB is availiable at boot. "Top" analysed gives a total resident memory of 55316K which strangely corresponds to top's given real memory of 55316K though it says 2432K is free (This is probably just a coincedence). Bufpages (from adb) add up to 46MB. Shared memory (ipcs) takes another 2MB. However that leaves 30MB un-accounted for. Any clue where this is used?

[Ken Scharpell]

Use swapinfo -mt to get a much better breakdown of memory utilisation. If you post its output we can explain it.

If your boot disk is c0t6d0 and you have primary swap on this disk also then that explains why your server appears to be 100% busy on that disk and appears to grind to a halt occasionally (as its quickly trying to swap in/out processes and this takes priority over all other processes!)

For a detailed explanation of vmstat the best ive seen is in an oreilelys book, cant remember which on exactly, NIS & NFS ?

bufpages should be dynamic, so the value from the kernel via adb is at bootup, as pressure increases this should reduce, as in your case. You need glance to see current level.

[Paul Davies]

Hi

For a fairly good book on performance tuning look at O'Reillys System Performance Tuning. ISBN 0-937175-60-9. This covers monitoring and vmstat very well.

Have you also looked at the ps command (eg ps avx on AIX) which will show you the exact real and virtual memory used by each process.

Hope this helps,


Paul.

[Andrew Brain or Yong Ji]

Additional info from Andrew Bruin@Warringah Council

NB Note my profile has been changed to reflect who I am rather than who I work for just in case your confused.

Here's some more info taken at the same time as my previous info. As noted vhand and swapper run times are rather active considering the system has been only up 15 days at this stage (20% of system run time). One thing that gets me if the swapper is only responsible for deactivating processes how come it has run more than vhand. A gather that if the swapper runs at all the system has serious problems or am I missing something here?

UID PID PPID C STIME TTY TIME COMMAND
root 0 0 0 Sep 28 ? 63:57 swapper
root 1 0 0 Sep 28 ? 0:01 init
root 2 0 255 Sep 28 ? 56:52 vhand
root 3 0 0 Sep 28 ? 41:01 statdaemon
root 4 0 0 Sep 28 ? 1:10 unhashdaemon
root 7 0 0 Sep 28 ? 37:21 ttisr
root 56 0 0 Sep 28 ? 11:56 nvsisr

Also here's the swapinfo information taken at the same time as the previous info. Strangely enough it doesn't seem to vary much over the period in question even though there are lots of page outs. Am I missing something here too?

Fri Oct 13 16:55:47 EDT 2000
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 327680 76104 251576 23% 0 - 1 /dev/vg00/lvol2
reserve - 171240 -171240
memory 113408 113408 0 100%
Fri Oct 13 16:55:55 EDT 2000
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 327680 76320 251360 23% 0 - 1 /dev/vg00/lvol2
reserve - 171364 -171364
memory 113408 113408 0 100%
Fri Oct 13 16:56:15 EDT 2000
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 327680 76588 251092 23% 0 - 1 /dev/vg00/lvol2
reserve - 171076 -171076
memory 113408 113408 0 100%

Anyway thanks for all the info so far. It hasn't been all that I asked for but I know I have asked for a lot.

[Andrew Brain or Yong Ji]

Earlier I posted the following

>Total memory is 160MB of which 130MB is availiable at boot. "Top" analysed gives a total resident memory of 55316K which strangely corresponds to top's given real memory of 55316K though it says 2432K is free (This is probably just a coincedence). Bufpages (from adb) add up to 46MB. Shared memory (ipcs) takes another 2MB. However that leaves 30MB un-accounted for. Any clue where this is used?

The reason top displays 55316K as given real memory and that all the resident memory usage (RES)adds up to the same is no coincedence. What top is displaying is exactly that. ie The total resident sizes of all the the processes combined (The sum of real memory used by all the processes).

However I'm still missing 30MB. Where is it used?

[Ansett Australia Unix]

Andrew,

Not sure where your missing memory has gone, but if you have access to "glance" or the graphical version GlancePlus, downloadable from HP it will be able to tell you exactly where all of your memory has been allocated. - jim.