swap utilization is high

[Hanry Zhou]

N class server with 6gb memory, and following is the output of swapinfo -tam:
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1024 0 1024 0% 0 - 1 /dev/vg00/lvol2
reserve - 613 -613
memory 4448 2905 1543 65%
total 5472 3518 1954 64% - 0 -
total 5472 3518 1954 64% - 0 -

The total of swap util seems to be 64%(so in glance), and 1024mb swap device is configured.

Here is the problem:
sometime, the system response time is slow, and sometime the ITO warning messages saying SWAP utilization reaches 90% high.

My questions are:
1. is there definition difference on swap util. between ITO and swapinfo command.

2. the swap space is configured too small? If it is too small, how come only 0% of device swap used since the system last time boot up?
The is the part troubling me most.

3. currently the total swap space is about 6gb (5gb pesuo memory + 1g device swap)which is 1:1 to physical memory. so, the swap spacw should be okay?

Thanks,

[Steven E. Protter]

If swap utilization is hitting 90% you are probably either overloading the system or have inadequate swap space. You may wish to measure swap usage over tiem(utility attached).

1) there are small calculation differences between the various utilities for showing swap.

2) Swap must be a minimum of half of RAM. It is generally set to between 1.5 and 2.0 times RAM. If analysis shows you need more than this, your system is being overloaded and adding swap will simply slow it down.

3) See my answer to item 2.

Summary: Swap is too small, based on my experience.

SEP

[Hanry Zhou]

How come the usage of device swap is 0% if I don't have enough swap sapce configured? I would expect the system would also swap processes over to the device swap, if the space of pseudo memory is not large enough for swapping?

[Ravi_8]

Hi,

the swap seems to be very less(6 GB physical and 1GB swap) atleast make swap half of physical memory.

[Hanry Zhou]

Ravi,

We have also more than 5gb pseudo memory swap space, the the total is equal to the 6g memory. Isn't?

[Hanry Zhou]

When we say that the swap space should be 1.5, 2, or whatever times of the size of Memory, does the swap space include Pseudo memory swap space?

In my case, I have >5gb psedo memory swap space, and 1gb device swap.

[Patrick Wallek]

According to your swapinfo output, you are not paging out yet. I think that ITO is reading the "memory" line and getting confused.

As long as you do not actually see the "dev" percentage go above 0, you are OK.

[Hanry Zhou]

Patrick,

YOu seems right, but how come the system response time is slow? there are not too many cpu activities, and i/o is not too intenisive either.

Can anybody please tell me if pseudo memory swap is part of swap space? and should be counted as part of 2 times of physical memory
?

[Ted Buis]

No question in my mind you need to increase device swap to at least equal memory and more if you are getting to 90% as you indicated. Swapmem_on=1 is only good if, as in your present case, you have more RAM than device swap. Good discussion in the book, "HP-UX 11i Internals" by Chris Cooper and Chris Moore. You may be experiencing inconsistent response because of pageouts, but you have 5GB that is locked in RAM due to being used a pseudo-swap which really means that it is bound to RAM. That's fast for those application that happen to be in that RAM area (pure luck), but means that the remained of your applications in areas that can be paged out will get hit repeatedly when real memory is used up. Disk space is cheap, always have as much device swap as RAM.

[Sridhar Bhaskarla]

Hi Hanry,

There are couple of things I would check if the system is responding slow. First the buffer cache. If it is set to 50% which is default and if it is all used, then the system might experience intermittent freezes. If it is at 50%, reduce it to around 5-10%. The other thing is to look for the network configuration. If it is on a 100MB network, make sure your lan card as well the network switch port are set to 100 Full Duplex - Auto Off.

May be there is some application process or something that is running intermettingly causing the swap to go upto 90%.

About your swap calculation, though you are running ok at this time, it is not a good idea to configure the device swap less than the size of memory. This is because, as the memory keeps getting locked, the used (or not available for reservation) column in the memory line will increase. At some point of time that may make the system to depend more and more on the device swap for reservation.

Also ensure that kernel parameters like nfile, nproc, ninode etc., are not configured very high than required.

-Sri

[Hanry Zhou]

Sri,

1. Buffer cache settings is okay: 460mb among 6gm RAM.

2. It(system slow) happens very often, so it is not itermittent. 90% is based on ITO, but 64% total based on swapinfo -tam

3. network is fine.

4. parameter is okay, not error/complain about these kernel parameters.

MY questions is since device swap is used 0%, Do I really have to increase the device swap space, and why?

[malay boy]

Harry,
If you swap space utlization is 0 and I don't think you need to increase it.You might be worried if you swap utilisation is 90% above.

my 2 cents.

regargs
mB

[Ted Buis]

The reason you need device swap is the way that HP-UX does paging. It reserves page-out space at the time the application is launched. If sufficient space cannot be reserved, then the application will not run. Psuedo-swap basically, locks pages in memory so that they will not be swapped and so don't need to reserve device swap. This is done so that applications don't start and then die in the middle of something when they have to pageout but there is then no place to put them. The idea is that it is better for them to not run at all if there is a chance they will die from lack of this resource. The book on "HP-UX 11i Internals" by Chris Cooper covers this nicely in Chapter 7.

[Hanry Zhou]

Ted,

thanks for your messages.

I believe what you are saying really got the point.

However, based on the result of swapinfo -tam, the device swap is just 0% used, so, can I conclude that I don't have the swap space pressure? I understand that if the process can not reserve the certain amount of swap space, it will not be launched, and an error message will come out. but in my case, since i don't see any error messages when the system is slow, so I can conclude the problem on this system is now swap space related.

Please advice.

[Sridhar Bhaskarla]

Hanry,

There are two things to swap utilization. Utilization due to reservation and utilization due to paging. Second one is what you don't want and that's not happening on your system. But that doesn't mean you don't need swap space. Pseudoswap covers you for 75% of the swap. So, one would think that you need only 25% of the device swap. However, as the system is used, more and more pages may get locked into memory which will decrease the amount of memory available for reservation. So, at one point of time, depending on the memory usage, you may end up with insufficent swap for reservation as your device swap (which is static) that insures pseduoswap is low.

But that may not be a problem in your case.

I would ask as very basic question. When you said there were intermittent slow downs, are they related to the response to the commands or they really slowing down the system so that processing is delayed?

-Sri

[Ted Buis]

Sridhar answered it, there is a difference between allocated, reserved and free swap.
Allocated swap is "in use", and reserved has be specifically set aside for running program pages that have not yet been paged out. If you have no free swap, you can run any more programs, as they cannot reserve any space to which to page-out. So it is possible to have none or very little free swap, but to have never actually have used (allocated) any swap at all. Reserved swap hasn't been "used".

[Ted Buis]

Correction:
If you have no free swap, you can't run any more programs, as they cannot reserve any space to which to page-out.
Sorry, for the typo.

[Bill Hassell]

swapinfo reports what is happening at the moment. It is trivial to write a program that allocates 1000megs of RAM, and run this program 6 times to use up all memory and swap, then kill the proceses. swapinfo will not 'remember' any of that. The key to swap is whether it is really being used. ITO just looks at swap allocation once in a while and may have seen a spike. You need to monitor whether any page outs atre ocurring (vmstat). 0-9=ignore, 10-30=monitor, 40-90=beg for more memory, over 90=demand more memory.

Note that you may not actually run any programs are swapped out, instead, massive use of mempry mapped files may be causing the swap usage.

[Hanry Zhou]

Sri, Ted, and Bill, very well put.

Sir,
What is the difference between slowing to the response to the commands and slowing so that processing is dealyed?

In my case, I'm sure it was slow to my commands, like bdf..etc.
I'm not so sure about if the process was delayed.

Also, it was not intermittent, it last for quite long time.

So, even though my device swap is 0%, there is still the possibility of lacking of device swap space, therefore, I should add more device swap?

[Ted Buis]

If reserved swap gets over 90% (allocated may still be zero), you have less than 100MB of free swap (new virtual memory). If you have dynamic buffer cache, I think (hope) the system is going to want to try to shrink that to free RAM, burning some cycles in extra overhead. Anytime virtual memory is short, system overhead will increase. Along these lines, when a file system gets close to full, performance also drops.

[Bill Hassell]

bdf is a bad command to measure performance/ It puts a very heavy load (momentarily) on each mounted filesystem and thus will appear to be very slow when one or more disks are busy. du is a similar command. Response time can be measured in a number of ways:

1) the time to press a key until a login prompt appears
2) the time to get a new prompt line in a shell by pressing return
3) delays between lines of a long listing
4) total login time after the password is given
5) time needed to complete a network intensive task such as ftp
6) application response and wallclock time

The first 3, if visibly delayed, usually indicate a high system overhead (sar -u 1) which will take some detailed work to locate the cause. The rest indicate a load but not necessarily unexpected in a busy system.

[Hanry Zhou]

Sri,

I would ask as very basic question. When you said there were intermittent slow downs, are they related to the response to the commands or they really slowing down the system so that processing is delayed?

Answer: I'm sure the system was slow in responding commands ( including ls -l..)
But, I am not sure if they really slowing down the system, so that processing is delayed. What are differences between these two?

In my case, the used% is 0 on device swap, and also I don't see any error messages came up, so I don't think I have to increase the swap space. Because if I don't have enough swap space, and thefore it causes that applications could not be launched, I should then get the error messages complaning things like not able to launch applications.
Am I right?

[Alzhy]

Hanry,

If this box is a Database server (Oracle) and also has some middleware application on it (ie. uses a lot of Java) .. then your SWAP usage may very well be usage based on UTILIZATION -- not paging.

I would urge you to add another swap device with a higher priority than your existing 1GB swap. Also, you may consider adjusting your dbc_max_pct so your buffer cache allocation never exceeds 800Mb.

[Sridhar Bhaskarla]

Hanry,

If there is an NFS server not responding or slow to respond, then you would see your ls taking long time along with bdf. It doesn't mean the entire system is processing slow.

Can you post the outputs of the following commmands when you see the response as slow?.

sar 2 5
sar -d 2 5
sar -q 2 5
uptime
sar -v 2 5
vmstat 2 5
kmtune

-Sri

[Hanry Zhou]

Hi,

Sorry for bringing up the issue again.

Here is the result of swapinfo -tam from one of our servers we have in the shop, with phsical 4gb memory. The total swap util is very high, but i don't feel any performance problem while the swap util was hitting so high. And also at this moment, the device swap (only 1gb configured )is 0%.

My questions is, Should increase the size of the device swap?

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1024 0 1024 0% 0 - 1 /dev/vg00/lvol2
reserve - 1024 -1024
memory 2973 2082 891 98%
total 3997 3106 891 98% - 0 -