swap space full

[Khairy]

hi all,

need some guidance from all the gurus out there. I'm running out of idea on how to solve this.

i have a gs140 server with 14 x 700mhz w/ 4GB memory and 8GB swap space.It runs TRU64 v4.0F

My application team will executes routine job which process data files. That count 8-16 jobs daily.

Things which puzzle me, the servers suddenly consumes all the available swapspace and server stop responding.
I went to check server during one midnight and still able to login at the main console.

But application team complain they can't login (telnet). I tested and confirmed telnet is useless.
I think swap space is the main cause. An output from `swapon -s`.

Swap partition /dev/vol/rootdg/swapvol:
Allocated space: 1048512 pages (8191MB)
In-use space: 1048497 pages ( 99%)
Free space: 15 pages ( 0%)

Swap partition /dev/vol/rootdg/swapvol1:
Allocated space: 1048512 pages (8191MB)
In-use space: 1048506 pages ( 99%)
Free space: 6 pages ( 0%)


Total swap allocation:
Allocated space: 2097024 pages (16383MB)
In-use space: 2097003 pages ( 99%)
Available space: 21 pages ( 0%)

As a result of this, I add 8GB more of swap space. Now the server has 16GB of swapspace.

But it happened again. This time, i refer System configuration and tuning guide manual.

I execute the following. Pls refer attached file.

Can anyone guide what cause of this. Do i need to tune any parameters.

I know this is an indication of lack of memory space but this doesnt happen in other server which has the same specs and memory.

Thanks in advance.

Rgds

[Khairy]

hi,

these are the sysconfigtab parameters.

lsm:
lsm_rootdev_is_volume = 1
lsm_swapdev_is_volume = 1

ipc:
msg-max=32768
msg-mnb=65535
msg-mni=128
msg-tql=256
shm-max=2139095040
shm-mni=1024
shm-seg=512
sem-mni=512
sem-msl=512
sem-opm=512
sem-ume=512
num-of-sems=512
max-kernel-ports=16739
port-hash-max-num=8381950
port-reserved-max-num=16739
set-max-num=49413

proc:
max-proc-per-user=2048
max-threads-per-user=4096
per-proc-stack-size=2147483648
max-per-proc-stack-size=21447483648
per-proc-data-size=68719476736
max-per-proc-data-size=68719476736
max-per-proc-address-space=68719476736
per-proc-address-space=68719476736
maxusers=2048
task-max=16405
thread-max=32898
num-wait-queues=1024
num-timeout-hash-queues=1024

vm:
ubc-minpercent=5
ubc-maxpercent=10
ubc-borrowpercent=10
vm-maxvas=16106127360
vm-vpagemax=131072


I modified all parameters to the following today:

> per-proc-stack-size=4294967296
> max-per-proc-stack-size=4294967296
> per-proc-data-size=4294967296
> max-per-proc-data-size=4294967296
> max-per-proc-address-space=4294967296
> per-proc-address-space=4294967296
> ubc-minpercent=10
> ubc-maxpercent=50
> ubc-borrowpercent=20
> vm-maxvas=4294967296
> vm-vpagemax=1966080

After applied the parameters.This time its getting worse, while accessing the server via telnet, it hangs!

I had to access the server from main console, copy the original sysconfigtab file and reboot the server straight away.

When the server startup, app team submit same process. It hangs this time but for only for a few seconds. Then it resumes normally.

[Kapil Jha]

Well I have never worked on V4 , what i would have done is started the syatem and I'll monitor it when only OS is running no aplication.
And if its OK then there must be some problem with application and will try to see the problem from service end.
Can you or ask application tem to check what is happening in application when it is starting , was that aplication running fine before??
BR,
Kapil

[Pieter 't Hart]

adding swapspace is no solution!
this only results in the system becomming even slower beacuse of excessive swapping from memory to disk and back.
allocating more swapspace then 1,5 a 2 times the physical memory has only advantage when there are a lot of "sleeping" processes.
if you run a process wich really needs this much memory you should add physical memory.

this to investigate are processes allocating exessive memory.
- a single process allocating lots of memory
- too many processes running at the same time.

Most common cause i've run into is a periodically scheduled job (or one of its child's), that creates a temporary file with non-unique filename.
when the first instance still runs when the seccond instance starts, they want to access the same file.
This results in problems like
- locking
- one of them processing the wrong data (file from other instance).
as result they both run when a third instance starts etc...

when reducing the swap-space instead of increasing , the faulty job will run into problems earlier wich may help in locating the fault.

a command like this command might help:
ps -e -o vsz,pcpu,pid,rss,majflt,minflt,pmem,command|grep -v COMMAND|sort -nr|head
(vsz, pid and command could be enough).

beware! the sort order is not as we really want it.

[Martin Moore]

Pieter is absolutely correct. The workload on your system is simply consuming too much memory. Your swapon output indicates that you're running in lazy (deferred) swap allocation mode. So your 4 GB of physical memory is getting filled up, then as the demand for memory continues, processes start swapping out until you fill up all of your swap space.

Tuning the system is not going to help much. You need to add more physical memory or reduce the workload or both. Adding more physical memory will probably do wonders for your system performance as well.

Martin

[Rob Leadbeater]

Hi,

The process list from your first attachment shows that there are many processes all trying to grab massive amounts of memory.

For example:

adm 2792 0.0 2.4 3.46G 98M ?? S Jun 14 7-15:28:10 /runss17/
adm/MSP4.7/corp/bin_d/streaming Generic/generic_xgsm2


That's one process of many, that looks like it's trying to grab 3.46GB of memory.

You need to ask the application guys why this and similar processes are trying to use so much space. You might alleviate the problem by adding more physical memory, but I would suggest there is probably a serious issue with the application somewhere...

Cheers,

Rob

[John Manger · Accepted solution]

The process listing shows a number of processes with a Virtual (VSZ) size of 3.4G, and a resident size (RSS) of 80-91M. The available Free Memory shown in the vmstat output (>30K pages) indicates there is still plenty of free memory.

You appear to be running with 'eager' swap allocation, which pre-allocates more or less the full VSZ of each running process from the available swap space. That is why you are running out of swap..... because of the numerous 3G+ VSZs.

You might consider running with lazy swap allocation IF your current process load is fairly static.

[Pieter 't Hart]

I agree with John.
cause of the long text in the first two posts i missed the attachment.

[Martin Moore]

It's definitely lazy swap already. If it were eager swap, there would be a "Reserved space" line in the last section of the swapon output.

But I agree, the basic idea is that the application appears to be trying to hog too much memory.

Martin

[Hein van den Heuvel]

proc:
per-proc-stack-size=2147483648
max-per-proc-stack-size=21447483648

I suspect it is a consequence of the very high stack size settings? WHY is that made so high at 2GB? That is about 1000 times larger than many applications require.
That would certainly explain the 3GB + VSZ's

Setting it to 4G as you did, 'clearly' (to me :-) will make matters worse. try 1GB!

Check out vmstat -P
I suspect it is consuming very many ptepages.

And check out the lzay vs eager spaw comments and concerns.

hth,
Hein