cancel
Showing results for 
Search instead for 
Did you mean: 

swap space full

SOLVED
Go to solution
Khairy
Esteemed Contributor

swap space full

hi all,

need some guidance from all the gurus out there. I'm running out of idea on how to solve this.

i have a gs140 server with 14 x 700mhz w/ 4GB memory and 8GB swap space.It runs TRU64 v4.0F

My application team will executes routine job which process data files. That count 8-16 jobs daily.

Things which puzzle me, the servers suddenly consumes all the available swapspace and server stop responding.
I went to check server during one midnight and still able to login at the main console.

But application team complain they can't login (telnet). I tested and confirmed telnet is useless.
I think swap space is the main cause. An output from `swapon -s`.

Swap partition /dev/vol/rootdg/swapvol:
Allocated space: 1048512 pages (8191MB)
In-use space: 1048497 pages ( 99%)
Free space: 15 pages ( 0%)

Swap partition /dev/vol/rootdg/swapvol1:
Allocated space: 1048512 pages (8191MB)
In-use space: 1048506 pages ( 99%)
Free space: 6 pages ( 0%)


Total swap allocation:
Allocated space: 2097024 pages (16383MB)
In-use space: 2097003 pages ( 99%)
Available space: 21 pages ( 0%)

As a result of this, I add 8GB more of swap space. Now the server has 16GB of swapspace.

But it happened again. This time, i refer System configuration and tuning guide manual.

I execute the following. Pls refer attached file.

Can anyone guide what cause of this. Do i need to tune any parameters.

I know this is an indication of lack of memory space but this doesnt happen in other server which has the same specs and memory.

Thanks in advance.

Rgds
11 REPLIES
Khairy
Esteemed Contributor

Re: swap space full

hi,

these are the sysconfigtab parameters.

lsm:
lsm_rootdev_is_volume = 1
lsm_swapdev_is_volume = 1

ipc:
msg-max=32768
msg-mnb=65535
msg-mni=128
msg-tql=256
shm-max=2139095040
shm-mni=1024
shm-seg=512
sem-mni=512
sem-msl=512
sem-opm=512
sem-ume=512
num-of-sems=512
max-kernel-ports=16739
port-hash-max-num=8381950
port-reserved-max-num=16739
set-max-num=49413

proc:
max-proc-per-user=2048
max-threads-per-user=4096
per-proc-stack-size=2147483648
max-per-proc-stack-size=21447483648
per-proc-data-size=68719476736
max-per-proc-data-size=68719476736
max-per-proc-address-space=68719476736
per-proc-address-space=68719476736
maxusers=2048
task-max=16405
thread-max=32898
num-wait-queues=1024
num-timeout-hash-queues=1024

vm:
ubc-minpercent=5
ubc-maxpercent=10
ubc-borrowpercent=10
vm-maxvas=16106127360
vm-vpagemax=131072


I modified all parameters to the following today:

> per-proc-stack-size=4294967296
> max-per-proc-stack-size=4294967296
> per-proc-data-size=4294967296
> max-per-proc-data-size=4294967296
> max-per-proc-address-space=4294967296
> per-proc-address-space=4294967296
> ubc-minpercent=10
> ubc-maxpercent=50
> ubc-borrowpercent=20
> vm-maxvas=4294967296
> vm-vpagemax=1966080

After applied the parameters.This time its getting worse, while accessing the server via telnet, it hangs!

I had to access the server from main console, copy the original sysconfigtab file and reboot the server straight away.

When the server startup, app team submit same process. It hangs this time but for only for a few seconds. Then it resumes normally.


Kapil Jha
Honored Contributor

Re: swap space full

Well I have never worked on V4 , what i would have done is started the syatem and I'll monitor it when only OS is running no aplication.
And if its OK then there must be some problem with application and will try to see the problem from service end.
Can you or ask application tem to check what is happening in application when it is starting , was that aplication running fine before??
BR,
Kapil
I am in this small bowl, I wane see the real world......
Pieter 't Hart
Honored Contributor

Re: swap space full

adding swapspace is no solution!
this only results in the system becomming even slower beacuse of excessive swapping from memory to disk and back.
allocating more swapspace then 1,5 a 2 times the physical memory has only advantage when there are a lot of "sleeping" processes.
if you run a process wich really needs this much memory you should add physical memory.

this to investigate are processes allocating exessive memory.
- a single process allocating lots of memory
- too many processes running at the same time.

Most common cause i've run into is a periodically scheduled job (or one of its child's), that creates a temporary file with non-unique filename.
when the first instance still runs when the seccond instance starts, they want to access the same file.
This results in problems like
- locking
- one of them processing the wrong data (file from other instance).
as result they both run when a third instance starts etc...

when reducing the swap-space instead of increasing , the faulty job will run into problems earlier wich may help in locating the fault.

a command like this command might help:
ps -e -o vsz,pcpu,pid,rss,majflt,minflt,pmem,command|grep -v COMMAND|sort -nr|head
(vsz, pid and command could be enough).

beware! the sort order is not as we really want it.
Martin Moore
HPE Pro

Re: swap space full

Pieter is absolutely correct. The workload on your system is simply consuming too much memory. Your swapon output indicates that you're running in lazy (deferred) swap allocation mode. So your 4 GB of physical memory is getting filled up, then as the demand for memory continues, processes start swapping out until you fill up all of your swap space.

Tuning the system is not going to help much. You need to add more physical memory or reduce the workload or both. Adding more physical memory will probably do wonders for your system performance as well.

Martin
I work for HP
A quick resolution to technical issues for your HP Enterprise products is just a click away HP Support Center Knowledge-base
See Self Help Post for more details

Rob Leadbeater
Honored Contributor

Re: swap space full

Hi,

The process list from your first attachment shows that there are many processes all trying to grab massive amounts of memory.

For example:

adm 2792 0.0 2.4 3.46G 98M ?? S Jun 14 7-15:28:10 /runss17/
adm/MSP4.7/corp/bin_d/streaming Generic/generic_xgsm2


That's one process of many, that looks like it's trying to grab 3.46GB of memory.

You need to ask the application guys why this and similar processes are trying to use so much space. You might alleviate the problem by adding more physical memory, but I would suggest there is probably a serious issue with the application somewhere...

Cheers,

Rob
John Manger
Valued Contributor
Solution

Re: swap space full

The process listing shows a number of processes with a Virtual (VSZ) size of 3.4G, and a resident size (RSS) of 80-91M. The available Free Memory shown in the vmstat output (>30K pages) indicates there is still plenty of free memory.

You appear to be running with 'eager' swap allocation, which pre-allocates more or less the full VSZ of each running process from the available swap space. That is why you are running out of swap..... because of the numerous 3G+ VSZs.

You might consider running with lazy swap allocation IF your current process load is fairly static.
Nobody can serve both God and Money
Pieter 't Hart
Honored Contributor

Re: swap space full

I agree with John.
cause of the long text in the first two posts i missed the attachment.
Martin Moore
HPE Pro

Re: swap space full

It's definitely lazy swap already. If it were eager swap, there would be a "Reserved space" line in the last section of the swapon output.

But I agree, the basic idea is that the application appears to be trying to hog too much memory.

Martin
I work for HP
A quick resolution to technical issues for your HP Enterprise products is just a click away HP Support Center Knowledge-base
See Self Help Post for more details

Hein van den Heuvel
Honored Contributor

Re: swap space full

proc:
per-proc-stack-size=2147483648
max-per-proc-stack-size=21447483648

I suspect it is a consequence of the very high stack size settings? WHY is that made so high at 2GB? That is about 1000 times larger than many applications require.
That would certainly explain the 3GB + VSZ's

Setting it to 4G as you did, 'clearly' (to me :-) will make matters worse. try 1GB!

Check out vmstat -P
I suspect it is consuming very many ptepages.

And check out the lzay vs eager spaw comments and concerns.

hth,
Hein
Khairy
Esteemed Contributor

Re: swap space full

hi guys,

sorry for late coming back late. Was out of office for a couple of days.

Yes, martin is right. /sbin/swapdefault does exist and that prove the system is running on lazy (deferred) swap mode.

I guess, i couldn't change this as I think this mode is better. If i would go to eager mode, that would definately (maybe) makes things worst as swap space will be reserve unncessarily for process.

But robs finding interest me. Yes, it seems most of the process owns by adm consumes a lot of memory. And i think hein comments would explain all why 3GB + VSZs.

Setting 4GB to the following value might not make sense at all:

per-proc-stack-size
max-per-proc-stack-size

I will try to reduce above values to 1GB first and retain the other parameters. What you guys think?

In meanwhile, i will discuss with the app guy is it normal for this process to consume a lot of memory for text files processing.

Thanks again!
Hein van den Heuvel
Honored Contributor

Re: swap space full

Please note that the 1Gb suggestion was just to see whether you are on the rigth track. That is still a pretty ridicoulous stack size. Maybe someone mistook bytes for Kbytes somewhere, as ulimit specifies the size in kbytes. Something in the 1 - 32 MB region is probably much more reasonable.

Stack space, besides being its obvious use for call stacks, is used for LOCAL variable.
That's typically kilobytes, but could be megabytes or indeed gigabytes if say a programmer chose to reserve an array to hold millions of lines from data files.
Typically one would use HEAP space for that though (malloc), or map the file, or scan an process as you go.

Hein.