cancel
Showing results for 
Search instead for 
Did you mean: 

Swapspace is full

samay_hcl
Advisor

Swapspace is full

I am not able to run any command on the host.

Error is below show.
# swapinfo -ta
sh: The fork function failed. There is not enough memory available.

# UNIX95= ps -ef -o vsz,ruser,pid|sort -rn
sh: The fork function failed. There is not enough memory available.

Experts,

can somebody help me?

Instead rebooting or adding lvol's, is there any other solution for this?

A few min before status was that i can see.

# swapinfo -atm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
dev 40000 0 40000 0% 0 - 1 /dev/vg00/lvol3
dev 40000 0 40000 0% 0 - 1 /dev/vg00/lvol7
dev 40000 0 40000 0% 0 - 1 /dev/vg00/lvol8
reserve - 124096 -124096
memory 25568 25565 3 100%
total 149664 149661 3 100% - 0 -


Thanks!

14 REPLIES
Kapil Jha
Honored Contributor

Re: Swapspace is full

U have huger swap and compared to memory, what is the application you are running.

As of now there seems to be no solution other than reboot the server and you have have to fine tune the application thins check for potential memoey leaks in the server.

BR,
Kapil+
I am in this small bowl, I wane see the real world......
Kapil Jha
Honored Contributor

Re: Swapspace is full

huge* swap

BR,
Kapil+
I am in this small bowl, I wane see the real world......
Jupinder Bedi
Respected Contributor

Re: Swapspace is full

run glance or top command ans see if its work. you need to find out which process is consuming more space . try to gpm this is exact thing which can tell you that which process is consuming more memory.


in your case physical memory is 100% full . rigght now there is no any option . you can take your server reboot.
All things excellent are as difficult as they are rare
Michal Kapalka (mikap)
Honored Contributor

Re: Swapspace is full

hi,

if the additional swap lvols are on the same disk, the io will kill your server.

check which app./process consume so much memory and swap space.

mikap
Michal Kapalka (mikap)
Honored Contributor

Re: Swapspace is full

hi,

if you can't "fork", the last solution is to reboot the server.

mikap
Jupinder Bedi
Respected Contributor

Re: Swapspace is full

I have one more solution for you . if you dont want to create the lvol than you can create the swap space by using mkfile command .
All things excellent are as difficult as they are rare
Javed Khan_1
Valued Contributor

Re: Swapspace is full

probably you application has some kind of memory leak which is causing memory and swap 100% utilization.

you can wait for some time and see if utilization comes down in order to run commands else reboot is the only option.

Once your system comes back to normal you will need to keep an eye on your application

Javed
Never Give Up
john korterman
Honored Contributor

Re: Swapspace is full

Hi,

your swapinfo looks weird:
/dev/vg00/lvol2 is normally used for primary swap, but on most hpux systems /dev/vg00/lvol3 is used for the root partition.
/dev/vg00/lvol8 is normally also used for an hpux partition, e.g. /usr
Do you run a standard hpux installation, or have you recently changed the swap configuration?


regards,
John K.
it would be nice if you always got a second chance
Dennis Handly
Acclaimed Contributor

Re: Swapspace is full

>Instead rebooting or adding lvols, is there any other solution for this?

You must kill enough processes so you can figure out what to do. Basically you have too much device swap (5 X memory) and too many processes running.
Rita C Workman
Honored Contributor

Re: Swapspace is full

You obviously added on swap........but what concerns me is where you did it!!

/dev/vg00/lvol3 is / on my box
/dev/vg00/lvol7 is /usr
/dev/vg00/lvol8 is /var

Now lvol7 & lvol8 might change as to the f/s associated...but lvol1, lvol2, lvol3 doesn't as a general rule.

So...I'd frankly if I'm seeing this right, you might need to rebuild that box from a recovery (tape or net).

Rita
Rita C Workman
Honored Contributor

Re: Swapspace is full

Apology to John Korterman...just noticed that he saw what I saw-first.

/rcw
Michael Steele_2
Honored Contributor

Re: Swapspace is full

"...U have huger swap and compared to memory, what is the application you are running...."

Sorry, I don't see the relevance.

"...if the additional swap lvols are on the same disk, the io will kill your server...."

Sorry, only to a point and because the o/s is also in vg00, else, io will be constant as 2ndary swap is written too in round robin order and not in parallel.

Going up: lvol2>lvol3>lvol7>lvol8
Going down: lvol8>lvol7>lvol3>lvol2

a) first reboot the server since you no longer have any memory

b) 2nd add more swap, another vg then vg00 would be preferred since the o/s is always the busiest disk. In the future, separate all your 2ndary swap from vg00 and onto another vg with other disks, then you'll be getting more spindles involved

c) Check for a memeory leak using this procedure:

1) 15 to 20 minute cron until id'd memeory leak

2) UNIX95=1 ps -ef -o vsz,pid,ppid,state,wchan,comm | sort -rn | head -15 >> /dir/outfile

Note: Put this into a script and add some headers and $DATE's with timestamps to make a readable report

3) Check the report from day to day, note any growth in the first column, the vsz column. A growing vsz will id the pid and ppid with the memory leak.
Support Fatherhood - Stop Family Law
Michael Steele_2
Honored Contributor

Re: Swapspace is full

OMG

"You obviously added on swap........but what concerns me is where you did it!!"

Geez Rita, you're right!!!
Support Fatherhood - Stop Family Law
T. M. Louah
Esteemed Contributor

Re: Swapspace is full

when I see 100% swap, I react by adding filesystem swap (not a good idea just short term temporary w/a) then I investigate where paging taking place and why.. once root cause identified, if i need to add swap, Device swap is best for performance and need to be assigned on different disks than primary swap.

t#
Little learning is dangerous!