topic Re: Swapspace is full in Operating System - HP-UX

Swapspace is full

samay_hcl — Mon, 01 Mar 2010 08:59:43 GMT

I am not able to run any command on the host.

Error is below show.
# swapinfo -ta
sh: The fork function failed. There is not enough memory available.

# UNIX95= ps -ef -o vsz,ruser,pid|sort -rn
sh: The fork function failed. There is not enough memory available.

Experts,

can somebody help me?

Instead rebooting or adding lvol's, is there any other solution for this?

A few min before status was that i can see.

# swapinfo -atm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
dev 40000 0 40000 0% 0 - 1 /dev/vg00/lvol3
dev 40000 0 40000 0% 0 - 1 /dev/vg00/lvol7
dev 40000 0 40000 0% 0 - 1 /dev/vg00/lvol8
reserve - 124096 -124096
memory 25568 25565 3 100%
total 149664 149661 3 100% - 0 -

Thanks!

Re: Swapspace is full

Kapil Jha — Mon, 01 Mar 2010 09:08:55 GMT

U have huger swap and compared to memory, what is the application you are running.

As of now there seems to be no solution other than reboot the server and you have have to fine tune the application thins check for potential memoey leaks in the server.

BR,
Kapil+

Re: Swapspace is full

Kapil Jha — Mon, 01 Mar 2010 09:09:34 GMT

huge* swap

BR,
Kapil+

Re: Swapspace is full

Jupinder Bedi — Mon, 01 Mar 2010 09:17:27 GMT

run glance or top command ans see if its work. you need to find out which process is consuming more space . try to gpm this is exact thing which can tell you that which process is consuming more memory.

in your case physical memory is 100% full . rigght now there is no any option . you can take your server reboot.

Re: Swapspace is full

Michal Kapalka (mikap) — Mon, 01 Mar 2010 09:17:29 GMT

hi,

if the additional swap lvols are on the same disk, the io will kill your server.

check which app./process consume so much memory and swap space.

mikap

Re: Swapspace is full

Michal Kapalka (mikap) — Mon, 01 Mar 2010 09:18:34 GMT

hi,

if you can't "fork", the last solution is to reboot the server.

mikap

Re: Swapspace is full

Jupinder Bedi — Mon, 01 Mar 2010 09:23:46 GMT

I have one more solution for you . if you dont want to create the lvol than you can create the swap space by using mkfile command .

Re: Swapspace is full

Javed Khan_1 — Mon, 01 Mar 2010 09:28:52 GMT

probably you application has some kind of memory leak which is causing memory and swap 100% utilization.

you can wait for some time and see if utilization comes down in order to run commands else reboot is the only option.

Once your system comes back to normal you will need to keep an eye on your application

Javed

Re: Swapspace is full

john korterman — Mon, 01 Mar 2010 10:14:55 GMT

Hi,

your swapinfo looks weird:
/dev/vg00/lvol2 is normally used for primary swap, but on most hpux systems /dev/vg00/lvol3 is used for the root partition.
/dev/vg00/lvol8 is normally also used for an hpux partition, e.g. /usr
Do you run a standard hpux installation, or have you recently changed the swap configuration?

regards,
John K.

Re: Swapspace is full

Dennis Handly — Mon, 01 Mar 2010 11:37:05 GMT

>Instead rebooting or adding lvols, is there any other solution for this?

You must kill enough processes so you can figure out what to do. Basically you have too much device swap (5 X memory) and too many processes running.

Re: Swapspace is full

Rita C Workman — Mon, 01 Mar 2010 13:37:30 GMT

You obviously added on swap........but what concerns me is where you did it!!

/dev/vg00/lvol3 is / on my box
/dev/vg00/lvol7 is /usr
/dev/vg00/lvol8 is /var

Now lvol7 & lvol8 might change as to the f/s associated...but lvol1, lvol2, lvol3 doesn't as a general rule.

So...I'd frankly if I'm seeing this right, you might need to rebuild that box from a recovery (tape or net).

Rita

Re: Swapspace is full

Rita C Workman — Mon, 01 Mar 2010 15:08:54 GMT

Apology to John Korterman...just noticed that he saw what I saw-first.

/rcw

Re: Swapspace is full

Michael Steele_2 — Mon, 01 Mar 2010 15:57:38 GMT

"...U have huger swap and compared to memory, what is the application you are running...."

Sorry, I don't see the relevance.

"...if the additional swap lvols are on the same disk, the io will kill your server...."

Sorry, only to a point and because the o/s is also in vg00, else, io will be constant as 2ndary swap is written too in round robin order and not in parallel.

Going up: lvol2>lvol3>lvol7>lvol8
Going down: lvol8>lvol7>lvol3>lvol2

a) first reboot the server since you no longer have any memory

b) 2nd add more swap, another vg then vg00 would be preferred since the o/s is always the busiest disk. In the future, separate all your 2ndary swap from vg00 and onto another vg with other disks, then you'll be getting more spindles involved

c) Check for a memeory leak using this procedure:

1) 15 to 20 minute cron until id'd memeory leak

2) UNIX95=1 ps -ef -o vsz,pid,ppid,state,wchan,comm | sort -rn | head -15 >> /dir/outfile

Note: Put this into a script and add some headers and $DATE's with timestamps to make a readable report

3) Check the report from day to day, note any growth in the first column, the vsz column. A growing vsz will id the pid and ppid with the memory leak.

Re: Swapspace is full

Michael Steele_2 — Mon, 01 Mar 2010 15:59:48 GMT

OMG

"You obviously added on swap........but what concerns me is where you did it!!"

Geez Rita, you're right!!!

Re: Swapspace is full

T. M. Louah — Tue, 02 Mar 2010 13:29:03 GMT

when I see 100% swap, I react by adding filesystem swap (not a good idea just short term temporary w/a) then I investigate where paging taking place and why.. once root cause identified, if i need to add swap, Device swap is best for performance and need to be assigned on different disks than primary swap.

t#