Swapspace is full

[samay_hcl]

I am not able to run any command on the host.

Error is below show.
# swapinfo -ta
sh: The fork function failed. There is not enough memory available.

# UNIX95= ps -ef -o vsz,ruser,pid|sort -rn
sh: The fork function failed. There is not enough memory available.

Experts,

can somebody help me?

Instead rebooting or adding lvol's, is there any other solution for this?

A few min before status was that i can see.

# swapinfo -atm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
dev 40000 0 40000 0% 0 - 1 /dev/vg00/lvol3
dev 40000 0 40000 0% 0 - 1 /dev/vg00/lvol7
dev 40000 0 40000 0% 0 - 1 /dev/vg00/lvol8
reserve - 124096 -124096
memory 25568 25565 3 100%
total 149664 149661 3 100% - 0 -


Thanks!

[Kapil Jha]

U have huger swap and compared to memory, what is the application you are running.

As of now there seems to be no solution other than reboot the server and you have have to fine tune the application thins check for potential memoey leaks in the server.

BR,
Kapil+

[Kapil Jha]

huge* swap

BR,
Kapil+

[Jupinder Bedi]

run glance or top command ans see if its work. you need to find out which process is consuming more space . try to gpm this is exact thing which can tell you that which process is consuming more memory.


in your case physical memory is 100% full . rigght now there is no any option . you can take your server reboot.

[Michal Kapalka (mikap)]

hi,

if the additional swap lvols are on the same disk, the io will kill your server.

check which app./process consume so much memory and swap space.

mikap

[Michal Kapalka (mikap)]

hi,

if you can't "fork", the last solution is to reboot the server.

mikap

[Jupinder Bedi]

I have one more solution for you . if you dont want to create the lvol than you can create the swap space by using mkfile command .

[Javed Khan_1]

probably you application has some kind of memory leak which is causing memory and swap 100% utilization.

you can wait for some time and see if utilization comes down in order to run commands else reboot is the only option.

Once your system comes back to normal you will need to keep an eye on your application

Javed

[john korterman]

Hi,

your swapinfo looks weird:
/dev/vg00/lvol2 is normally used for primary swap, but on most hpux systems /dev/vg00/lvol3 is used for the root partition.
/dev/vg00/lvol8 is normally also used for an hpux partition, e.g. /usr
Do you run a standard hpux installation, or have you recently changed the swap configuration?


regards,
John K.

[Dennis Handly]

>Instead rebooting or adding lvols, is there any other solution for this?

You must kill enough processes so you can figure out what to do. Basically you have too much device swap (5 X memory) and too many processes running.

[Rita C Workman]

You obviously added on swap........but what concerns me is where you did it!!

/dev/vg00/lvol3 is / on my box
/dev/vg00/lvol7 is /usr
/dev/vg00/lvol8 is /var

Now lvol7 & lvol8 might change as to the f/s associated...but lvol1, lvol2, lvol3 doesn't as a general rule.

So...I'd frankly if I'm seeing this right, you might need to rebuild that box from a recovery (tape or net).

Rita

[Rita C Workman]

Apology to John Korterman...just noticed that he saw what I saw-first.

/rcw

[Michael Steele_2]

"...U have huger swap and compared to memory, what is the application you are running...."

Sorry, I don't see the relevance.

"...if the additional swap lvols are on the same disk, the io will kill your server...."

Sorry, only to a point and because the o/s is also in vg00, else, io will be constant as 2ndary swap is written too in round robin order and not in parallel.

Going up: lvol2>lvol3>lvol7>lvol8
Going down: lvol8>lvol7>lvol3>lvol2

a) first reboot the server since you no longer have any memory

b) 2nd add more swap, another vg then vg00 would be preferred since the o/s is always the busiest disk. In the future, separate all your 2ndary swap from vg00 and onto another vg with other disks, then you'll be getting more spindles involved

c) Check for a memeory leak using this procedure:

1) 15 to 20 minute cron until id'd memeory leak

2) UNIX95=1 ps -ef -o vsz,pid,ppid,state,wchan,comm | sort -rn | head -15 >> /dir/outfile

Note: Put this into a script and add some headers and $DATE's with timestamps to make a readable report

3) Check the report from day to day, note any growth in the first column, the vsz column. A growing vsz will id the pid and ppid with the memory leak.

[Michael Steele_2]

OMG

"You obviously added on swap........but what concerns me is where you did it!!"

Geez Rita, you're right!!!

[T. M. Louah]

when I see 100% swap, I react by adding filesystem swap (not a good idea just short term temporary w/a) then I investigate where paging taking place and why.. once root cause identified, if i need to add swap, Device swap is best for performance and need to be assigned on different disks than primary swap.

t#