system error- too many subprocesses cannot fork errno=12 (354)

[Dean Ross-Smith]

I'm getting this error on an hp 9000 box running 11.11. In the system log i see repeated "Deferred swap reservation failure pid: XXXXXX".
From what I've seen on the forums, this is typically memory related and usually relates to not having enough swap available for preallocation for apps. The system is running many progress openedge databases and associated apps including web speed.
It's 6cpu box with 18GB ram. Where I think I have an issue is that the system has only a single 4GB swap swap device. But looking at swapinfo it shows ram usage is up against the limit too.

nclass:/ >swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 435 3661 11% 0 - 1 /dev/vg00/lvol2
reserve - 3661 -3661
memory 14465 14003 462 97%
total 18561 18099 462 98% - 0 -
nclass:/ >

To get me out of a pickle, will adding swap actually help in this case? Have we added enough stuff to the server that we need to add more ram too?
And if I can get out of this by adding more swap, can I add a swap file or device(lun) without rebooting the box? This system is used across every timezone in the US(including Hawaii) and it's a pain to get a maint window.

[Matti_Kurkela]

The "memory" line in "swapinfo -tam" output refers to so-called "pseudo-swap".

http://docs.hp.com/en/5965-4641/ch01s09.html

> will adding swap actually help in this case?

It allows your applications to keep running instead of crashing because the system cannot allocate any more memory. So yes, it will help.

At the very least, it will change the overload behaviour from "sudden death" to "slowing to a crawl". The former tends to require the users to re-do things; the latter is often a good reason to have a coffee break :-)

> [Do] we need to add more ram too?

That depends on the behaviour of your workload. Start monitoring the number of page-out events (the "po" column in the vmstat output, for example): when you see a lot of page-outs happening, your system is spending time furiously shuffling things between RAM and swap space and therefore is not working very efficiently.

In a heavily-loaded machine, small spikes of page-out events occasionally can be accepted, but if the system is constantly paging things out, you will really need more RAM.

> can I add a swap file or device(lun) without rebooting the box?

Yes to both. Removing an existing swap requires a reboot in HP-UX, but adding a new one does not.

MK

[Dean Ross-Smith]

the system is doing nearly zero paging. I'm just trying to confirm that adding swap is the correct path here.

[Dennis Handly]

>I'm just trying to confirm that adding swap is the correct path here.

It looks like it. You are only using 435 Mb swap and reserved the rest.

[Ganesan R · Accepted solution]

Hi,

It looks you are not running out of physical memory but running out of swap. There is no swap space to reserve for new process. Note reserving the swap and using the swap is differnt activities altogether.

I don't thing you have insufficient memory since you have 18GB. However you can confirm the memory utilisation using glance tool. But you have very less swap. Just 4GB swap is not sufficient for 18GB ram. Ideally you should have swap space equal or more size of physical memory.

So creating additional device swap surely solve your problems.

Refer this threat to create additional swap using SAM.

http://forums13.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1249817269840+28353475&threadId=1288851

[Sudeesh]

You have only 4GB swap configured now. From swapinfo o/p 'reserve' is very high, means lot of applications are started which are reserving swap area during their startup. I would recommed adding another 4GB swap considering you got 18GB physical Memory.

Verify kernel parameter swapmem_on and if it is set to 1, then dont worry about 'memory' being at 97%.

[Dennis Handly]

>Ganesan: It looks you are not running out of physical memory but running out of swap.

These are related, except one is more a expensive solution. :-)

> Sudeesh: Verify kernel parameter swapmem_on and if it is set to 1

As MK said, we know this is set because of the "memory" line in swapinfo.

[Dean Ross-Smith]

Thanks all- I'll be adding in a 15GB swap lun.