MMAP-Not enough memory on multiple SSH sessions

[panchpan]

Hello.
We encountered "MMAP-Not enough memory" issue in the syslog when we did a load test on Monday. The load test simulated concurrent SSH sessions while some key batch jobs were running. Everything works well until we try to open more than 313 SSH sessions (our target is 450). The error starts appearing and beyond that, no other SSH session can be established with the server and other processes cannot be started.
I am thinking of below solution to resolve this error:
Edit /etc/opt/ssh/sshd_config
From
#Compression yes to
Compression yes

Please advice.

[TY 007 · Accepted solution]

Hello Panchalp,

Increase the following OS Kernel Parameter: shmmni

Thanks

[TY 007]

Hello Panchalp,

After OS Kernel Parameter "shmmni" increase, perform further testing.

If issue persists, increase OS Kernel Parameter value "nproc" as well.

Thanks

[TY 007]

Hello Panchalp,

Install Latest Version of HP-UX Secure Shell (A.04.50.009/010/011):
http://h20293.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=T1471AA

Defects Fixed in above Latest Version:
http://docs.hp.com/en/5992-1942/ch01s04.html
http://docs.hp.com/en/5992-1942/ch01s05.html

Thanks

[TY 007]

Hello Panchalp,

HP-UX OS Patches Requirement:
http://docs.hp.com/en/5992-1942/ch01s09.html#babjebhg

For HP-UX 11.11 OS:
PHCO_35250 s700_800 11.11 libpam_unix cumulative patch
PHCO_35743 s700_800 11.11 libc cumulative patch
PHCO_36229 s700_800 11.11 pthread library cumulative patch

Thanks

[panchpan]

IF we increase the values of kernel parameters. We are not able to bring up the server. With little low values we can start the server but get "MMAP-Not enough memory" error.

Please advice.

[D Block 2]

what is your ratio of dbc_max/min_pct ?
you can gain back some memory by lowering the dbc_min_pct and dbc_max_pct.

at least you will have more user memory to use.

[panchpan]

How do I find this ration and values of it?

[D Block 2]

kmtune |grep dbc

you can lower the min value to gain back more User available memory. Change the dbc_min values to a low percentage. Run Glance and enter M for memory and look at the allocated or lockable memory on Boot. The dbc min can lower this reserve and give back to User available memory. just a thought.

[D Block 2]

for grins, type:

# dmesg

and post please.

[Dennis Handly]

>We encountered "MMAP- Not enough memory"

For private mmaps, this could mean out of swap space. Can you do "swapinfo -tam" around the time you are getting the above error?

For mmap on a file, it could be due to you running out of shared memory or having it fragmented. Can you do "ipcs -ma"

[TY 007]

Hello Panchalp,

>> IF we increase the values of kernel parameters
>> We are not able to bring up the server

Should not have any issue. Something is not right here.

Post attachment:
# kmtune > kmtune.txt
# swlist -l product > swist.txt

Advise - Original & NEW OS Kernel Parameter(s) Value that you tried

Thanks

[panchpan]

Hello.
Though the problem was resolved by having latest version of ssh.

However, I am scared that the problem might occur in future. The dynamic kernel parameters are at its high. Will need your expertise on whether you see anything that we should be alarmed with?

Actually, We are migrating our application from Server A to Server B. Server A is well configured and never had any issues with memory. Below are some requested outputs and comparisons from both servers:

1. dmesg command gave me below error on peg1 and cceor10

Can't read kernel memory

However, I could get the below information from /var/adm/messages file.

For Server A Memory Information:
physical page size = 4096 bytes, logical page size = 4096 bytes
Physical: 14680064 Kbytes, lockable: 11559420 Kbytes, available: 13258888 Kbytes

For Server B:
The message files is of zero byte on cceor10.
/var/adm/messages file is empty on cceor10
-rw-r--r-- 1 root root 0 Oct 11 19:43 /var/adm/messages

I have attached output of ipcs â ma. and output of Glance

Server A
Total VM : 1.23gb Sys Mem : 971.0mb User Mem: 5.29gb Phys Mem: 14.0gb
Active VM: 97.1mb Buf Cache: 1.12gb Free Mem: 6.64gb

Server B
Total VM : 3.9gb Sys Mem : 3.4gb User Mem: 2.6gb Phys Mem: 16.0gb
Active VM: 3.8gb Buf Cache: 1.9gb Free Mem: 8.0gb

Please advice - IF any actions are needed on Server B to avoid such memory issues in future?

Many Thanks!!!

[panchpan]

Sorry I missed the attachements !!!

[Dennis Handly]

Your ipcs output seems to come from different OS versions? (The size of the ID column is larger in B.)

It seems the two are similar. There are lots of 16 Mb segments.
The total is very large:
$ awk_size.sh ipcs@serverA.txt
Total MB size: 1153.11
$ awk_size.sh ipcs@serverB.txt
Total MB size: 1472.58

What does swapinfo -tam have for each?

[panchpan]

Hi Dennis,
Thanks for your reply.

By seeing quickly the differences - Do you see anything missing or scope of improvements on Server B? Is it good enough?

Yes Server B is 11.11 and A is just 11.

Sorry I did not understand the lines about awk_size.sh ?

[Dennis Handly]

>Do you see anything missing or scope of improvements on Server B? Is it good enough?

The max* values seemed to be bigger.

>I did not understand the lines about awk_size.sh?

A script I wrote to add up the shared memory segment sizes.