SWAP error - Plz help

[Gops_1]

I am having an HP NNM server in HP UX 11.0. Suddenly all my sevices was stopped and i coudln't restart too. I have did a hardware restart of my hp UX server. My sevices and dmesg output as follows

btlan3: Initializing 10/100BASE-TX card at 0/0/0/0....

System Console is on the Built-In Serial Interface
btlan5: Initializing 10/100BASE-TX card at 0/2/0/0....
Logical volume 64, 0x3 configured as ROOT
Logical volume 64, 0x2 configured as SWAP
Logical volume 64, 0x2 configured as DUMP
Swap device table: (start & size given in 512-byte blocks)
entry 0 - major is 64, minor is 0x2; start = 0, size = 8388608
Unable to add all swap for device: Primary Swap. Increase the tunable parameter maxswapchunks by 1024 and re-configure your system.
Dump device table: (start & size given in 1-Kbyte blocks)
entry 0 - major is 31, minor is 0x1f000; start = 519008, size = 4194304
Starting the STREAMS daemons-phase 1
Create STCP device files
B2352B/9245XB HP-UX (B.11.00) #1: Wed Nov 5 22:38:19 PST 1997

Memory Information:
physical page size = 4096 bytes, logical page size = 4096 bytes
Physical: 1048576 Kbytes, lockable: 827384 Kbytes, available: 861768 Kbytes

You can see a line "Unable to add all swap for device: Primary Swap. Increase the tunable parameter maxswapchunks by 1024 and re-configure your system".

but when i check my kmtune -q maxswapchunks its 1024 only.

I ll give max. points to all... Please help me

What log file you want plz tell i ll provide you...

i am getting this kind of message alert in my OVO server as am monitoring this NNM server.

"Global swap space is nearly full - Swap Space Utilization = 100.00% ( START: 13:05:00 04/25/08 ) INOC (INOCNet) Critical HP 9000 PA-"

"Cron command of root with pid 8509 failed (rc = 1) INOC (INOCNet) Critical HP 9000 PA-RISC"

Whenever i do a ping it give the following error:

"SH:cannot fork no swap space."

Thanks
gV

[whiteknight · Accepted solution]

GV,

You system has insufficient resources, especially in OV environment, make sure youe swapspace have 2x of size of your physical memory.

swapinfo -tam will tell you the status of the system.


WK

[Gops_1]

Thanks for your imm response,

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 2048 0% 0 - 1 /dev/vg00/lvol2
reserve - 112 -112
total 4096 112 1936 3% - 0 -

Above is my swapinfo_tam

Sometime 3% increases to 50%.

sudo vmstat -n
VM
memory page faults
avm free re at pi po fr de sr in sy cs
4212 189361 48 20 5 0 26 0 7 152 4272 1307
CPU
cpu procs
us sy id r b w
3 2 95 0 0 0
(/opt/home/x57750)$


above is my vmstat output.

Pls let me know what should do next ?

how should i check whether my swap is 2X of my physical mem.

Thanks
gV

[YAQUB_1]

U can check "dmesg" what is Ur physical memory....then U "swapinfo -a"....send us output

How U can increase swap size....here is process

1) lvcreate -C y -m 1 -L -n lvolx /dev/vg00
2) Next create an entry in /etc/fstab like this:
/dev/vg00/lvolx ... swap pri=2 0 0 --- (check same parameter already is there)

3) swapon -a

[Venkatesh BL]

From http://docs.hp.com/en/TKP-90202/TKP-90202.pdf, I see some possible causes and solution for your problem. Check it out.

[Venkatesh BL]

I think you have enough swap space. Its just that the kernel table for holding these chunks is not configured correctly. Once you increase the table size, you should be able to use the space properly.

[Jeeshan]

Hi gV

how many swap space you have available in your system?

if you enable pseudoswap?

may be problem is in swap utilization.

[Gops_1]

Hi all,
It correct that we have enough swap space. I am not going to increase the swap space. Venkatesh thanks for you suggestion.

allocate_fs_swapmap 0
maxswapchunks 1024
nswapdev 10
nswapfs 10
remote_nfs_swap 0
swapmem_on 0

My swap kernel parameters are above.

Yes Ahsan,
There is a problem with swap utilization

Please see the below output,

(/opt/home/x57750)$ sudo swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 2048 0% 0 - 1 /dev/vg00/lvol2
reserve - 117 -117
total 4096 117 1931 3% - 0 -
We are using a script to start our NNM services. When we use this script we are facing a peak 50% in Total 50% and it uses only the reserver. I dont know y is it not using the dev swap space.
You can see the below output after i ran the NNM script to start my NNM. U can see the "total" is "USED" as 50%

(/opt/home/x57750)$ sudo swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 2048 0% 0 - 1 /dev/vg00/lvol2
reserve - 2045 -2045
total 4096 2045 3 50% - 0 -

Please advice me what should i do ? Now am totally confused. Its a production server too. How suddenly this can happen?

Thanks
gV

[Don Morris_1]

Your maxswapchunks is 1024, which with the default value of swchunk (2048 -- see http://docs.hp.com/en/TKP-90202/re97.html which is the 11.11 page, but 11.0 is the same) means you can represent 1024 * 2048 * 1K of swap [the block size is almost always 1Kb, so total swap = maxswapchunks * swchunk * 1Kb]. That's 2 * 1024 * 1024 * 1024 or 2Gb.

Since you want to have 4Gb of swap, your system is trying to tell you to set maxswapchunks to 2048 instead of 1024 so you can represent the 4Gb (2048 * 2048 * 1024). Hence the *increase* the tunable by 1024 [the code is written to simply calculate how much more you need, not the final value].

With only 2Gb of swap, that means you can only have 2Gb of virtual objects since all virtual objects must reserve backing store. [File backed objects such as shared mmap() don't use swap, they use the file -- anything allocated via malloc(), anonymous mmaps, stacks, SysV shared memory, etc. is swap backed]. This is your "reserve" line in swapinfo. Since 2048 of your swap total is not enabled on the system, when your reserve hits 2048 as well -- all your swap is reserved and no new virtual objects are likely to be made. This is when you will see failures from fork() / malloc() / whatnot.
That's why you never hit 100% for USED by the way -- swapinfo doesn't represent "There but not Enabled (and hence not USED)" all that well, in my opinion [but I'm just a kernel guy, not a UI person]. Since 50% of your swap can't be USED but isn't free either, your real total available is only 50% until you get that tunable up.

First, set the tunable like the system is trying to tell you to do (at least 2048). After that, you can look at your reservation peaks (since you obviously have a transient virtual memory load when the NNM stuff runs) to see if you need to enable more swap, or look at kernel tunables [maxdsiz, maxssiz] / setrlimit usage to limit how much virtual address space (and hence swap reservations) an allocation can use. I'd set the swapmem_on tunable to 1 as well so you can have memory swap enabled.

[Gops_1]

Thank you so much morris. I tried opening SAM to change the kernel but its not opening. Can u provide me the exact command to change the parameter to 2048. Also tell me whether i need to do a restart of the machine.


Thaanks
gV

[Yogeeraj_1]

hi gV,

Unfortunately, maxswapchunks cannot be dynamically changed:

(this can be observed by running the command below)
SRV2:home/root>kmtune -q maxswapchunks
Parameter Current Dyn Planned Module Version
===============================================================================
maxswapchunks 16384 - 16384
SRV2:home/root>kmtune -s maxswapchunks=
SRV2:home/root>

What error are you getting when running SAM?

to set the value, you can try the following comamnd:
SRV2:home/root>kmtune -s maxswapchunks=




hope this helps!

kind regards
yogeeraj

[Yogeeraj_1]

hi again,

Sorry. I am not sure it the command will work on 11.0. I tried the commands in my previous post on 11.11. I do not have a 11.0 at hand right now.


Anyway, if you fix the SAM problem, you will be able to change the value correctly. Update us on the error message that you are getting when running SAM.

kind regards
yogeeraj

[Gops_1]

It works thank yogeeraj. Do i need to do a restart of the my HP UX server ?

reply soon..

Thanks
gV

[Yogeeraj_1]

Yes. You will have to do so for the parameter to take effect. As i said, this kernel parameter cannot be change dynamically (in version 11.11 - confirmed)


good luck!
kind regards
yogeeraj

[Gops_1]

After i changed the parameter now am not getting into the server. Whenever i do ssh it say the following error.

Received disconnect from 202.61.9.20: 2: fork failed: Resource temporarily unavailable


Any help...

Thanks gV

[YAQUB_1]

Changes to this tunable take effect at the next reboot.

[Yogeeraj_1]

hi,

Sounds as if your process table is full.

glance -t

this should show you the real problem.

e.g. (glance -t output)
--------------------------------------------------------------------------------
SYSTEM TABLES REPORT Users= 2

System Table Available Used Utilization High(%)
--------------------------------------------------------------------------------
Proc Table (nproc) 4096 177 4 4
File Table (nfile) 63498 674 1 1
Shared Mem Table (shmmni) 512 10 2 2
Message Table (msgmni) 4096 2 0 0
Semaphore Table (semmni) 4096 21 1 1
File Locks (nflocks) 4096 26 1 1
Pseudo Terminals (npty) 60 0 0 0
Buffer Headers (nbuf) na 67320 na na



revert!

kind regards
yogeeraj

[Gops_1]

B3692A GlancePlus C.03.55.00 18:00:54 pnmp2s 9000/800 Current Avg High
--------------------------------------------------------------------------------
CPU Util SU | 1% 12% 66%
Disk Util F | 1% 9% 31%
Mem Util S SU UB B | 34% 34% 35%
Swap Util R R | 5% 5% 5%
--------------------------------------------------------------------------------
SYSTEM TABLES REPORT Users= 1

System Table Available Used Utilization High(%)
--------------------------------------------------------------------------------
Proc Table (nproc) 1460 85 6 6
File Table (nfile) 21111 552 3 3
Shared Mem Table (shmmni) 200 6 3 3
Message Table (msgmni) 50 2 4 4
Semaphore Table (semmni) 64 20 31 31
File Locks (nflocks) 200 22 11 11
Pseudo Terminals (npty) 60 1 2 2
Buffer Headers (nbuf) na 8631 na na


This is my glance -t output.

Even after i changed those two kernel parameter as given morris doesn't helped me.
First of all my ping itself is not working. Whenever i do ping it just hangs...

thanks
gV

[YAQUB_1]

send us "kctune" or "kmtune" output....

"swchunk" should be considered when modifying "maxswapchunks". BTW, put the output....

[Ganesan R]

Try to capture the syslog messages and post it. it may help us what is going on the server

[Yogeeraj_1]

hi,

Are you getting any messages on your console?

Also check the syslog.log. Any new error messages?

revert!
kind regards
yogeeraj

[Gops_1]

yah swchunk is 2048 and now i have changed the mmaxswapchunks to 2048. But it didn't helped me.
Syslog is not giving anything. Its one big dumbo. It just gives the command I type. It doesn't gives any error. Its an idiot.



thanks
gV

[YAQUB_1]

A swap chunk is a collection of 1 KB blocks (the exact size is determined by the tunable swchunk).

swchunk sets the size of a swap chunk in 1024 byte pages, so increasing the chunk size yields less chunks needed in the table (and a smaller needed maxswapchunks) for a given swap space. Similarly, lowering swchunk means that more chunk indices in the swap table will be needed.

[Ganesan R]

Hi,

We are interested to know the following after you started all the service/apps/scripts.

1.Swap utilisation
#swapinfo -mat
2.CPU/Memory/disk utilisation
#glance
#top
#sar -u 2 2
#sar -d 2 2

3.System resource utilisation
#glance -t
#sar -v 2 2

What is the current status? Aren't you able to ping/telnet/ssh to the server?
Where you are getting the error? what is the exact error you are getting now ?


If nothing is helped out, you can consider to make a clean reboot of the server since you already made few changes to the kernel and it needs to take effect.

[YAQUB_1]

Hi,

U can change Ur both kernel parameter for below value:

maxswapchunks----16384 &
swchunk----------16384

then reboot Ur system & check current status........