topic Re: SWAP error - Plz help in Operating System - HP-UX

SWAP error - Plz help

Gops_1 — Fri, 25 Apr 2008 07:03:25 GMT

I am having an HP NNM server in HP UX 11.0. Suddenly all my sevices was stopped and i coudln't restart too. I have did a hardware restart of my hp UX server. My sevices and dmesg output as follows

btlan3: Initializing 10/100BASE-TX card at 0/0/0/0....

System Console is on the Built-In Serial Interface
btlan5: Initializing 10/100BASE-TX card at 0/2/0/0....
Logical volume 64, 0x3 configured as ROOT
Logical volume 64, 0x2 configured as SWAP
Logical volume 64, 0x2 configured as DUMP
Swap device table: (start & size given in 512-byte blocks)
entry 0 - major is 64, minor is 0x2; start = 0, size = 8388608
Unable to add all swap for device: Primary Swap. Increase the tunable parameter maxswapchunks by 1024 and re-configure your system.
Dump device table: (start & size given in 1-Kbyte blocks)
entry 0 - major is 31, minor is 0x1f000; start = 519008, size = 4194304
Starting the STREAMS daemons-phase 1
Create STCP device files
B2352B/9245XB HP-UX (B.11.00) #1: Wed Nov 5 22:38:19 PST 1997

Memory Information:
physical page size = 4096 bytes, logical page size = 4096 bytes
Physical: 1048576 Kbytes, lockable: 827384 Kbytes, available: 861768 Kbytes

You can see a line "Unable to add all swap for device: Primary Swap. Increase the tunable parameter maxswapchunks by 1024 and re-configure your system".

but when i check my kmtune -q maxswapchunks its 1024 only.

I ll give max. points to all... Please help me

What log file you want plz tell i ll provide you...

i am getting this kind of message alert in my OVO server as am monitoring this NNM server.

"Global swap space is nearly full - Swap Space Utilization = 100.00% ( START: 13:05:00 04/25/08 ) INOC (INOCNet) Critical HP 9000 PA-"

"Cron command of root with pid 8509 failed (rc = 1) INOC (INOCNet) Critical HP 9000 PA-RISC"

Whenever i do a ping it give the following error:

"SH:cannot fork no swap space."

Thanks
gV

Re: SWAP error - Plz help

whiteknight — Fri, 25 Apr 2008 07:15:03 GMT

GV,

You system has insufficient resources, especially in OV environment, make sure youe swapspace have 2x of size of your physical memory.

swapinfo -tam will tell you the status of the system.

WK

Re: SWAP error - Plz help

Gops_1 — Fri, 25 Apr 2008 07:22:07 GMT

Thanks for your imm response,

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 2048 0% 0 - 1 /dev/vg00/lvol2
reserve - 112 -112
total 4096 112 1936 3% - 0 -

Above is my swapinfo_tam

Sometime 3% increases to 50%.

sudo vmstat -n
VM
memory page faults
avm free re at pi po fr de sr in sy cs
4212 189361 48 20 5 0 26 0 7 152 4272 1307
CPU
cpu procs
us sy id r b w
3 2 95 0 0 0
(/opt/home/x57750)$

above is my vmstat output.

Pls let me know what should do next ?

how should i check whether my swap is 2X of my physical mem.

Thanks
gV

Re: SWAP error - Plz help

YAQUB_1 — Fri, 25 Apr 2008 07:31:30 GMT

U can check "dmesg" what is Ur physical memory....then U "swapinfo -a"....send us output

How U can increase swap size....here is process

1) lvcreate -C y -m 1 -L -n lvolx /dev/vg00
2) Next create an entry in /etc/fstab like this:
/dev/vg00/lvolx ... swap pri=2 0 0 --- (check same parameter already is there)

3) swapon -a

Re: SWAP error - Plz help

Venkatesh BL — Fri, 25 Apr 2008 07:32:29 GMT

From http://docs.hp.com/en/TKP-90202/TKP-90202.pdf, I see some possible causes and solution for your problem. Check it out.

Re: SWAP error - Plz help

Venkatesh BL — Fri, 25 Apr 2008 07:36:09 GMT

I think you have enough swap space. Its just that the kernel table for holding these chunks is not configured correctly. Once you increase the table size, you should be able to use the space properly.

Re: SWAP error - Plz help

Jeeshan — Fri, 25 Apr 2008 07:47:26 GMT

Hi gV

how many swap space you have available in your system?

if you enable pseudoswap?

may be problem is in swap utilization.

Re: SWAP error - Plz help

Gops_1 — Fri, 25 Apr 2008 09:46:37 GMT

Hi all,
It correct that we have enough swap space. I am not going to increase the swap space. Venkatesh thanks for you suggestion.

allocate_fs_swapmap 0
maxswapchunks 1024
nswapdev 10
nswapfs 10
remote_nfs_swap 0
swapmem_on 0

My swap kernel parameters are above.

Yes Ahsan,
There is a problem with swap utilization

Please see the below output,

(/opt/home/x57750)$ sudo swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 2048 0% 0 - 1 /dev/vg00/lvol2
reserve - 117 -117
total 4096 117 1931 3% - 0 -
We are using a script to start our NNM services. When we use this script we are facing a peak 50% in Total 50% and it uses only the reserver. I dont know y is it not using the dev swap space.
You can see the below output after i ran the NNM script to start my NNM. U can see the "total" is "USED" as 50%

(/opt/home/x57750)$ sudo swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 2048 0% 0 - 1 /dev/vg00/lvol2
reserve - 2045 -2045
total 4096 2045 3 50% - 0 -

Please advice me what should i do ? Now am totally confused. Its a production server too. How suddenly this can happen?

Thanks
gV

Re: SWAP error - Plz help

Don Morris_1 — Fri, 25 Apr 2008 10:12:30 GMT

Your maxswapchunks is 1024, which with the default value of swchunk (2048 -- see http://docs.hp.com/en/TKP-90202/re97.html which is the 11.11 page, but 11.0 is the same) means you can represent 1024 * 2048 * 1K of swap [the block size is almost always 1Kb, so total swap = maxswapchunks * swchunk * 1Kb]. That's 2 * 1024 * 1024 * 1024 or 2Gb.

Since you want to have 4Gb of swap, your system is trying to tell you to set maxswapchunks to 2048 instead of 1024 so you can represent the 4Gb (2048 * 2048 * 1024). Hence the *increase* the tunable by 1024 [the code is written to simply calculate how much more you need, not the final value].

With only 2Gb of swap, that means you can only have 2Gb of virtual objects since all virtual objects must reserve backing store. [File backed objects such as shared mmap() don't use swap, they use the file -- anything allocated via malloc(), anonymous mmaps, stacks, SysV shared memory, etc. is swap backed]. This is your "reserve" line in swapinfo. Since 2048 of your swap total is not enabled on the system, when your reserve hits 2048 as well -- all your swap is reserved and no new virtual objects are likely to be made. This is when you will see failures from fork() / malloc() / whatnot.
That's why you never hit 100% for USED by the way -- swapinfo doesn't represent "There but not Enabled (and hence not USED)" all that well, in my opinion [but I'm just a kernel guy, not a UI person]. Since 50% of your swap can't be USED but isn't free either, your real total available is only 50% until you get that tunable up.

First, set the tunable like the system is trying to tell you to do (at least 2048). After that, you can look at your reservation peaks (since you obviously have a transient virtual memory load when the NNM stuff runs) to see if you need to enable more swap, or look at kernel tunables [maxdsiz, maxssiz] / setrlimit usage to limit how much virtual address space (and hence swap reservations) an allocation can use. I'd set the swapmem_on tunable to 1 as well so you can have memory swap enabled.

Re: SWAP error - Plz help

Gops_1 — Fri, 25 Apr 2008 10:36:44 GMT

Thank you so much morris. I tried opening SAM to change the kernel but its not opening. Can u provide me the exact command to change the parameter to 2048. Also tell me whether i need to do a restart of the machine.

Thaanks
gV

Re: SWAP error - Plz help

Yogeeraj_1 — Fri, 25 Apr 2008 10:46:53 GMT

hi gV,

Unfortunately, maxswapchunks cannot be dynamically changed:

(this can be observed by running the command below)
SRV2:home/root>kmtune -q maxswapchunks
Parameter Current Dyn Planned Module Version
===============================================================================
maxswapchunks 16384 - 16384
SRV2:home/root>kmtune -s maxswapchunks=
SRV2:home/root>

What error are you getting when running SAM?

to set the value, you can try the following comamnd:
SRV2:home/root>kmtune -s maxswapchunks=

hope this helps!

kind regards
yogeeraj

Re: SWAP error - Plz help

Yogeeraj_1 — Fri, 25 Apr 2008 10:51:09 GMT

hi again,

Sorry. I am not sure it the command will work on 11.0. I tried the commands in my previous post on 11.11. I do not have a 11.0 at hand right now.

Anyway, if you fix the SAM problem, you will be able to change the value correctly. Update us on the error message that you are getting when running SAM.

kind regards
yogeeraj

Re: SWAP error - Plz help

Gops_1 — Fri, 25 Apr 2008 10:54:29 GMT

It works thank yogeeraj. Do i need to do a restart of the my HP UX server ?

reply soon..

Thanks
gV

Re: SWAP error - Plz help

Yogeeraj_1 — Fri, 25 Apr 2008 11:06:23 GMT

Yes. You will have to do so for the parameter to take effect. As i said, this kernel parameter cannot be change dynamically (in version 11.11 - confirmed)

good luck!
kind regards
yogeeraj

Re: SWAP error - Plz help

Gops_1 — Fri, 25 Apr 2008 11:09:02 GMT

After i changed the parameter now am not getting into the server. Whenever i do ssh it say the following error.

Received disconnect from 202.61.9.20: 2: fork failed: Resource temporarily unavailable

Any help...

Thanks gV

Re: SWAP error - Plz help

YAQUB_1 — Fri, 25 Apr 2008 11:10:23 GMT

Changes to this tunable take effect at the next reboot.

Re: SWAP error - Plz help

Yogeeraj_1 — Fri, 25 Apr 2008 11:17:27 GMT

hi,

Sounds as if your process table is full.

glance -t

this should show you the real problem.

e.g. (glance -t output)
--------------------------------------------------------------------------------
SYSTEM TABLES REPORT Users= 2

System Table Available Used Utilization High(%)
--------------------------------------------------------------------------------
Proc Table (nproc) 4096 177 4 4
File Table (nfile) 63498 674 1 1
Shared Mem Table (shmmni) 512 10 2 2
Message Table (msgmni) 4096 2 0 0
Semaphore Table (semmni) 4096 21 1 1
File Locks (nflocks) 4096 26 1 1
Pseudo Terminals (npty) 60 0 0 0
Buffer Headers (nbuf) na 67320 na na

revert!

kind regards
yogeeraj

Re: SWAP error - Plz help

Gops_1 — Fri, 25 Apr 2008 11:32:52 GMT

B3692A GlancePlus C.03.55.00 18:00:54 pnmp2s 9000/800 Current Avg High
--------------------------------------------------------------------------------
CPU Util SU | 1% 12% 66%
Disk Util F | 1% 9% 31%
Mem Util S SU UB B | 34% 34% 35%
Swap Util R R | 5% 5% 5%
--------------------------------------------------------------------------------
SYSTEM TABLES REPORT Users= 1

System Table Available Used Utilization High(%)
--------------------------------------------------------------------------------
Proc Table (nproc) 1460 85 6 6
File Table (nfile) 21111 552 3 3
Shared Mem Table (shmmni) 200 6 3 3
Message Table (msgmni) 50 2 4 4
Semaphore Table (semmni) 64 20 31 31
File Locks (nflocks) 200 22 11 11
Pseudo Terminals (npty) 60 1 2 2
Buffer Headers (nbuf) na 8631 na na

This is my glance -t output.

Even after i changed those two kernel parameter as given morris doesn't helped me.
First of all my ping itself is not working. Whenever i do ping it just hangs...

thanks
gV

Re: SWAP error - Plz help

YAQUB_1 — Fri, 25 Apr 2008 11:44:46 GMT

send us "kctune" or "kmtune" output....

"swchunk" should be considered when modifying "maxswapchunks". BTW, put the output....

Re: SWAP error - Plz help

Ganesan R — Fri, 25 Apr 2008 11:48:32 GMT

Try to capture the syslog messages and post it. it may help us what is going on the server