topic Re: swapinfo - Correct Parameter - Kernel in Operating System - HP-UX

swapinfo - Correct Parameter - Kernel

cfeitosa — Thu, 24 Jul 2008 15:10:00 GMT

Hello!

Please, I need a help.

I'm using HP-UX B.11.23 U ia64 with a model ia64 hp server rx2660.

We have 8GB of physical memory and I getting some messages:

"/oracle9i/app/oracle/product/920/backups/monit_arc.sh: cannot fork: no swap space
/oracle9i/app/oracle/product/920/backups/monit_arc.sh: cannot fork: no swap space

/usr/lib/hpux32/dld.so: Cannot map bss for library: mmap(0x0, 0xa128, 0x3, 0x52, -1, 0x0) returns Not enough space."

Our swap space:

u158 /# swapinfo -tan
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 8388608 0 8388608 0% 0 - 1 /dev/vg00/lvol2
dev 8388608 0 8388608 0% 0 - 0 /dev/vg00/lvol10
dev 8388608 0 8388608 0% 0 - 0 /dev/vg00/lvol12
reserve - 4255096 -4255096
total 25165824 4255096 20910728 17% - 0 -

In this moment it's look like good, but I would like to understand why the system isn't using the lvol's swap.

u158 /# swapon -p 1 /dev/vg00/lvol2
swapon: /dev/vg00/lvol2 is already enabled for paging.
u158 /# swapon -p 1 /dev/vg00/lvol10
swapon: /dev/vg00/lvol10 is already enabled for paging.
u158 /# swapon -p 1 /dev/vg00/lvol12
swapon: /dev/vg00/lvol12 is already enabled for paging.

u158 /# kmtune | grep swap
allocate_fs_swapmap 0 Default
dmp_swapdev_is_vol 0 Default
nswapdev 10 Default
nswapfs 10 Default
remote_nfs_swap 0 Default
swapmem_on 0 0
vol_swapdev_is_vol 0 Default

I need to change the "swapmem_on" parameter to 1?

Why theses areas shows kb used = 0??

Please, could someone give me a help?

Thanks!

Re: swapinfo - Correct Parameter - Kernel

Grayh — Thu, 24 Jul 2008 15:41:47 GMT

Still Ur system memory utilization 98%, pls check the system kernel parameter for below parameter value:
====================================
dbc_max_pct 50 - 50
====================================
If the value is 50, rearrange new value 8 or 10. After value change, required systems reboot.

After reboot the systems then execute top or glance command, hope fully Ur systems problem will be resolve.

***dbc_max_pct - the maximum percentage of memory to be used by dynamic buffer cache.

Values - Minimum - Maximum - Default
dbc_max_pct 2 - 90 - 50

But HP allows recommended 8 to 10, I have already inform U my previous email.

dbc_max_pct must be greater than or equal to dbc_min_pct.

Re: swapinfo - Correct Parameter - Kernel

cfeitosa — Thu, 24 Jul 2008 19:53:01 GMT

See...

dbc_max_pct 15 15

Re: swapinfo - Correct Parameter - Kernel

Venkatesh BL — Fri, 25 Jul 2008 02:36:36 GMT

Hi,

The 'Reserve' value alone shown as USED tells me that the system has not actually PAGED OUT anything from memory yet. This is a good thing!

The device to which a page goes to is not decided till the actual time of page out. So, till that point, the kernel just modifies the 'available swap space' counters to ensure that its usage is within the actual available space.

http://docs.google.com/View?docid=dc7s6mzr_56cgsg9g has some good info.

Re: swapinfo - Correct Parameter - Kernel

Venkatesh BL — Fri, 25 Jul 2008 03:06:21 GMT

Oops, I missed the first part of the query. The situation looks strange to me. Are you on the latest patch for 11.23? Did the problem go away after changing the dbc values?

Re: swapinfo - Correct Parameter - Kernel

V. Nyga — Fri, 25 Jul 2008 06:47:58 GMT

Hi,

check this doc:
http://www12.itrc.hp.com/service/cki/docDisplay.do?docId=emr_na-c00904819-7&docLocale=en

Check also your kernel settings as discribed in the doc.

HTH
Volkmar

Re: swapinfo - Correct Parameter - Kernel

V. Nyga — Fri, 25 Jul 2008 06:49:26 GMT

.. and yes - as the doc says - you should enable pseudo swap with swapmem_on to 1.

V.

Re: swapinfo - Correct Parameter - Kernel

Dennis Handly — Fri, 25 Jul 2008 09:08:42 GMT

You should also use "swapinfo -tam" so you can get the output in easier to understand Mb.

Re: swapinfo - Correct Parameter - Kernel

cfeitosa — Fri, 25 Jul 2008 11:35:58 GMT

Hi!

Thanks for all replies.

Patches installed:
u158 /# swlist | grep -i GOLD
u158 /#
u158 /# swlist | grep -i PACK
QPKBASE B.11.23.0712.070a Base Quality Pack Bundle for HP-UX 11i v2, December 2007
u158 /#

The doc that V.Nyga said I can't to access because I don't have privileges. So, please, could someone send me this doc in TXT file?

Thanks

Re: swapinfo - Correct Parameter - Kernel

Dennis Handly — Fri, 25 Jul 2008 12:01:36 GMT

>The doc that V.Nyga said I can't to access because I don't have privileges. So, please, could someone send me this doc in TXT file?

You must have a support contract to look at it. Sending/posting the document would violate copyright laws.

Re: swapinfo - Correct Parameter - Kernel

V. Nyga — Fri, 25 Jul 2008 12:36:54 GMT

Hi,

do you have a support contract?
In your profile go to SUM and add your contract number to your profile.
Then you can see Technical knowledge base - documents.

As you already asked above: set swapmem_on parameter to 1.

For big swap space you have to connfigure your 'maxswapchunks' kernel parameter.
What's your current setting?

V.

Re: swapinfo - Correct Parameter - Kernel

clefeitosa — Fri, 25 Jul 2008 13:44:30 GMT

Ok! Now I can see the doc.
Thanks!

So, In this moment I don't know if I have to use the pseudo swap (swapmem_on = 1) because I have three areas that I'm not using and I don't know why.

I still do no understand why the devices below are with MB USED = 0 (zero)

u158 /# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 8192 0 8192 0% 0 - 1 /dev/vg00/lvol2
dev 8192 0 8192 0% 0 - 0 /dev/vg00/lvol10
dev 8192 0 8192 0% 0 - 0 /dev/vg00/lvol12
reserve - 4194 -4194
total 24576 4194 20382 17% - 0 -

I have a shell that is monitoring the page in and page out and when I had problems (Not enough space) the system did not use the devices areas.

See the return of the shell:

16/07/08-19:43:00 82 18 0 0 7.87 100 2 288 369083 0 0
16/07/08-19:53:00 82 18 0 0 7.64 100 2 320 296896 0 0
16/07/08-20:03:00 79 21 0 0 7.18 100 2 374 204885 0 0
16/07/08-20:13:00 78 22 0 0 7.16 100 2 518 7689 0 0
16/07/08-20:23:00 78 22 0 0 7.20 100 2 650 7371 25 787
16/07/08-20:33:01 78 22 0 0 7.21 100 2 780 21264 17 0
16/07/08-20:43:00 78 22 0 0 7.15 100 2 743 10396 0 0
16/07/08-20:53:00 79 21 0 0 7.11 100 2 739 8150 0 0
16/07/08-21:03:01 75 25 0 0 7.07 100 2 739 9470 0 0
16/07/08-21:13:00 78 22 0 0 7.11 100 2 741 9461 0 0
16/07/08-21:23:00 78 22 0 0 7.12 100 2 737 9478 0 0

The last two colums show the page in and page out respectively. In this moment the page in was 25 and page out was 787 and the system did not use the lvol's swap devices.

Why??

I have only theses kernel parameters in my system. I don't have 'maxswapchunks' kernel.

u158 /# kmtune | grep maxswapchunks
u158 /#
u158 /# kmtune | grep swap
allocate_fs_swapmap 0 Default
dmp_swapdev_is_vol 0 Default
nswapdev 10 Default
nswapfs 10 Default
remote_nfs_swap 0 Default
swapmem_on 0 0
vol_swapdev_is_vol 0 Default

u158 /# kmtune | grep dbc_max_pct
dbc_max_pct 15 15 Immed

Please, could explain me?

I have other system SD 11.23 that shows MB USED = 97

u35:/# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 64000 97 63903 0% 0 - 1 /dev/vg00/lvol2
reserve - 25270 -25270
total 64000 25367 38633 40% - 0 -

The difference that I see:

u35:/# swlist | grep -i PACK
QPKAPPS B.11.23.0609.053a Applications Quality Pack Bundle for HP-UX 11i v2, September 2006
QPKBASE B.11.23.0609.053a Base Quality Pack Bundle for HP-UX 11i v2, September 2006

u158 /# swlist | grep -i PACK
QPKBASE B.11.23.0712.070a Base Quality Pack Bundle for HP-UX 11i v2, December 2007

I don't have the QPKAPPS in the server "u158".

What you think?

Thanks a lot!

Re: swapinfo - Correct Parameter - Kernel

Don Morris_1 — Fri, 25 Jul 2008 14:11:35 GMT

You _are_ using the device swap for reservation, that's what the "reserve" line means.

[See the Memory Management white paper for full details on how this works (http://docs.hp.com/en/1218/mem_mgt.html) but the short version is that new virtual allocations [malloc and friends] have to reserve swap before any actual use. So swap reservation space is roughly the total virtual address space for anything that isn't backed by a file on the entire system.]

When these problems happened -- was there a dramatically different workload running? (Say... something that might be creating a big shared memory segment? Huge stacks [what's maxssiz and maxssiz_64bit?] A bunch of malloc's? (maxdisz/maxdsiz_64bit caps this). Certainly your system looks fine now -- the question is what was eating a bunch of virtual space when you hit the failures.

Track the swapinfo reserve status over time and the related VSS of the largest (virtual) user processes to correlate this with what happens. If you have Glance, you should also be able to set alarms so you can catch the system at 90% swap reservation/consumption and take a look if I recall correctly.

Re: swapinfo - Correct Parameter - Kernel

Don Morris_1 — Fri, 25 Jul 2008 14:22:36 GMT

Oh, and don't bother with maxswapchunks -- as you've already noted, it is obsolete on your release. It isn't your problem anyway (if the devices were not fully added because of running out of metadata to represent them, you'd see that reflected in the swapinfo output).

Re: swapinfo - Correct Parameter - Kernel

V. Nyga — Fri, 25 Jul 2008 14:45:00 GMT

Maybe this helps:
http://www11.itrc.hp.com/service/cki/docDisplay.do?docLocale=en&docId=emr_na-c00958840-8

Try to find docs in the Technical knowledge base:
http://www11.itrc.hp.com/service/cki/enterService.do?

V.

Re: swapinfo - Correct Parameter - Kernel

Dennis Handly — Fri, 25 Jul 2008 23:01:09 GMT

>I don't know if I have to use the pseudo swap (swapmem_on = 1)

Yes, you'll still want to do that.

Re: swapinfo - Correct Parameter - Kernel

Venkatesh BL — Mon, 28 Jul 2008 02:25:58 GMT

Yes, you need to set 'swapmem_on' to 1. No doubt. It is just a mechanism to ensure that the kernel does not fail the memory requests when the swap space runs out. Because, if portion of the RAM is still available, the kernel can still use it, except that it has to 'wire' it now as there is no swap space to back it.

Re: swapinfo - Correct Parameter - Kernel

cfeitosa — Tue, 29 Jul 2008 14:28:59 GMT

maxssiz 401604608 401604608 Immed
maxssiz_64bit 2147483648 2147483648 Immed

Re: swapinfo - Correct Parameter - Kernel

Don Morris_1 — Tue, 29 Jul 2008 14:40:00 GMT

Well, I'd certainly lower maxssiz. Your original BSS mapping failure (BSS for a shared library is still a private data per-mapper if I recall correctly) was for a 32-bit app. You've got the stack pre-allocating around 400Mb of the private address space. By default, that only leaves around 600Mb for data/private mmaps, etc. Even assuming more interesting compilation options (like Share Magic), that would be just under 1/4th of the process private address space. Only using MPAS would make this a bit less.

If you don't need a huge stack, lower maxssiz to something more reasonable. I know Java and Oracle like big stacks... so check their documentation (I'd think 128Mb or so would be more than sufficient, but I didn't write their applications either).