emergency quesction,Experts come in please!!

[FireDragon.Xu_1]

Recently our system has been unstable because of an application that exhausts all the swap space.

Nobody can login whenever it is running. I met the following message in my last login attempt:

Pid 1040 received a SIGSEGV for stack growth failure.

Possible causes: insufficient memory or swap space,or stack size exceeded maxssiz.

I would like to ask you several questions:
1. I sent an email to HP and was told to add physical memory. I wonder if I can solve the
problem by simply adding additional swap memory instead of physcial memory or modifying
kernel parameters?

2. I wonder if I can assign sepcific CPU time and swap space to the application so that it
does not eat up all the system resources. The running time of the application is not an issue.

3. Where can I find information about usage of Glance?I can not find the some useful message on docs.hp.com I needed.

4. Is it possible to make the application send an email whenver it starts?

Best regards.

FireDragon.Xu

[Steven E. Protter · Accepted solution]

1)Run swapinfor -tam to check swap utilization and total vailable memory, which is defined as physical memory plus swap.

If you arleady have swap at twice physical memory, and are using most of it, increasing swap will severely impact performance. Physical memory is the way to go.

Gather some performance data with the scripts I'm attaching and see if you need further kernel work. See attachement.

2) No.

3) Glance will help, my sar scripts will help more. There is a report in Glance on swap usage.

4) Yes.

appp_script 2>&1 mailx -s "The app started ...." yourname@yourdoman.net

SEP

[T G Manikandan]

insufficient memory or swap space,or stack size

This could be of two reasons

1.The amount of memory/swap on the server is very less

2.The kernel parameter maxssiz limits are crossed.

you can just do a
swapinfo -ta
and find out the memory and swap usage.
If their usage is high then you need to increase the amount of free memory on the machine.

If you have plenty of free memory/swap then tune the kernel parameter

maxssiz-->32 bit application
maxssiz_64-->64 bit application

Also make sure you do not have more memory allocated for buffer cache.

What is your system?how much memory you have?What is the value of dbc_max_pct on the kernel.

Post the output of
#swapinfo -ta
Values of maxssiz,maxssiz_64,dbc_max_pct on the kernel.

to allocate memory and CPU to a particular application/process you can check the

Process Resource Manager from HP.

[Ian Dennison_1]

Some questions,...

1. What Application are you running? Does this application have a configurable memory component? If you own the source code to the application, can you tune the application to work in a more efficient way? (Fix the problem, not the symptom)
Have you investigated the forum or support site for the Application, to see if this is a known bug or if there is patch sets available?

2. Can you post a copy of the following commands,....

sysdef
top
vmstat 5 5
sar -w 2 5
sar -v 2 5
swapinfo -ta
ps -eafl

3. Glance manuals can be viewed on the following link,...

http://ovweb.external.hp.com/lpe/doc_serv/

Please provide more input for us to assist you.

Share and Enjoy! Ian

[T G Manikandan]

Get the documentation here for Glance

http://ovweb.external.hp.com/lpe/doc_serv/

[V.Tamilvanan]

Hi,

1.If your system is not swapping or paging out . You can add swap memory instead of physical memory. Check man pages of sar -w and vmstat .
2. Yes u can assign specific CPU time. If you have Procees Resource Manager Software (Add on license required)
3.You can use the online help which is bundled with Glance itself. For more details
http://www.openview.hp.com/products/gplus/index.html
4. Yes it is possible . You can do it with scripting itself.

HTH

[Michael Tully]

Yes you could add additional swap space, but depending on the application this may only provide a very short term solution. Adding additional swap may need an increase to 'maxswapchunks' which will require a reboot.

Changing CPU and to a lesser extent to allow your application to only get a certain percentage of resources requires the use of PRM (process resource manager) or workload resouce manager license (not free)

You can install a 60 trial version from your application CD set. You can get additional information on it from here: http://managementsoftware.hp.com/products/gplus/index.html

If your talking about sending an email when your application starts, you will need to modify the start-up script that is used.

Regards
Michael
"When I have trouble spelling, it's called fat finger syndrome"

[T G Manikandan]

PRM

http://docs.hp.com/cgi-bin/onlinedocs.py?mpn=B8733-90001&service=hpux&path=../B8733-90001/00/00/1&title=HP%20Process%20Resource%20Manager%20User%27s%20Guide

[FireDragon.Xu_1]

I have read all of your suggestions and heartly appreciate your help.

We run e-business software on our mainframe. The software includes webserver and BV1T01(broadversion). I have assigned additional 1 GB harddrive space for the swap.Based on two days' observation, it seems that the situation has been improved.

The following was obtained using
swapinfo -atm:
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 2400 963 1437 40% 0 - 1 /dev/vg00/lvol2
dev 1024 1022 2 100% 0 - 0 /dev/vg00/swapExtend
reserve - 658 -658
total 3424 2643 781 77% - 0 -

The newly added swap space was exhausted. The system condition still is not well.

As Mr. S. Protter suggested, if this still persists after adding additional swap space, then the only solution is to add physical memory.

Theoretical the physical memory and the swap space is sufficient. I do not understand why this software causes so much pressure on my system. It had run for one year without any problem. The problem only surfaced in recent months. I believe the problem is not related with the OS and the hardware. I wonder if it has anything to do with the access volume to our website.

Is there anybody who is familiar with the bv1to1? I have already cut the number of bv1to1 engines by half but the situation did not improve.

I'd appreciate any help from you!

FireDragon.Xu

[Sridhar Bhaskarla]

Hi,

I haven't worked on this bv1to1. However, your condition may indicate a probable memory leak.
To find it out, keep track of the following output.

UNIX95= ps -e -o "vsz args"
(note the space after = and ps)

Collect the output at regular intervals and see if the vsz values for your application processes are gradually increasing. If so, then there is memory leaks.

Also keep observing 'vmstat 5 10' and see if there are high 'po' value. If you have glance, type "m" and observe the pageouts and deactivations. If you see high values for pageouts or any value in deactivations, then you have to add memory.

-Sri

[Bill Hassell]

You have a very serious problem with the amount of RAM since there is so much swapping taking place. I would expect that this server is horribly slow and can be verified with vmstat, looking at the po (page out) column. If it is larger than 9 or so (like 250) then you need a very large increase in RAM. Yes, you can add 20 or 30Gbytes to swap and the problem can be averted because there will be enough to fit everrything into virtual memory. But this much swapping can be a 100:1 to 300:1 penalty against performance. Swap usage is not usually a desirable situation.

You should probably add 4 to 8Gb of additional RAM. To see which programs are taking up the most space, sort the processes by size:

UNIX95= ps -eo "vsz,pid,args" | sort -rn | more

The VSZ column is approximately the amount of kilobytes needed by the program. Another problem is that your buffer cache is far too large. Change the max and min percentages to create a max cache of 200 to 500 megs. And the BroadVision application may use shared memory and/or memory-mapped files, all leading to severe memory starvation. BroadVision support can tell you how much RAM you will need.

And as mentioned, the applications may be hitting a kernel fence that is too low. Look at maxdsiz and maxssiz (or the 64bit versions of the fences if the applications are 64bit). The defaults are VERY small for large applications. Change maxdsiz to 900 megs and maxssiz to perhaps 16 megs. However, BroadVision should be able to tell you the maximum sizes for these applications.

[yogesh_4]

hi ,
you are getting segment violation error. which can be solved by setting maxssiz_64 for 64bit OS parameter provided you are not running out of memory. Pls check the it by #swapinfo -tam command.

2. Regarding Glance you will get best info at http://ovweb.external.hp.com/lpe/doc_serv/