high workloads causes the killing of applications

[Franky Leeuwerck]

Hello,

Everytime when there are high instantaneous workloads (lots of memory used) the system kills an application. In this case the victim is always an Ingres DBMS. Sudden outbursts of memory needs are seen when running Informix applications, Omniback backup application but also when running security analysis reports on the host.

The main Ingres DBMS processes are killed without any trace of error message.

Is there a way to avoid this and why does Ingres always has to be the victim.

Franky

[Steve Steel]

HI


PRM Process Resource Manager would probably help.


Are there any messages in /var/adm/syslog/syslog.log when process is killed.

Could be problems with tuning.


Steve Steel

[Franky Leeuwerck]

In the syslog there are no error messages, nor in the Ingres errlog.

[Klaus Crusius]

Hi,

you could try to check the swap space size. The glance utility can show you the actual size and current usage. Sam will help to increase. If you redirect stderr and stdout of the INGRES process to a dedicated place, is there some output? You could target truss to the process to find out, what were its last system calls.

mfg, Klaus

[Steve Steel]

Hi

If the machine beings to run out of shared memory the scheduler will terminate processes it feels to be unimprotant to save continuity in the machine.

Could you please post

1)swapinfo -ta

2)How much memory you have.

In the first 100 lines of syslog

3)kmtune -l

for swapmem_on maxswapchunks dbc_max_pct dbc_min_pct

4)/var/adm/syslog/syslog.log

The part just before and after the fault

Steve Steel

[Franky Leeuwerck]

Steve,

Only the kmtune command is not present.
A the phenomen occurred at 10h02 (syslog extract)

zhpux21:/ # swapinfo -ta
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 524288 16308 507980 3% 0 - 1 /dev/vg00/lvol2
dev 524288 16488 507800 3% 0 - 1 /dev/vg01/lvswap2
reserve - 231684 -231684
memory 241340 206004 35336 85%
total 1289916 470484 819432 36% - 0 -
zhpux21:/ #


vmunix: Memory Information:
vmunix: physical page size = 4096 bytes, logical page size = 4096 bytes
vmunix: Physical: 327680 Kbytes, lockable: 239908 Kbytes, available: 278164 Kbytes

May 6 09:48:45 zhpux21 inetd[20354]: registrar/tcp: Connection from zhpux21 (19
5.207.194.81) at Mon May 6 09:48:45 2002
May 6 09:50:46 zhpux21 inetd[20397]: registrar/tcp: Connection from zhpux21 (19
5.207.194.81) at Mon May 6 09:50:46 2002
May 6 09:52:00 zhpux21 inetd[20399]: telnet/tcp: Connection from dtcmanger (193
.74.166.31) at Mon May 6 09:52:00 2002
May 6 09:52:46 zhpux21 inetd[20403]: registrar/tcp: Connection from zhpux21 (19
5.207.194.81) at Mon May 6 09:52:46 2002
May 6 09:54:46 zhpux21 inetd[20416]: registrar/tcp: Connection from zhpux21 (19
5.207.194.81) at Mon May 6 09:54:46 2002
May 6 09:56:46 zhpux21 inetd[20435]: registrar/tcp: Connection from zhpux21 (19
5.207.194.81) at Mon May 6 09:56:46 2002
May 6 09:58:46 zhpux21 inetd[20493]: registrar/tcp: Connection from zhpux21 (19
5.207.194.81) at Mon May 6 09:58:46 2002
May 6 09:59:21 zhpux21 inetd[20802]: telnet/tcp: Connection from unknown (195.2
07.204.101) at Mon May 6 09:59:21 2002
May 6 10:00:46 zhpux21 inetd[21587]: registrar/tcp: Connection from zhpux21 (19
5.207.194.81) at Mon May 6 10:00:46 2002
May 6 10:02:46 zhpux21 inetd[23806]: registrar/tcp: Connection from zhpux21 (19
5.207.194.81) at Mon May 6 10:02:46 2002
May 6 10:03:19 zhpux21 inetd[24179]: ftp/tcp: Connection from p86533a (195.207.
204.221) at Mon May 6 10:03:19 2002
May 6 10:03:20 zhpux21 ftpd[24179]: connection from p86533a.dcz.bekaert.com at
Mon May 6 10:03:20 2002
May 6 10:03:24 zhpux21 ftpd[24179]: FTP LOGIN FROM p86533a.dcz.bekaert.com, eai
May 6 10:03:38 zhpux21 ftpd[24179]: PORT
May 6 10:03:49 zhpux21 last message repeated 2 times
May 6 10:04:09 zhpux21 ftpd[24179]: PORT
May 6 10:04:23 zhpux21 inetd[24306]: telnet/tcp: Connection from dcz01s30 (195.
207.194.202) at Mon May 6 10:04:23 2002
May 6 10:04:46 zhpux21 inetd[24462]: registrar/tcp: Connection from zhpux21 (19
5.207.194.81) at Mon May 6 10:04:46 2002
May 6 10:05:42 zhpux21 inetd[24515]: telnet/tcp: Connection from dtcmanger (193
.74.166.31) at Mon May 6 10:05:42 2002
May 6 10:06:46 zhpux21 inetd[24520]: registrar/tcp: Connection from zhpux21 (19
5.207.194.81) at Mon May 6 10:06:46 2002
May 6 10:08:46 zhpux21 inetd[24554]: registrar/tcp: Connection from zhpux21 (19
5.207.194.81) at Mon May 6 10:08:46 2002
May 6 10:10:46 zhpux21 inetd[25544]: registrar/tcp: Connection from zhpux21 (19
5.207.194.81) at Mon May 6 10:10:46 2002
May 6 10:12:47 zhpux21 inetd[25692]: registrar/tcp: Connection from zhpux21 (19
5.207.194.81) at Mon May 6 10:12:47 2002
May 6 10:14:47 zhpux21 inetd[25709]: registrar/tcp: Connection from zhpux21 (19
5.207.194.81) at Mon May 6 10:14:47 2002

[Dan Herington]

Franky,

Are you using PRM on this machine?

Dan

[Franky Leeuwerck]

Thanks for the question.
No, we are not using PRM ?

(By the way what is it exactly ?).

Franky

[Dan Herington]

We have two products that give you control over the amount of resources each of your applications can use. Process Resource Manager (PRM) is a tool you can use for what we call "Resource Partitioning" which basically means you can specify how much CPU, memory and disk i/o each group of processes can use. You can also specify what processes belong in each "group".

The other product is Workload Manager. You can specify priorities and performance goals for your applications and WLM will adjust the amount of CPU available to the app to ensure that the high priority goals are met. It does this by tweaking the PRM config.

PRM is included in the Enterprise OE and WLM is in the Mission Critical OE.

Based on what I have seen so far in this thread, it looks like an ingres bug. Have you seen any core droppings? You can run the command file against the core file to see what process generated it and possibly a reason why it died. If you find core files that belong to ingres, you should contact ingres support. Commercial products should never generate core files.

Dan

[Tim D Fulford]

I can't speak for Ingres, but Informix should not ave peaks in it's memory usage (I'm talking about Informix instance/database). What are the memory requirements for each app? Do you gate malloc messages? You may need more memory? This sounds like a problem with Ingres!

Tim

[Franky Leeuwerck]

Thanks for the answers.
In the meantime the problem has been solved.

According to the local HP support a bad mirroring disk caused excessive swapping, taking a lot of extra memory.


Franky