topic Re: Memory & swap space in Operating System - HP-UX

Memory & swap space

KINGSLEY_1 — Tue, 28 Sep 2010 10:48:44 GMT

Hi Sirs,

We are running T24 banking application. At peak period the memory ultilization is 99% and the swap is 42%. What should i do to reduce the memory ultilization so that my database does not crashed.

Thank you very much.

Re: Memory & swap space

Hakki Aydin Ucar — Tue, 28 Sep 2010 11:00:16 GMT

in the first place you have to check which OS you have:

# uname -a

if you have 11iv3, check this info;

# machinfo

in generally speaking, you need to extend memory (RAM) , reduce the number of non-critical application schedule non- busy hours if available.

Re: Memory & swap space

KINGSLEY_1 — Tue, 28 Sep 2010 11:08:06 GMT

Sirs,

The OS is 11.23 and phisical memory is 16GB.

Thanks.

Kingsley

Re: Memory & swap space

James R. Ferguson — Tue, 28 Sep 2010 11:18:55 GMT

Hi:

You don't offer any details. Do you get ENOMEM errors? What does 'dbc_max_pct' and 'dbc_min_pct' look like?

# kmtune -q dbc_max_pct
# kmtune -q dbc_min_pct

What does swap configuration and utilization look like?

# swapinfo -tam

Do you see page-out activity (indicating swap pressure)?

# vmstat ...

Do you have a database backend with its own buffer cache (e.g. Oracle)?

Regards!

...JRF...

Re: Memory & swap space

sarfaraj ahmad — Tue, 28 Sep 2010 11:26:00 GMT

what is the buffer cache status now in the syetem and how much memory it is taking?

also check dbc_max_pct and dbc_min_pct kernel parameter in the system.

closely monitor through glance and check is there any unnecessary process is running on that time?

also let us know process or load has been increased on the server now?

Re: Memory & swap space

KINGSLEY_1 — Tue, 28 Sep 2010 11:36:23 GMT

Dear Sirs,

The following are the readings.

Thank you.

Kingsley

dbc_max_pct 50 Default Immed

dbc_min_pct 5 Default Immed

#swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 16384 2972 13412 18% 0 - 1 /dev/vg00/lvol2
reserve - 6322 -6322
memory 16363 4488 11875 27%
total 32747 13782 18965 42% - 0 -

#vmstat
procs memory page
faults cpu
r b w avm free re at pi po fr de sr in
sy cs us sy id
1 2 0 1174466 47810 124 2 10 10 1 0 84 5840
45456 1522 7 5 88

Database: oracle 10g

Re: Memory & swap space

KINGSLEY_1 — Tue, 28 Sep 2010 11:55:23 GMT

Dear Sirs,

Below is the readings.

Thank you

kingsley

dbc_max_pct 50 Default Immed

dbc_min_pct 5 Default Immed

#swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 16384 2972 13412 18% 0 - 1 /dev/vg00/lvol2
reserve - 6322 -6322
memory 16363 4488 11875 27%
total 32747 13782 18965 42% - 0 -

#vmstat
procs memory page
faults cpu
r b w avm free re at pi po fr de sr in
sy cs us sy id
1 2 0 1174466 47810 124 2 10 10 1 0 84 5840
45456 1522 7 5 88

Database: oracle 10g

Re: Memory & swap space

Patrick Wallek — Tue, 28 Sep 2010 11:58:26 GMT

>>dev 16384 2972 13412 18% 0 - 1 /dev/vg00/lvol2

Since you are actually using some device swap space, I would add more RAM to your system. You are using almost 3 GB of device swap, so you need at least that much more RAM. For sufficient growth, if your system can handle it I would add 8 GB or 16 GB more RAM for a total of 24 or 32 GB.

Re: Memory & swap space

Hein van den Heuvel — Tue, 28 Sep 2010 14:08:15 GMT

>> so that my database does not crashed.

Did it? What errors? ORA-600 ?

>> At peak period the memory ultilization is 99%

It is easy to say, 'just add more memory', and 'turn down the unified buffer cache'.
And maybe that is the solution.
But please before you go there take a step back, test and verify.
What is using the memory?
- process memory?
- shared memory? ( check: ipcs -m )
- ...

That said, here would be my WAG towards what _might_ be a component of the issue.

You mention Oracle database.
That is likely significant, please tell us more.
How many instances? How much memory is each allowed to take ? (sga_max_size/sga_target)
Do they really need all of that (check SGA Target Advisory in AWR report)
Do they all need to be there are the same time?

Maybe, just maybe, it is simply over committed memory: 16 GB, 3 Oracle instances of 5 GB don't fit.
Maybe reducing those footprints to 3 GB each and all it well.

Or maybe you had PROD, DEV and QA all set the same and DEV and QA can be reduced to 1/2 the size of PROD.

Or maybe shutting down one DB during a critical window is the way to reduce the need.

One needs to understand the way the system is used to understand the options.

If the DBA is not forthcoming with help, then you can start understanding this studying the ipcs -m output.

Hope this help some,
Good luck!
Hein van den Heuvel
HvdH Performance Consulting

Re: Memory & swap space

KINGSLEY_1 — Tue, 28 Sep 2010 14:53:05 GMT

Dear sirs,

This are the details you asked for:

Physical memory: 16GB

instance: 1

instance size: 126GB

sga Maxsize: 6GB

SGA Target: 4GB

#ipcs -m
IPC status from /dev/kmem as of Tue Sep 28 15:32:16 2010
T ID KEY MODE OWNER GROUP
Shared Memory:
m 0 0x411c11ec --rw-rw-rw- root root
m 1 0x4e0c0002 --rw-rw-rw- root root
m 2 0x412016a9 --rw-rw-rw- root root
m 3 0x06347849 --rw-rw-rw- root root
m 262148 0x0c6629c9 --rw-r- root root
m 32773 0x4914685c -rw-r--r-- root root
m 4718598 0x7d0a116c -rw-rw oracle oinstall

Re: Memory & swap space

Dennis Handly — Tue, 28 Sep 2010 16:00:54 GMT

>#swapinfo -tam
memory 16363 4488 11875 27%
total 32747 13782 18965 42%

I don't see that 99% memory usage? Was that during the peak?

Re: Memory & swap space

KINGSLEY_1 — Tue, 28 Sep 2010 16:56:40 GMT

dear Sirs,

I see the 99% when i use glance.

Thanks

Re: Memory & swap space

Hein van den Heuvel — Tue, 28 Sep 2010 18:30:57 GMT

> Physical memory: 16GB
> instance: 1
> instance size: 126GB
> sga Maxsize: 6GB
> SGA Target: 4GB

Ok, 6GB SGA for 16 GB system feels alright. So that's not directly the image.

We'll have to figure out where the rest of the memory went. With dbc_max_pct @50, it could be sitting in the buffer cache to the tune of 8GB, ready to help or or give back under pressure. With 11.31 the giving back works well. With 11.23 it is my understanding that it is less effecient, and thus you may want to reduce dbc_max_pct to say 10%.

#ipcs -m

Ooops, I should have asked ipcs -ma or -mb
That will give a size column

Anyway, I'm not yet convinced there even is a problem. So the memory is in high use, with low free. Great! The system is using the memory that was bought for it. Excellent.

What is the problem?
Performance issues? How do you tell? Tell us.
Instability (why is that due to low memory)?
Is there a high PI or PO rate during regular work?

Regards,
Hein

Re: Memory & swap space

KINGSLEY_1 — Wed, 29 Sep 2010 11:11:17 GMT

Dear Sirs,

Our banking application hangs during peak hours. And the memory ultilisation rises to 99% when monitered with glance.

If i may ask , can i reduce " dbc_max_pct 50 Default Immed" to let us sy 15 or 20? And what will be the implication on the banking application and also the oracle database?

Thank you.

Kingsley

Re: Memory & swap space

James R. Ferguson — Wed, 29 Sep 2010 11:32:46 GMT

Hi (again):

If i may ask , can i reduce " dbc_max_pct 50 Default Immed" to let us sy 15 or 20? And what will be the implication on the banking application and also the oracle database?

I think that would be a good move to make. Since Oracle has its own buffer cache, caching more of the same in the UNIX buffer cache is wasteful. You could monitor read-hit ratios with 'sar -b' before and after adjusting the 'dbc_max_pct' if you like. Empirical measurements will tell you the implication of your change in *your* environment.

Regards!

...JRF...

Re: Memory & swap space

Hein van den Heuvel — Wed, 29 Sep 2010 12:39:46 GMT

>> Our banking application hangs during peak hours.

define 'hang'! We all have our own understanding/expectation for that.

I suspect that here it means that all end user applications became unresponsive.

But what it the system doing at that time besides using memory?

Why do you thing high memory use is relevant at all for the purpose of the hang? Please be specific!

- Is the CPU busy? top?
- Apparently you can use GLANCE.
- What else can you use? df? ls -l for Oracle files? df ?...
- What can you NOT use... (errors, or getting stuck).
- Are all Networks operational (ping, telnet, ftp, ssh, tnsping, remote non-application table oracle access)
- What does Oracle (Enterprise Manager) report when/while it hangs?
- Is Oracle responsive to local SQLplus queries at all? (select * from v$instance)
- Can local Oracle execute some banking query

Just the fact that you only mentioned 'hang' without specifics other then a potentially irrelevant high memory usage suggest to me that you may want to escalate this issue, and get more external and internal help.

You may want to prepare some answers to the questions above to get the best possible help. For now it is not clear whether yo shoudl engage an Oracle, Network, Storage or HPUX expert. Personally I would engage them in the order I list them. yes, that impleis that I think this is least likely to be an HPUX problem, allthough a good HPUX resource may help yo pinpoint the problem.

Good luck
Hein

Re: Memory & swap space

KINGSLEY_1 — Wed, 29 Sep 2010 13:47:57 GMT

Dear Sirs,

One more hint. During peak hours, when the banking application slows, syslog.log reads

"Sep 28 16:12:25 t24dbpro inetd[1057]: chargen/tcp: accept: No buffer space available

Sep 28 16:12:25 t24dbpro inetd[1057]: discard/tcp: accept: No buffer space available

Sep 28 16:12:25 t24dbpro inetd[1057]: daytime/tcp: accept: No buffer space available

Sep 28 16:12:25 t24dbpro inetd[1057]: telnet/tcp: accept: No buffer space available

Sep 28 16:12:25 t24dbpro sshd[931]: error: accept: No buffer space available

Sep 28 16:12:25 t24dbpro inetd[1057]: ftp/tcp: accept: No buffer space available

Sep 28 16:12:26 t24dbpro inetd[1057]: swat/tcp: accept: No buffer space available

Sep 28 16:12:26 t24dbpro inetd[1057]: dtspc/tcp: accept: No buffer space available

Sep 28 16:12:26 t24dbpro inetd[1057]: printer/tcp: accept: No buffer space available

Sep 28 16:12:26 t24dbpro inetd[1057]: auth/tcp: accept: No buffer space available

Sep 28 16:12:26 t24dbpro inetd[1057]: exec/tcp: accept: No buffer space available

Sep 28 16:12:26 t24dbpro inetd[1057]: shell/tcp: accept: No buffer space available

Sep 28 16:12:26 t24dbpro inetd[1057]: login/tcp: accept: No buffer space available

Thank you

kingsley

Re: Memory & swap space

Hein van den Heuvel — Wed, 29 Sep 2010 14:02:27 GMT

Hmm,

Does that perhaps also suggest that _some_ (new) end users hang, but others can keep working)?

This may be a TCP configrutation issues such as : tcp_conn_request_max

See for example:
http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=237561

In there Rick Jones (albeit in 2003) wrote
"Indeed, 99 times out of ten, this message has nothing to do with availability of memory. It means that by the time the server application got around to calling accept() on the listen socket, the remote client had given-up (for wahtever reason) and aborted the connection. The exact reason why an ENOBUF is returned in this case in HP-UX 11.X is starting to get lost in the mists of time"

also....

https://forums.sdn.sap.com/thread.jspa?threadID=1477203

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=969702

good luck,
Hein

Re: Memory & swap space

KINGSLEY_1 — Wed, 29 Sep 2010 16:14:45 GMT

Dear Sir,

tcp_conn_request_max=4096

Thank you

Kingsley

Re: Memory & swap space

chris huys_4 — Wed, 29 Sep 2010 18:41:51 GMT

Hi,

The problem is memory pressure.

Log a case with HP support, get the kmeminfo utility, issue kmeminfo;kmeminfo -user;swapinfo -tam, during a "hang", and this info should give enough indication as to what causes the problem and how to resolve it.

Greetz,
Chris