topic Re: Shared Memory - IPCS in Operating System - HP-UX

Shared Memory - IPCS

Ranjith_5 — Wed, 24 Aug 2005 03:30:34 GMT

Hi Gurus,

On my database server ipms -ma gives the following result. Is this any way affecting my system performance?

ULTRAMAX:root-/>ipcs -ma|grep root
m 0 0x4118025d --rw-rw-rw- root root root root 0 348 613 613 5:15:58 5:15:58 5:15:50
m 1 0x4e0c0002 --rw-rw-rw- root root root root 1 61760 613 613 5:15:54 5:15:58 5:15:50
m 2 0x411c0a3f --rw-rw-rw- root root root root 1 8192 613 625 5:15:54 5:15:50 5:15:50
m 8195 0x00005643 --rw-rw-rw- root root root root 1 1024 2583 2583 5:16:44 no-entry 5:16:44
m 4 0x00005654 --rw-rw-rw- root root root root 1 1024 2583 2583 5:16:44 no-entry 5:16:44
m 3077 0x4d4e5251 --rw-r--r-- root root root root 2 140288 2568 7129 23:56:55 no-entry 5:16:47
m 6 0x55315352 --rw-rw-rw- root root root root 1 4096 2568 2568 5:16:47 no-entry 5:16:47
m 7 0x44525354 --rw-r--r-- root root root root 3 393216 2568 4359 13:56:52 13:56:52 5:16:47
m 8 0x53494152 --rw-r--r-- root root root root 1 1024 2568 2568 5:16:47 no-entry 5:16:47
m 37897 0x0c6629c9 --rw-r----- root root root root 2 37790828 2871 12304 13:30:01 13:55:14 5:17:08
m 10 0x06347849 --rw-rw-rw- root root root root 1 77384 2871 2894 5:17:15 5:17:08 5:17:08
m 1035 0x49101e6f --rw-r--r-- root root root root 0 22908 2833 2894 13:57:01 13:57:01 5:17:10

Regards,
Syam

Re: Shared Memory - IPCS

Joseph Loo — Wed, 24 Aug 2005 03:50:53 GMT

hi syam,

ipcs will not tell u what is affecting your performance, it only reports active share memory segments (-m).

from the output, u should not grep only root as it do not tell us much about the other share memory, i.e. for oracle DB, grep the oracle user which the SEGSZ will show the amount of semaphores.

use vmstat for memory unless u have glance to monitor performance.

regards.

Re: Shared Memory - IPCS

Ranjith_5 — Wed, 24 Aug 2005 04:07:39 GMT

Hi,

here is the out put for oracle related process.

ipcs -ma|grep ora
m 240652 0x831883f8 --rw-rw---- oradb dba oradb dba 1659 6990901248 7815 6271 14:35:56 14:35:56 5:21:39
m 258063 0x3f5daca8 --rw-r----- oracle dba oracle dba 138 555585536 13901 6183 14:35:49 14:35:49 13:57:17

is this anyway affecting my performance. I have glance too on my server. How do I find out unutilised process & memory resources?

Regards,
Syam

Re: Shared Memory - IPCS

Ranjith_5 — Wed, 24 Aug 2005 04:13:11 GMT

in glance it shows that the disk utilisation is 100%. Is there anything fishy..?

regards,
Syam

Re: Shared Memory - IPCS

Alessandro Pilati — Wed, 24 Aug 2005 04:17:51 GMT

Syam,
taking a look to "ipcs -ma" output, take care that the oracle-related use a maximum of 70%-80% of total memory.
In any case, if there are performance problems, often you can find it regardless the ipcs output...
Other outputs are needed ( ex. vmstat , iostat, sar ecc... )
Post these other outputs to have a deeper analysis.

Rgds,
Alex

Re: Shared Memory - IPCS

Alessandro Pilati — Wed, 24 Aug 2005 04:24:03 GMT

Syam,
if there are disk problems, it can be due to an excessive swapping activity, or DB related problems such as a lost of an index and a consequent DB activity of scan all tables, or other causes...

Please post the output of
sar -d 2 15
to identify initialy the disk bottlenecked.

Rgds,
Alex

Re: Shared Memory - IPCS

Ranjith_5 — Wed, 24 Aug 2005 04:26:22 GMT

here are the outputs..please tell me if any detailed info is required.

ULTRAMAX:root-/>iostat -t
tty cpu
tin tout us ni sy id
0 238 29 118 -93 46

device bps sps msps

c0t6d0 0 0.0 1.0
c0t0d0 0 0.0 1.0
c1t0d0 0 0.0 1.0
c1t6d0 0 0.0 1.0
c16t1d0 0 0.0 1.0
c16t3d2 0 0.0 1.0
c16t2d6 0 0.0 1.0
c16t2d7 0 0.0 1.0
c16t4d2 0 0.0 1.0
c16t0d0 0 0.0 1.0
c16t0d1 0 0.0 1.0
c16t0d2 0 0.0 1.0
c16t0d3 0 0.0 1.0
c16t0d4 0 0.0 1.0
c16t0d5 0 0.0 1.0
c16t3d3 0 0.0 1.0
c16t3d4 0 0.0 1.0
c16t3d5 0 0.0 1.0
c16t3d6 0 0.0 1.0
c16t3d1 0 0.0 1.0
c16t1d1 0 0.0 1.0
c16t3d0 0 0.0 1.0
c16t4d1 0 0.0 1.0
c16t1d4 0 0.0 1.0
c16t1d5 0 0.0 1.0
c16t1d6 0 0.0 1.0
c16t1d7 0 0.0 1.0
c16t2d0 0 0.0 1.0
c16t2d1 0 0.0 1.0
c16t2d2 0 0.0 1.0
c16t2d3 0 0.0 1.0
c16t3d7 0 0.0 1.0
c16t4d0 0 0.0 1.0
c16t0d6 0 0.0 1.0
c16t0d7 0 0.0 1.0
c16t8d5 0 0.0 1.0
c16t2d4 0 0.0 1.0
c16t2d5 0 0.0 1.0
c16t8d6 0 0.0 1.0
c16t1d2 0 0.0 1.0
c16t1d3 0 0.0 1.0
c16t4d3 0 0.0 1.0
c16t4d4 0 0.0 1.0
c16t4d5 0 0.0 1.0
c16t4d6 0 0.0 1.0
c16t4d7 0 0.0 1.0
c16t5d0 0 0.0 1.0

======================================================

ULTRAMAX:root-/>vmstat
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
25 35 0 2859728 5419903 504 23 1 0 0 0 11 20729 177086 18968 46 5 50
=================================================
ULTRAMAX:root-/>sar -u 7

HP-UX ULTRAMAX B.11.11 U 9000/800 08/24/05

14:54:47 %usr %sys %wio %idle
14:54:54 63 11 21 4

Regards,
Syam

Re: Shared Memory - IPCS

Ranjith_5 — Wed, 24 Aug 2005 04:30:10 GMT

attached the sar -d 2 15 output.

Regards,
Syam

Re: Shared Memory - IPCS

Alessandro Pilati — Wed, 24 Aug 2005 05:12:41 GMT

Syam,
it seems you have some disk overloaded:
14:57:53 device %busy avque r+w/s blks/s avwait avserv

Average c16t0d0 86.15 0.50 584 9337 4.99 3.12
Average c16t3d3 83.95 0.50 463 13462 5.00 4.07
Average c16t3d5 99.37 0.52 658 10522 5.05 6.45
Average c16t3d6 80.35 0.50 488 10732 5.02 3.06
Average c16t1d6 85.25 0.50 873 13957 5.02 1.88
Average c16t4d3 99.90 0.58 1204 19264 5.14 4.08
Average c16t4d4 98.13 0.50 739 19656 5.08 3.65
Average c16t4d5 90.59 0.50 362 5803 5.01 6.08
Average c16t4d7 89.46 0.50 395 14425 5.09 5.31

Infact for these disks you have %busy near to 90-100% at any moment.

Have a look to the Logical Volumes owned by that disks with the command
pvdisplay -v /dev/dsk/
and see if they are Oracle dependent or other filesystems-dependent.

Please post the output of
swapinfo -t

Attach the output of
vmstat 5 20

Then launch the following commands:
sar -o sarfile 5 60
sar -Af sarfile > sar.report

and attach the sar.report file.

Rgds,
Alex

Re: Shared Memory - IPCS

Joseph Loo — Wed, 24 Aug 2005 05:15:24 GMT

hi,

just back from a meeting.

as what alessandro has mention, if performance is affected because of high disk usage, a possible problem may be excessive swapping.

# swapinfo -tam

to also look at paging problem:
# vmstat 3 10
run 10 of 3-seconds interval, look at the po column.

looking at your ipcs info on the oracle database, u have allocated a pretty big shared memory for oradb user, i.e. 6990901248 -> 7GB. any reasons why?

to guide u in performance monitoring, read these:

http://www1.itrc.hp.com/service/cki/search.do?category=c0&mode=id&searchString=UPERFKBAN00000726&searchCrit=allwords&docType=Security&docType=Patch&docType=EngineerNotes&docType=BugReports&docType=Hardware&docType=ReferenceMaterials&docType=ThirdParty&search.x=31&search.y=7

http://www1.itrc.hp.com/service/cki/search.do?category=c0&mode=id&searchString=KBRC00000947&searchCrit=allwords&docType=Security&docType=Patch&docType=EngineerNotes&docType=BugReports&docType=Hardware&docType=ReferenceMaterials&docType=ThirdParty&search.x=21&search.y=11

regards.

Re: Shared Memory - IPCS

Ranjith_5 — Wed, 24 Aug 2005 06:09:02 GMT

Hi ,

pleas give me some time to come back to you with the required outputs..some urgent work came in between...

Regards,
Syam

Re: Shared Memory - IPCS

Raj D. — Wed, 24 Aug 2005 07:16:47 GMT

Syam ,

Check and monitor your system performane through glance.

Also check
# sar -v 5 10 [ see system talble parameters, whether they are exhausting ..]
# vmstat 5 10 [ check if excessive swapping ..]
# iostat 5 20 [ Disk activity ]

From your attachment it seems disk acitivity is high, u can write a script and monitor which disk is highly loaded..

Hope this will help..
Cheers ,

Raj

Re: Shared Memory - IPCS

TwoProc — Wed, 24 Aug 2005 07:54:29 GMT

About the only thing you can glean from ipcs that's meaningful re: Oracle is that you've set your kernel parameters large enough so that your SGA isn't chunked up into a bunch of little pieces - and yours isn't, it's fine. The first number (~7G) is probably your SGA size, and the second is probably run-time heap.

Re: Shared Memory - IPCS

TwoProc — Wed, 24 Aug 2005 08:00:21 GMT

Run a couple of Oracle statspack snapshots and have a look at what's going on. It will report to you the queries with the highest disk I/O rates - pay extra attention to those. Also, review your cache hit ratios and the hit ratios for the shared pool and large pool. Make sure that these numbers are (and stay) in the high 90's. I suspect that they are not. That doesn't necessarily mean that they need to be bigger, it's probably that you've got some poorly running code in there, and it is doing so much disk i/o everything else tends to get bumped from cache. Statspack will give you the necessary information to get started.

Re: Shared Memory - IPCS

Ranjith_5 — Wed, 24 Aug 2005 08:27:46 GMT

Hi,

Attached the sar,vmstat,swapinfo outputs. Hope you can conclude something.

Regards,
Syam

Re: Shared Memory - IPCS

Alessandro Pilati — Wed, 24 Aug 2005 09:06:36 GMT

It seems you have not swap problem (
"PCT" for "dev" in swapinfo output is 0%
"po" in vmstat output is costantly 0
"swpot/s" ( Swapouts per second ) in sar output is costantly 0
"swpocc" (percent swap queue occupied) is costantly < 5
)

Your Buffer Cache seems ok (BUFPAGES parameter) (
"%rcache" is costantly > 90
"%wcache" is almost costantly > 65
)

Also your CPU charge seems normal.

I think there is an Oracle related problem that causes a overload on the following disks:
c16t0d0
c16t3d3
c16t3d5
c16t3d6
c16t1d6
c16t4d3
c16t4d4
c16t4d5
c16t4d7

Please take a snapshot for Oracle activity.
See what logical volumes are present in the disks above with the pvdisplay -v command and if you have DB-managed Logical Volumes check to which tables/indexes they correspond ( and wait for some deeper answer from some Oracle-Guru )

Hope this can help...
Rgds,
Alex

Re: Shared Memory - IPCS

Ranjith_5 — Wed, 24 Aug 2005 09:16:58 GMT

Hi,

I already spoke to the dba team about this they said it is because of load on the system and it is normal as per them. But apart from this do you find any other problem.

Regards,
Syam

Re: Shared Memory - IPCS

Alessandro Pilati — Wed, 24 Aug 2005 09:19:15 GMT

Syam,
I have also a very good Oracle Database tuning guide, unfortunately it is 5MB and I cannot attach it...

Re: Shared Memory - IPCS

Alessandro Pilati — Wed, 24 Aug 2005 09:23:06 GMT

Syam I don't see any other problem at all.
Your server doesn't seem to be CPU or MEMORY bound.
Ask to DBAs to do a DB analysis to check if is normal that Oracle create a so heavy access to those disks.

Re: Shared Memory - IPCS

Ranjith_5 — Wed, 24 Aug 2005 09:26:50 GMT

Alessandro Pilati,

Thanks for taking your time to analyse the problem..It was really great.thanks a lot.

Regards,
Syam