Machine crawls

[brian_31]

my 11.11 machine is very slow even if i shutdown and restart SAP does not help..below are some figures i have gathered

# vmstat 2 30
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
1 0 0 2164661 58412 57 33 138 83 69 0 1484 1750 27960 1516 11 2 87
1 0 0 2164661 58336 43 8 25 0 0 0 0 1042 20439 2306 1 0 99
1 0 0 2164661 58320 28 4 18 0 0 0 0 1035 14084 1674 2 0 98
1 0 0 2164661 58305 23 3 14 0 0 0 0 1031 9384 1212 0 0 100
1 0 0 1708724 58288 93 17 14 0 0 0 0 1036 8289 978 0 1 98
1 0 0 1708724 58281 71 15 12 0 0 0 0 1029 6116 804 0 0 100
1 0 0 1708724 58280 46 9 7 0 0 0 0 1030 4743 659 1 0 99
1 0 0 1708724 58280 36 7 4 0 0 0 0 1003 6071 503 18 0 82
1 0 0 1708724 58275 35 9 4 0 0 0 0 1012 4976 438 1 0 99
1 0 0 1169129 58211 23 5 2 0 0 0 0 1013 3451 335 0 1 99
1 0 0 1169129 58208 76 16 4 0 0 0 0 1015 3943 319 0 0 100
1 0 0 1169129 58212 99 28 7 0 0 0 0 1028 4629 311 1 1 98
# dmesg |grep -i phy
physical page size = 4096 bytes, logical page size = 4096 bytes
Physical: 8386560 Kbytes, lockable: 6443724 Kbytes, available: 7402132 Kbytes
# ipcs -ma
IPC status from /dev/kmem as of Mon Dec 18 18:56:30 2006
T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME
Shared Memory:
m 0 0x410c0904 --rw-rw-rw- root root root root 0 348 757 757 22:28:35 22:28:35 22:28:29
m 1 0x4e0c0002 --rw-rw-rw- root root root root 1 61760 757 757 22:28:31 22:28:35 22:28:29
m 2 0x411402bd --rw-rw-rw- root root root root 1 8192 757 769 22:28:31 22:28:29 22:28:29
m 15363 0x5e0c5618 --rw------- root root root root 1 512 2001 2001 22:29:45 no-entry 22:29:45
m 60932 0x6f01d604 --rw-rw---- orapd0 dba orapd0 dba 58 1946165248 3125 8060 18:54:04 18:54:04 22:52:59
m 5 0x00004dbe --rw-rw-rw- root sapsys root sapsys 1 313048 3407 27955 18:56:19 18:56:19 22:54:14
m 518 0x00000000 --rw-rw-rw- pd0adm sapsys pd0adm sapsys 1 1024 3450 3450 22:54:25 no-entry 22:54:25
m 7 0x0382be84 --rw-rw-rw- pd0adm sapsys pd0adm sapsys 47 4096 3456 3516 22:54:47 no-entry 22:54:25
m 8 0x00002749 --rw-r----- pd0adm sapsys pd0adm sapsys 44 1090548 3483 8039 18:54:38 18:54:38 22:54:43
m 2569 0x0000271a --rw-r----- pd0adm sapsys pd0adm sapsys 45 124000000 3483 3517 22:54:49 no-entry 22:54:43
m 1034 0x00002711 --rw-r----- pd0adm sapsys pd0adm sapsys 46 572 3483 3446 18:56:01 18:56:01 22:54:45
m 1035 0x00002712 --rw-r----- pd0adm sapsys pd0adm sapsys 46 54627976 3483 3446 18:56:01 18:56:01 22:54:45
m 1036 0x00002713 --rw-r----- pd0adm sapsys pd0adm sapsys 46 13059200 3483 3446 18:56:01 18:56:01 22:54:45
m 1037 0x00002744 --rw-r----- pd0adm sapsys pd0adm sapsys 46 72968 3483 3446 18:56:01 18:56:01 22:54:45
m 1038 0x0000272f --rw-r----- pd0adm sapsys pd0adm sapsys 46 5771456 3483 3446 18:56:01 18:56:01 22:54:45
m 1039 0x00002743 --rw-r----- pd0adm sapsys pd0adm sapsys 43 23619684 3483 8039 18:54:05 18:54:05 22:54:45
m 1040 0x00000000 D-rw-rw-rw- pd0adm sapsys pd0adm sapsys 43 2147483648 3483 3483 22:54:45 no-entry 22:54:45
m 529 0x0000274e --rw-r----- pd0adm sapsys pd0adm sapsys 44 86132792 3483 8039 18:54:05 18:54:05 22:54:45

from the pcs are there any memory which are not released..how do i find out? what to look

Thanks

brian

[TwoProc]

Do you have Glance? (There is a free trial version on your Applications CD's).

Can you see if you are disk bound or cpu bound?

The only thing I can glean from the above is that for an 8G machine you've got a buffer cache for Oracle that is 1.9 GB - could be a little larger, but without knowing more, I wouldn't go there yet.

Have you runs a statspack report for the period of time during the slowdown?

Have you looked at the worst queries running in Oracle affecting your system load ?

Your problem is probably a SQL tuning one - just a guess b/c you're running a SQL database...

Need more information.

[Bill Hassell]

As always, the machine is slow when it is expected to return results faster. There is virtually nothing you can do to fix a poorly tuned database or serial searches. Getting rid of unused shared memory segments won't help much since the idle areas will be paged out if necessary. The task before you requires a LOT more information and if you use sar, vmstat and top, it's going to take a long time to figure out a plan. As mentioned, Glance is the tool of choice for the OS while Statspak from Oracle will be needed to analyze the SQL code.

Just a note about extra shared memory segments -- never use kill -9 for any database or SAP processes. It is fairly difficult to identify orphaned segments.

[Raj D.]

Brian,

Looks like your swap usage is more and and excessive swap usage can make the system crawl. Check the pi and po values on vmstat , any non zero value shows heavy swap usuage. For the output it shows pi value is more. You can check through glance as well.

Cheers,
Raj.

[Bill Hassell]

Actually, you just ignore the pi column in ps. page-in has two very different meanings. When a program is first started, the kernel will page in the program from the original executable. And if a program has some or all of it's page stored in swap, then returning those pages to memory is also a page in function. Thus, pi is perfectly normal when there is zero usage in swap. Always refer to the po (page out) numbers which refer to actual memory pressure actions.

[John Guster]

ipcs -mopb to check the column of NATTCH, if there is any zero number of processes. These are the memory not being used nor leased. Use ipcrm to release these memory.
run top to see which processes are running
Cheers.