topic Re: Shared Memory error? in Operating System - HP-UX

Shared Memory error?

Filosofo — Thu, 22 May 2003 13:46:36 GMT

Hallo......I'm always Filo...unfortunatly..
I have in Single view Log this error :
SHM: Could not get shared memory segment with key 5C12397

The OS version is 11.11 .
I have a L3000 with 4Gb of memory.

Please help me.

Re: Shared Memory error?

Steven E. Protter — Thu, 22 May 2003 13:49:18 GMT

kernel

shmmax
shmseg

Might want to bump those up.

Might want to collect some performance data, script attached.

ipcs -mob

This will give you some data with which to act.

swapinfo -tam

SEP

Re: Shared Memory error?

Steven E. Protter — Thu, 22 May 2003 13:50:22 GMT

Sorry misssed one.

echo total_lockable_mem/D | adb /stand/vmunix /dev/mem

Post results so that someone can help you.

SEP

Re: Shared Memory error?

A. Clay Stephenson — Thu, 22 May 2003 13:58:10 GMT

Hi:

Nobody is going to be able to help you very much because there is so little useful data. Do an ipcs -ma. The key value you refer to is the hex 'KEY' column in this display. It is the value that a related group of processes agree upon so that they can all attach a common shared memory segment. Typically, the function ftok() is called and given a filename as an argument (filenames are more meaningful than arbitrary hex values) and a key is generated.

What is happening to you is that someone/something is removing the shared memory segments (perhaps via ipcrm) andc then processes that depend upon them are now in trouble.

I have seen some dangerous advice given that tells you if you run ipcs and see that nattach (the number of processes that are attached to this shmid) is zero then it is safe to remove this shmid. While nattach = 0 is a necessary condition, it may not be a sufficient condition. Often, a process sets up shared memory and exit - nattach = 0. It is only when subsequent processes need that memory - perhaps days later - that there is an actual problem.
You have to know how the the shared memory was intended to be used.

You need to dig deeper but the place to start is ipcs -ma.

Re: Shared Memory error?

Filosofo — Thu, 22 May 2003 13:59:06 GMT

Hello this is the output of your command:

shmmax 4294967296
shmseg 32

ipcs -mob
IPC status from /dev/kmem as of Thu May 22 16:55:41 2003
T ID KEY MODE OWNER GROUP NATTCH SEGSZ
Shared Memory:
m 0 0x411c025e --rw-rw-rw- root root 0 348
m 1 0x4e0c0002 --rw-rw-rw- root root 1 61760
m 2 0x41200097 --rw-rw-rw- root root 1 8192
m 3 0x301c63c0 --rw-rw-rw- root root 3 1048576
m 2564 0x5e10000b --rw------- root root 1 512
m 9221 0x00000000 D-rw------- root root 6 1052672
m 518 0x00000000 D-rw------- www other 6 184324
m 105479 0xe07e7728 --rw-rw---- oracle dba 91 286228480
m 520 0x16c5a37c --rw-rw-rw- gbcldv25 sv 6 500580
m 521 0x0000ef6a --rw-rw-rw- gbcldv25 sv 96 1773272
m 522 0x00000000 --rw------- gbcldv25 sv 3 652
m 11 0x00009669 --rw------- gbcldv25 sv 5 5617029
m 12 0x00008193 --rw------- gbcldv25 sv 5 5617029
m 525 0x0001d299 --rw------- gbcldv25 sv 5 5668149
m 526 0x16c9ea2d --rw-rw-rw- gbcldv25 sv 6 477452
m 527 0x16c95e7a --rw-rw-rw- gbcldv25 sv 18 1779752
m 528 0x16c56899 --rw-rw-rw- gbcldv25 sv 17 2076280
m 1553 0xc8277587 --rw-rw-rw- gbcldv25 sv 16 364324
m 13842 0x0fd93c00 --rw-rw---- oracle dba 61 93609984
m 531 0x21dc6698 --rw-rw---- oracle dba 72 66617344
m 11796 0x00019ec3 --rw-rw-rw- gbcldv19 sv 77 1773272
m 8725 0x00000000 --rw------- gbcldv19 sv 2 652
m 8726 0x000393c2 --rw------- gbcldv19 sv 5 5668149
m 8727 0x00025792 --rw------- gbcldv19 sv 5 5617029
m 8728 0x00018704 --rw------- gbcldv19 sv 5 5617029
m 2073 0x00037bf5 --rw------- gbcldv21 sv 5 5668149
m 2074 0x00023fc5 --rw------- gbcldv21 sv 5 5617029
m 2075 0x000209cf --rw------- gbcldv21 sv 5 5617029
m 2076 0x00028006 --rw-rw-rw- gbcldv21 sv 84 819616
m 2077 0x00000000 --rw------- gbcldv21 sv 1 804
m 542 0x1644deb0 --rw-rw-rw- gbcldv21 sv 5 500916
m 543 0x1640b3c1 --rw-rw-rw- gbcldv21 sv 5 477548
m 544 0x1640280e --rw-rw-rw- gbcldv21 sv 17 5925216
m 545 0x163c322d --rw-rw-rw- gbcldv21 sv 16 2137864
m 546 0xa84ed080 --rw-rw-rw- gbcldv21 sv 16 367192

echo total_lockable_mem/D | adb /stand/vmunix /dev/mem
total_lockable_mem:
total_lockable_mem: 852081

swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 860 3236 21% 0 - 1 /dev/vg00/lvol2
reserve - 2749 -2749
memory 3260 793 2467 24%
total 7356 4402 2954 60% - 0 -

Thanks Filo

Re: Shared Memory error?

Steven E. Protter — Thu, 22 May 2003 14:25:58 GMT

4 Gig of shared memory is enough

I however have 4000 shared memory segments.

shmseg should be increased.

Its dynamic, you should be able to do it on the fly.

If you believe you have hung shared memory segments here is a command to clear them.

ipcrm -m
ipcrm -s

Also might want to look into increasing nproc,nfiles and posting a complete kmtune commond for better analysis.

If we help, please assign a point or two.

SEP

Re: Shared Memory error?

Filosofo — Thu, 22 May 2003 14:36:46 GMT

What happen if I increas shmseg from 32 to 100 for example.

Thanks

Filo

Re: Shared Memory error?

A. Clay Stephenson — Thu, 22 May 2003 14:46:34 GMT

You can increase shmseg to 512 or so but I rather doubt that that is your problem. This is the per process limit of separate shmid's that can attach. It is very, very unusual for a process to attach more than about 8 - though there are exceptions. I would have a very serious talk with a programmer who ever thought he needed more than 32. Because it's dynamic, you can change shmseg "on the fly". If there is also a small number (< 200) listed anywhere in your log then that could be very meaningful because that is almost certainly errno. Errno gets sets when a system call (like shmget()) fails.

Re: Shared Memory error?

Steven E. Protter — Thu, 22 May 2003 18:13:34 GMT

No points deserved for anything I wrote.

HP has just gone over my system performance because Oracle has been performing like dog doo doo.

Seems while Oracle ias was out of whack some guy who looked a lot like me played around with the kernel, increasing shared memory a bit too much, and shmseg. When the problem was solved by other means, this same guy didn't back out the changes.

Note:

shmmax on my system is 4 Gig. It should be dropped to 1.5 Gig. I will be doing so.

shmseg is set for 4000, I'm only using 13.

It's dropping to 100 or so.

My buffer cache is too big and by dbc_max_pct is too high(I think).

Here is the complete post from HP support, because it highlights a good analysis which hasn't really been done in this thread(though A. Clay did a lot better than me).

-----
Memory

There is a sufficient quantity of memory on the system , however there are
some configuration areas that need to be optimized .

The size and utilization of the dynamic buffer cache indicates a need for a
change .

The average percent write to cache was 25-39% . Sizing dbc_max_pct to 40%
causes 2 issues : 1) excessive volatility of the cache.
2) intrusion into available RAM forces paging to disk , increasing disk I/O
. The point of using the buffer cache is to facilitate faster read/write
transactions to disk .

For the first run of data the total average throughput per second was only
2.06Mb for all disk combined , for the set run this a.m. it is 1.56 Mb /sec
. To allocate up to 819 Mb of RAM for buffering this amount of data is a
waste.

In Oracle and Unix Performance Tuning , Ahmed Alomari recommends and average
write to cache of 95% or greater . He is a Senior Oracle engineer , and his
books are quite good although not HP-UX specific.

It would be prudent to reduce the size of the buffer cache to no more than
10 % maximum. As your lockable memory is 19.2% of RAM , to avoid memory
contention setting dbc_max_pct to no greater than 6 would be best . This
will decrease the amount of cpu time cleaning dirty buf pages and still
provide more than enough memory to buffer the read/writes. It will also
prevent paging to disk , this is currently happening due to the lack of
clean memory pages .

The other configuration issue has to do with shared memory . You have
shmmax set to shmmax 4200000000 or 3.91 GB. this parameter sets the largest
segment size for shared memory , which is limited by the size of the memory
quadrant. As you have 6GB of total system memory , your quadrants are 1.5GB
each . While there are no performance penalties for oversizing this
parameter , you won't be able to get that size segment with the current
configured memory . Setting shmmax to 1.5Gb=0x60000000 would be
appropriate . In order to be able to have a 3.91GB quadrant you would have
to increase system memory by 9.64GB. The total amount of shared memory in
use is currently only 379Mb.

The current available shared memory for 64 bit processes is 2.75Gb , for 32
bit 1.75GB .

The other shared memory parameter that is questionable is shmseg set at 4000
, the default is 120 . Currently you are only using 13 .

-----

There is more, but its not relavent to this thread.

You have twice the system memory I do, so make appropriate adjustments while doing your analysis.

If you want the analysis tools I used, I will be happy to put them together for you, but currently need to be updated.

I do apologize for my earlier post. It was unworthy of someone with my ability.

SEP

Re: Shared Memory error?

Steven E. Protter — Thu, 22 May 2003 19:46:33 GMT

Here is a document on performance that might be of assistance.

* beats the dead horse again....

http://www1.itrc.hp.com/service/cki/search.do?category=c0&docType=Security&docType=Patch&docType=EngineerNotes&docType=BugReports&docType=Hardware&docType=ReferenceMaterials&docType=ThirdParty&searchString=UPERFKBAN00000726&mode=id&admit=-682735245+1053636168960+28353475&searchCrit=allwords&printable=true

SEP