Shared memory segment removal, swap close to 97% question

[Q4you]

swapinfo shows this ->
# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 8000 0 8000 0% 0 - 1 /dev/vg00/lvol2
dev 12000 3910 8090 33% 0 - 0 /dev/vg00/lv_swap1
reserve - 16090 -16090
memory 12737 11837 900 93%
total 32737 31837 900 97% - 0 -

Shared memory segment ->
m 2582 0x00000000 D-rw-rw-rw- abpadm sapsys abpadm sapsys 39
4290772992 17295 17295 11:33:45 no-entry 11:33:45
m 22840 0x00000000 D-rw-rw-rw- arcadm sapsys arcadm sapsys 26
6442450944 18017 18017 10:51:09 no-entry 10:51:09


The defunct processes ->
# ps -ef |grep defunc
abpadm 18849 17231 1 20:05:30 ? 0:00
abpadm 29291 17231 1 23:15:44 ? 0:00
abpadm 18312 17231 1 20:01:00 ? 0:00
root 16626 8415 1 12:55:23 pts/2 0:00 grep defunc
arcadm 16864 16862 4 05:12:10 ? 0:00


How do I ensure safe removal of defunct process or "defunct shared mem segment" to give relief to high swap utilization ?


-TIA

Q

[Steven E. Protter]

If you have a defunct process.

Try kill -9

That may work, but probably will not.

The only way at that point to deal with it is by restarting the system.

You can as root take a shot at the shared memory segment with ipcrm command

if it shows up in ipscs as -s use ipcrm -s if its a -m use ipcrm -m

SEP

[SS_6]

To check and remove shared memory use the following commands. make sure your application is down during that time.
#ipcs |grep for processes
#ipcrm -s pid
do man on ipcs and ipcrm.
-s is for active semaphores
-m about active shared memory segments.

[Fred Ruffet]

Apparently, they all are child of process 17231. Terminating this one may get the other to disappear.

otherwise, you could use
ipcrm -m
ro remove shm seg. (shmid is the second field in ipcs)

[A. Clay Stephenson]

You have zombie processes. They are already dead and can't be killed. You should probably kill the parent process.

To determine if it is safe to remove a shared memory identifier (shmid), do an ipca -ma and look in the NATTCH column. If this value is zero, it is a necessary but not necessarily a sufficient condition to use ipcrm. It all depends upon how the software that uses the shmid is wriiten. For example, it's rather common to have a process set up a shared memory region but then detach itself so that NATTCH = 0. Later processes which actually use the shared memory will then fail because the expected shared memory identifier is not found. Only your software developer will know for sure unless you have documentation or prior experince with this particular application.

By the way, the number one reason that dangling shmid's and sema4's get left is the use of kill -9.

[Q4you]

Thanks to all for quick replies.

knew about the ipcrm and kill etc. the question is more towards "safe removal" or checks be made before using the sword to chop off the rope ( when you don't know you may be hanging on the same rope :)

As Clay mentioned, the each shared mem segment in the question above, have 39 & 26 attachments resp. but its is marked as "D-rw-rw-rw" and not used/updated since 10:51 and 11:33 AM. The parent processes 17295 & 18017 are running sap instances each has many more child processes running at this time.

If recycle/boot is "THE" solution then its not a solution :)) I am still keeping points in my bag..

-TIA

Q

[DIPAK KUMAR ROY]

You probably can (99%) kill defunc processes by killing the ppid .. In your case

kill -9 17231

Then use ipcs -m to find out shared memory segments that are marked as "D" and user ipcrm to remove them.

Thanks

[Steve Lewis]

Having a large shared memory segment with a D flag (marked for deletion) is standard with SAP. It is the way SAP works, similar to Apache. I suspect that you have 2 SAP instances running on your machine, is that right?

The master SAP daemon creates the segment, forks the other processes which attach to the shared memory, then uses shmctl() to remove it - see man shmctl for how it works. Dont try to delete it - you cannot remove it anyway, until the processes that are attached to it exit on their own.

I suspect it is a way of automatically removing the shared memory when the master SAP daemon exits, but other SAP gurus may have more correct information.

Whats more important in your case is the SAP system has taken all of your memory and swap. I have seen this frequently before and I suspect that it is something to do with the SAP tuning recommendations. If you run vmstat -S 5 20 and see it paging out a lot (po>0), then you will have to 'take a view' on whether to reduce the amount of memory you SAP system is allowed, or to say that the system is running at a reasonable speed and you don't have a serious problem after all.

As has been said above, you wont be able to remove the zombie processes unless (a) you reboot or (b) they clean themselves up

[Q4you]

The server was rebooted for another change so it cleared up the problem for now. thanks for your inputs/replies.

Appreciate if someone could post good doc on SAP customization and/or Performance guide on HPUX.

-Q