topic Re: SAP problem - some processes are restarting constantly in Operating System - HP-UX

SAP problem - some processes are restarting constantly

Viktor Balogh — Fri, 18 Dec 2009 11:49:58 GMT

Hi,

We have some problem with SAP, some processes are restarting constantly. The problem persist from 15. december. I was asked to look after it on the OS side. Until now, I have free'd up some unused shared memory segment, and found a disk device that has above-average utilization.

Could you give me some hint what can be looked after?

I have organized the LVM structure so, that both FC HBAs have roughly the same load. (The primary FC links were shared between the two HBAs)

And a final question: why is the system swapping when we have 63% physical memory free? Is this ok?

# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 2397 1699 59% 0 - 1 /dev/vg00/lvol2
dev 32768 2406 30362 7% 0 - 1 /dev/vg01/lvswap
reserve - 27642 -27642
memory 53056 33371 19685 63%
total 89920 65816 24104 73% - 0 -

# uname -a
HP-UX hptdp03 B.11.11 U 9000/800 3782497108 unlimited-user license

Re: SAP problem - some processes are restarting constantly

Viktor Balogh — Fri, 18 Dec 2009 11:52:35 GMT

these are the processes in question:

Â tdpadmÂ Â 193 15946 20 09:44:14 ?Â Â Â Â Â Â Â Â 0:31 dw.sapTDP_DVEBMGS00 pf=/usr/sap/TDP/SYS/profile/TDP_DVEBMGS00_h
Â tdpadm 26139 15946Â 0 09:55:34 ?Â Â Â Â Â Â Â Â 3:27 dw.sapTDP_DVEBMGS00 pf=/usr/sap/TDP/SYS/profile/TDP_DVEBMGS00_h
Â tdpadm 14342 15946 244 10:22:54 ?Â Â Â Â Â Â Â 14:55 dw.sapTDP_DVEBMGS00 pf=/usr/sap/TDP/SYS/profile/TDP_DVEBMGS00_h
Â tdpadm 10392 15946 27 10:42:54 ?Â Â Â Â Â Â Â Â 0:00 dw.sapTDP_DVEBMGS00 pf=/usr/sap/TDP/SYS/profile/TDP_DVEBMGS00_h
Â tdpadmÂ 3703 15946 29 10:35:34 ?Â Â Â Â Â Â Â Â 0:17 dw.sapTDP_DVEBMGS00 pf=/usr/sap/TDP/SYS/profile/TDP_DVEBMGS00_h
Â tdpadmÂ 7424 15946Â 0 09:03:34 ?Â Â Â Â Â Â Â 31:46 dw.sapTDP_DVEBMGS00 pf=/usr/sap/TDP/SYS/profile/TDP_DVEBMGS00_h
Â tdpadmÂ 8252 15946 53 10:41:54 ?Â Â Â Â Â Â Â Â 0:03 dw.sapTDP_DVEBMGS00 pf=/usr/sap/TDP/SYS/profile/TDP_DVEBMGS00_h
Â tdpadm 26973 15946Â 0 14:43:34 ?Â Â Â Â Â Â Â Â 6:02 dw.sapTDP_DVEBMGS00 pf=/usr/sap/TDP/SYS/profile/TDP_DVEBMGS00_h
Â tdpadm 14411 15946 255 12:00:54 ?Â Â Â Â Â Â Â 71:09 dw.sapTDP_DVEBMGS00 pf=/usr/sap/TDP/SYS/profile/TDP_DVEBMGS00_h

let me know if you need some more details

Re: SAP problem - some processes are restarting constantly

Michael Steele_2 — Fri, 18 Dec 2009 13:32:18 GMT

HI

A) Need to see contents of :

/usr/sap/TDP/SYS/profile, and,
usr/sap/TDP/SYS/profile/TDP_DVEBMGS00_h

...with 'set -xv' on the 2nd line of each file. The output will be long so use attachements.

'set -xv' turn debugging on.

B) What is the /var/adm/syslog/syslog.log error?

C) Can you recreate the error? If so then include how.

Re: SAP problem - some processes are restarting constantly

Viktor Balogh — Fri, 18 Dec 2009 14:12:40 GMT

A)

# ll /usr/sap/TDP/SYS/profile
lrwxr-xr-x 1 root sys 19 Apr 25 2005 /usr/sap/TDP/SYS/profile -> /sapmnt/TDP/profile

# ll /sapmnt/TDP/profile
total 320
-rw-rw---- 1 tdpadm sapsys 4518 Oct 30 2008 DEFAULT.BAK
-rw-r--r-- 1 tdpadm sapsys 4578 Oct 30 2008 DEFAULT.PFL
-rw-r--r-- 1 tdpadm sapsys 1947 Sep 18 2008 START_D00_ptdpdi01
-rw-r--r-- 1 tdpadm sapsys 1947 Sep 18 2008 START_D00_ptdpdi02
-rw-r--r-- 1 tdpadm sapsys 1946 Sep 18 2008 START_D00_ptdpdi03
-rw-r--r-- 1 tdpadm sapsys 2669 Sep 18 2008 START_DVEBMGS00_hpptdp1r
-rw-r----- 1 tdpadm sapsys 6210 Aug 6 11:31 TDP_D00_ptdpdi01
-rw-rw---- 1 tdpadm sapsys 6102 Aug 6 11:31 TDP_D00_ptdpdi01.BAK
-rw-r----- 1 tdpadm sapsys 4876 Apr 9 2008 TDP_D00_ptdpdi01.org
-rw-r----- 1 tdpadm sapsys 6210 Aug 6 11:32 TDP_D00_ptdpdi02
-rw-rw---- 1 tdpadm sapsys 6102 Aug 6 11:32 TDP_D00_ptdpdi02.BAK
-rw-r----- 1 tdpadm sapsys 6210 Aug 6 11:33 TDP_D00_ptdpdi03
-rw-rw---- 1 tdpadm sapsys 6102 Aug 6 11:33 TDP_D00_ptdpdi03.BAK
-rw-r--r-- 1 tdpadm sapsys 6670 Feb 7 2009 TDP_DVEBMGS00_hpptdp1r
-rw-rw---- 1 tdpadm sapsys 6605 Feb 7 2009 TDP_DVEBMGS00_hpptdp1r.BAK
-rw-r--r-- 1 tdpadm sapsys 259 Oct 26 2008 dev_dpmon
-rw-r--r-- 1 tdpadm sapsys 638 Oct 26 2008 dev_icmmon
drwxr-xr-x 2 tdpadm sapsys 8192 Apr 19 2008 old
-rw-r--r-- 1 tdpadm sapsys 2775 Jul 30 2008 sapccm4x.install.log
-rw-r--r-- 1 tdpadm sapsys 2302 Jul 30 2008 sapccm4x.install.log.old

# ll /usr/sap/TDP/SYS/profile/TDP_DVEBMGS00_hpptdp1r
-rw-r--r-- 1 tdpadm sapsys 6670 Feb 7 2009 /usr/sap/TDP/SYS/profile/TDP_DVEBMGS00_hpptdp1r

B)

there isn't any syslog messages besides the 'syslog is running' ones.

C)> Can you recreate the error? If so then include how.

The problem exists even now. SAP processes are restarting constantly.

Start Workproc 1, 61.Mal seit Systemstart, Pid 3 1422
Start Workproc 1, 61.Mal seit Systemstart, Pid 4 1434
Start Workproc 3, 20.Mal seit Systemstart, Pid 8 1683
Start Workproc 3, 20.Mal seit Systemstart, Pid 9 2086

Re: SAP problem - some processes are restarting constantly

Viktor Balogh — Fri, 18 Dec 2009 14:17:23 GMT

In the TDP_DVEBMGS00_hpptdp1r there are some SAP settings, but I am not allowed to poke with this. After all, my task is only to check the OS side, and the SAP colleagues said that there weren't any modifications to SAP by that time the problem first occured.

The syslog of 15. dec shows also nothing, just 'syslog is running' messages. What about swapping? Is this normal to use swap area when there is plenty free mem there? Btw. there were any known changes to the system around that time.

Re: SAP problem - some processes are restarting constantly

Michael Steele_2 — Fri, 18 Dec 2009 15:33:39 GMT

Hi

When you add 'set -xv' to the 2nd line of those two files, you cause everyline's execution to be displayed to STDOUT.

A) Do you know how to vi these files?
B) Once completed, login with 'tpdadm' or restart the processes, probably by restarting the application.
C) Do you log in with 'tpdadm' normally?
D) Is 'tpdadm' only owned by the application?

Re: SAP problem - some processes are restarting constantly

Michael Steele_2 — Fri, 18 Dec 2009 15:39:45 GMT

Hi, and well,

"... but I am not allowed to poke with this ..."

I'm 99% certain this is not going to be an O/S problem but a SAP problem. A command owned by SAP is trying to execute and having problems.

From your end you can verify the ownership and permissions of SAP Applications files by comparing them on another box, or by running 'swverify' or 'check_patches'. But this should also be a SAP Admin responsibility. I've worked with SAPPER's for a decade and they always determined ownerships and permissions of their own stuff. And 'swverify' and 'check_patches' will only verify O/S files. An equivalent should exist in the SAP Admin's world.

Re: SAP problem - some processes are restarting constantly

Viktor Balogh — Fri, 18 Dec 2009 21:07:20 GMT

A) > Do you know how to vi these files?
good joke, of course i do! :)

B) > Once completed, login with 'tpdadm' or restart the processes, probably by restarting the application.
I am neither allowed to su to tdpadm nor to restart SAP. Strange thing I know, but these tasks are duties of the SAP team. (However, it would help a lot some time to debug problems...)

C) Do you log in with 'tpdadm' normally?
Yeah, they log in with this user

D) Is 'tpdadm' only owned by the application?
yes, tdpadm is the app-user

As you suggested in your last post, I will check the permissions perhaps there is something wrong. It's a good idea, thanks.

Re: SAP problem - some processes are restarting constantly

Michael Steele_2 — Fri, 18 Dec 2009 21:50:52 GMT

Hi

As I said, this is 99% percent certain a SAP problem and if you google for it you'll find this link for SAP CCMS Agents:

http://help.sap.com/saphelp_nw04/helpdata/EN/48/6a76607a7c91409474ad6c7b6f5a26/frameset.htm

Where all the processes indicated in your thread are CCMS agents:

/usr/sap//SYS/profile/__

If the Sappers don't have any log files then refer to this link and have them enable the logfiles:

http://help.sap.com/saphelp_nw04/helpdata/en/65/f3156d443e744abe15dbe14e4e32b5/content.htm

Note: I've been working with Sappers for a decade and they have all had the biggest bullying attitude about Unix admins and their Application's Goldenness, yet 9/10, that goldenness is just lead in anybody else's eyes.

Re: SAP problem - some processes are restarting constantly

Viktor Balogh — Sat, 19 Dec 2009 11:28:08 GMT

Michael, thanks for your help, there is a planned restart for SAP on sunday, after that I'll report how it went.

Re: SAP problem - some processes are restarting constantly

Viktor Balogh — Fri, 05 Mar 2010 13:25:35 GMT

Thread closed. The SAP team had found the problem on their side.