SAP problem - some processes are restarting constantly

[Viktor Balogh]

Hi,

We have some problem with SAP, some processes are restarting constantly. The problem persist from 15. december. I was asked to look after it on the OS side. Until now, I have free'd up some unused shared memory segment, and found a disk device that has above-average utilization.

Could you give me some hint what can be looked after?

I have organized the LVM structure so, that both FC HBAs have roughly the same load. (The primary FC links were shared between the two HBAs)

And a final question: why is the system swapping when we have 63% physical memory free? Is this ok?

# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 2397 1699 59% 0 - 1 /dev/vg00/lvol2
dev 32768 2406 30362 7% 0 - 1 /dev/vg01/lvswap
reserve - 27642 -27642
memory 53056 33371 19685 63%
total 89920 65816 24104 73% - 0 -

# uname -a
HP-UX hptdp03 B.11.11 U 9000/800 3782497108 unlimited-user license

[Viktor Balogh]

these are the processes in question:


  tdpadm   193 15946 20 09:44:14 ?         0:31 dw.sapTDP_DVEBMGS00 pf=/usr/sap/TDP/SYS/profile/TDP_DVEBMGS00_h
  tdpadm 26139 15946  0 09:55:34 ?         3:27 dw.sapTDP_DVEBMGS00 pf=/usr/sap/TDP/SYS/profile/TDP_DVEBMGS00_h
  tdpadm 14342 15946 244 10:22:54 ?        14:55 dw.sapTDP_DVEBMGS00 pf=/usr/sap/TDP/SYS/profile/TDP_DVEBMGS00_h
  tdpadm 10392 15946 27 10:42:54 ?         0:00 dw.sapTDP_DVEBMGS00 pf=/usr/sap/TDP/SYS/profile/TDP_DVEBMGS00_h
  tdpadm  3703 15946 29 10:35:34 ?         0:17 dw.sapTDP_DVEBMGS00 pf=/usr/sap/TDP/SYS/profile/TDP_DVEBMGS00_h
  tdpadm  7424 15946  0 09:03:34 ?        31:46 dw.sapTDP_DVEBMGS00 pf=/usr/sap/TDP/SYS/profile/TDP_DVEBMGS00_h
  tdpadm  8252 15946 53 10:41:54 ?         0:03 dw.sapTDP_DVEBMGS00 pf=/usr/sap/TDP/SYS/profile/TDP_DVEBMGS00_h
  tdpadm 26973 15946  0 14:43:34 ?         6:02 dw.sapTDP_DVEBMGS00 pf=/usr/sap/TDP/SYS/profile/TDP_DVEBMGS00_h
  tdpadm 14411 15946 255 12:00:54 ?        71:09 dw.sapTDP_DVEBMGS00 pf=/usr/sap/TDP/SYS/profile/TDP_DVEBMGS00_h

let me know if you need some more details

[Michael Steele_2]

HI

A) Need to see contents of :

/usr/sap/TDP/SYS/profile, and,
usr/sap/TDP/SYS/profile/TDP_DVEBMGS00_h

...with 'set -xv' on the 2nd line of each file. The output will be long so use attachements.

'set -xv' turn debugging on.

B) What is the /var/adm/syslog/syslog.log error?

C) Can you recreate the error? If so then include how.

[Viktor Balogh]

A)

# ll /usr/sap/TDP/SYS/profile
lrwxr-xr-x 1 root sys 19 Apr 25 2005 /usr/sap/TDP/SYS/profile -> /sapmnt/TDP/profile

# ll /sapmnt/TDP/profile
total 320
-rw-rw---- 1 tdpadm sapsys 4518 Oct 30 2008 DEFAULT.BAK
-rw-r--r-- 1 tdpadm sapsys 4578 Oct 30 2008 DEFAULT.PFL
-rw-r--r-- 1 tdpadm sapsys 1947 Sep 18 2008 START_D00_ptdpdi01
-rw-r--r-- 1 tdpadm sapsys 1947 Sep 18 2008 START_D00_ptdpdi02
-rw-r--r-- 1 tdpadm sapsys 1946 Sep 18 2008 START_D00_ptdpdi03
-rw-r--r-- 1 tdpadm sapsys 2669 Sep 18 2008 START_DVEBMGS00_hpptdp1r
-rw-r----- 1 tdpadm sapsys 6210 Aug 6 11:31 TDP_D00_ptdpdi01
-rw-rw---- 1 tdpadm sapsys 6102 Aug 6 11:31 TDP_D00_ptdpdi01.BAK
-rw-r----- 1 tdpadm sapsys 4876 Apr 9 2008 TDP_D00_ptdpdi01.org
-rw-r----- 1 tdpadm sapsys 6210 Aug 6 11:32 TDP_D00_ptdpdi02
-rw-rw---- 1 tdpadm sapsys 6102 Aug 6 11:32 TDP_D00_ptdpdi02.BAK
-rw-r----- 1 tdpadm sapsys 6210 Aug 6 11:33 TDP_D00_ptdpdi03
-rw-rw---- 1 tdpadm sapsys 6102 Aug 6 11:33 TDP_D00_ptdpdi03.BAK
-rw-r--r-- 1 tdpadm sapsys 6670 Feb 7 2009 TDP_DVEBMGS00_hpptdp1r
-rw-rw---- 1 tdpadm sapsys 6605 Feb 7 2009 TDP_DVEBMGS00_hpptdp1r.BAK
-rw-r--r-- 1 tdpadm sapsys 259 Oct 26 2008 dev_dpmon
-rw-r--r-- 1 tdpadm sapsys 638 Oct 26 2008 dev_icmmon
drwxr-xr-x 2 tdpadm sapsys 8192 Apr 19 2008 old
-rw-r--r-- 1 tdpadm sapsys 2775 Jul 30 2008 sapccm4x.install.log
-rw-r--r-- 1 tdpadm sapsys 2302 Jul 30 2008 sapccm4x.install.log.old



# ll /usr/sap/TDP/SYS/profile/TDP_DVEBMGS00_hpptdp1r
-rw-r--r-- 1 tdpadm sapsys 6670 Feb 7 2009 /usr/sap/TDP/SYS/profile/TDP_DVEBMGS00_hpptdp1r


B)

there isn't any syslog messages besides the 'syslog is running' ones.

C)> Can you recreate the error? If so then include how.

The problem exists even now. SAP processes are restarting constantly.

Start Workproc 1, 61.Mal seit Systemstart, Pid 3 1422
Start Workproc 1, 61.Mal seit Systemstart, Pid 4 1434
Start Workproc 3, 20.Mal seit Systemstart, Pid 8 1683
Start Workproc 3, 20.Mal seit Systemstart, Pid 9 2086

[Viktor Balogh]

In the TDP_DVEBMGS00_hpptdp1r there are some SAP settings, but I am not allowed to poke with this. After all, my task is only to check the OS side, and the SAP colleagues said that there weren't any modifications to SAP by that time the problem first occured.

The syslog of 15. dec shows also nothing, just 'syslog is running' messages. What about swapping? Is this normal to use swap area when there is plenty free mem there? Btw. there were any known changes to the system around that time.

[Michael Steele_2]

Hi

When you add 'set -xv' to the 2nd line of those two files, you cause everyline's execution to be displayed to STDOUT.

A) Do you know how to vi these files?
B) Once completed, login with 'tpdadm' or restart the processes, probably by restarting the application.
C) Do you log in with 'tpdadm' normally?
D) Is 'tpdadm' only owned by the application?

[Michael Steele_2 · Accepted solution]

Hi, and well,

"... but I am not allowed to poke with this ..."

I'm 99% certain this is not going to be an O/S problem but a SAP problem. A command owned by SAP is trying to execute and having problems.

From your end you can verify the ownership and permissions of SAP Applications files by comparing them on another box, or by running 'swverify' or 'check_patches'. But this should also be a SAP Admin responsibility. I've worked with SAPPER's for a decade and they always determined ownerships and permissions of their own stuff. And 'swverify' and 'check_patches' will only verify O/S files. An equivalent should exist in the SAP Admin's world.

[Viktor Balogh]

A) > Do you know how to vi these files?
good joke, of course i do! :)

B) > Once completed, login with 'tpdadm' or restart the processes, probably by restarting the application.
I am neither allowed to su to tdpadm nor to restart SAP. Strange thing I know, but these tasks are duties of the SAP team. (However, it would help a lot some time to debug problems...)


C) Do you log in with 'tpdadm' normally?
Yeah, they log in with this user

D) Is 'tpdadm' only owned by the application?
yes, tdpadm is the app-user


As you suggested in your last post, I will check the permissions perhaps there is something wrong. It's a good idea, thanks.

[Michael Steele_2]

Hi

As I said, this is 99% percent certain a SAP problem and if you google for it you'll find this link for SAP CCMS Agents:

http://help.sap.com/saphelp_nw04/helpdata/EN/48/6a76607a7c91409474ad6c7b6f5a26/frameset.htm

Where all the processes indicated in your thread are CCMS agents:

/usr/sap//SYS/profile/__

If the Sappers don't have any log files then refer to this link and have them enable the logfiles:

http://help.sap.com/saphelp_nw04/helpdata/en/65/f3156d443e744abe15dbe14e4e32b5/content.htm

Note: I've been working with Sappers for a decade and they have all had the biggest bullying attitude about Unix admins and their Application's Goldenness, yet 9/10, that goldenness is just lead in anybody else's eyes.

[Viktor Balogh]

Michael, thanks for your help, there is a planned restart for SAP on sunday, after that I'll report how it went.