execve failure

[VCS Engineering]

When I executed the application with tusc, the application is failing with the following error:

exit(9) [implicit (execve failure)]......................WIFEXITED(9).

What does this indicate?

[Dennis Handly]

All you did was just use tusc and normally it works? What exact tusc command did you use?

[VCS Engineering]

Hi Dannis,

The application internally does fork and calls execve to run another program. But it is failing.
I am trying to find out why the execve is failing. I observed the above message in tusc output. So, I am trying to understand the meaning of the same.

Thanks for your help.

[Dennis Handly]

It would help if we saw more of the tusc system calls. Use -fp and -ea options.

[VCS Engineering]

Hi Dennis,

Here is the tusc output for that process

[16952] fork() .......................................................................................................... (returning as child ...) ......... = 16951
[16952] getpid() ........................................................................................................................................... = 16952 (16951)
[16952] _lwp_self() ........................................................................................................................................ = 6994636
[16952] set_userthreadid(1) ................................................................................................................................ = 1
[16952] sysconf(_SC_CCNUMA_PM) ............................................................................................................................. = 1
[16952] mpctl(MPC_GETLWP_BINDVALUE, 0, -4) ................................................................................................................. = 104
[16952] mpctl(MPC_SETLWP, -7, -4) .......................................................................................................................... = -6
[16952] _lwp_self() ........................................................................................................................................ = 6994636
[16952] _lwp_getscheduler(LWP_SCHED_SPECIFIC, 6994636, 0x9fffffffbf50d288, 0x9fffffffbf50d28c) ............................................................. = 0
[16952] chdir("/") ......................................................................................................................................... = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigprocmask(SIG_SETMASK, 0x9ffffffffffed010, 0x9ffffffffffed030) ................................................................................... = 0
[16952] sigprocmask(SIG_SETMASK, 0x9ffffffffffed030, 0x9ffffffffffed010) ................................................................................... = 0
[16952] sigsetreturn(NULL, 0x6212003, 48640) ............................................................................................................... = 0
[16952] sigprocmask(SIG_SETMASK, 0x9ffffffffffed010, 0x9ffffffffffed030) ................................................................................... = 0
[16952] sigprocmask(SIG_SETMASK, 0x9ffffffffffed030, 0x9ffffffffffed010) ................................................................................... = 0
[16952] sigvec(SIGHUP, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................. = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGINT, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................. = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGQUIT, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................ = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGILL, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................. = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGTRAP, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................ = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGABRT, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................ = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGEMT, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................. = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGFPE, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................. = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGBUS, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................. = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGSEGV, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................ = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGSYS, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................. = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGPIPE, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................ = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGALRM, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................ = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGTERM, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................ = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGUSR1, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................ = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGUSR2, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................ = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGCLD, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................. = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGTSTP, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................ = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGCONT, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................ = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGTTIN, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................ = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGTTOU, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................ = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGURG, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................. = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGLOST, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................ = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGRESERVE, 0x9ffffffffffed050, 0x9ffffffffffed060) ......................................................................................... ERR#22 EINVAL
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGDIL, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................. ERR#22 EINVAL
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGXCPU, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................ = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGXFSZ, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................ = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGCANCEL, 0x9ffffffffffed050, 0x9ffffffffffed060) .......................................................................................... = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGGFAULT, 0x9ffffffffffed050, 0x9ffffffffffed060) .......................................................................................... ERR#22 EINVAL
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGRT1, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................. = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGRT2, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................. = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGRT3, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................. = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGRT4, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................. = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGRT5, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................. = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGRT6, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................. = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGRT7, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................. = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] sigvec(SIGRT8, 0x9ffffffffffed050, 0x9ffffffffffed060) ............................................................................................. = 0
[16952] sysconf(_SC_SIGRTMAX) .............................................................................................................................. = 44
[16952] write(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0".., 216) ................................................................................................ = 216
[16952] open("/dev/null", O_RDWR|0x800, 0330) .............................................................................................................. = 3
[16952] dup2(3, 0) ......................................................................................................................................... = 0
[16952] dup2(3, 1) ......................................................................................................................................... = 1
[16952] dup2(3, 2) ......................................................................................................................................... = 2
[16952] getpid() ........................................................................................................................................... = 16952 (1)
[16952] pstat(PSTAT_PROC, 0x9ffffffffffecbb0, 1216, 0, 16952) .............................................................................................. = 1
[16952] close(3) ........................................................................................................................................... = 0
[16952] close(4) ........................................................................................................................................... = 0
[16952] close(5) ........................................................................................................................................... = 0
[16952] execve(0x9fffffffffffe130, 0x600000000001d130, 0x9ffffffffffff928) ................................................................................. [entry]
[16952] execve(0x9fffffffffffe130, 0x600000000001d130, 0x9ffffffffffff928) ................................................................................. [entry]
[16952] exit(9) [implicit (execve failure)] ................................................................................................................ WIFEXITED(9)

[VCS Engineering]

Hi Dennis,

I saw the following messages on terminal

tusc: ttrace(TT_PROC_RDDATA, 23045, 0, 9ffffffffffff9d0, 4, 9ffffffffffff170): Bad address
tusc: ttrace(TT_PROC_RDDATA, 23045, 0, 9ffffffffffff9e0, 4, 9ffffffffffff170): Bad address

[Dennis Handly]

>[16952] open("/dev/null", O_RDWR|0x800, 0330) = 3
[16952] dup2(3, 0) = 0
[16952] dup2(3, 1) = 1
[16952] dup2(3, 2) = 2

Are you creating a demon here by using /dev/null?

>[16952] execve(0x9fffffffffffe130, 0x600000000001d130, 0x9ffffffffffff928) [entry]

You should immediately see the argv and env values here.

>[16952] exit(9) [implicit (execve failure)] . WIFEXITED(9)

I'm not sure what this means. exec(2) doesn't return errno 9, EBADF.

You'll need to put some printfs in your code to see how you handle a bad status from exec(2). Also print the name, arguments and errno.

>I saw the following messages on terminal
tusc: ttrace(TT_PROC_RDDATA, 23045, 0, 9ffffffffffff9d0, 4, 9ffffffffffff170): Bad address

These are tusc errors, you can ignore them.

[VCS Engineering]

It is third party application. So I cannot modify the source code for debugging.
Please see my answers inline starting with Mangesh>

>[16952] open("/dev/null", O_RDWR|0x800, 0330) = 3
[16952] dup2(3, 0) = 0
[16952] dup2(3, 1) = 1
[16952] dup2(3, 2) = 2

Are you creating a demon here by using /dev/null?

Mangesh> That's correct.

>[16952] execve(0x9fffffffffffe130, 0x600000000001d130, 0x9ffffffffffff928) [entry]

You should immediately see the argv and env values here.

Mangesh> I see both the argv and env values.


>[16952] exit(9) [implicit (execve failure)] . WIFEXITED(9)

I'm not sure what this means. exec(2) doesn't return errno 9, EBADF.

You'll need to put some printfs in your code to see how you handle a bad status from exec(2). Also print the name, arguments and errno.

Mangesh> Does it mean that there is some issue with file descriptors shared between the parent process and execed process.

>I saw the following messages on terminal
tusc: ttrace(TT_PROC_RDDATA, 23045, 0, 9ffffffffffff9d0, 4, 9ffffffffffff170): Bad address

These are tusc errors, you can ignore them.

[Dennis Handly]

>I cannot modify the source code for debugging.

Then you have to do assembly debugging.

>I see both the argv and env values.

It helps if we see them too, the exact output of "tusc -fp -ea".

>Does it mean that there is some issue with FDs shared between the parent process and execed process.

Possibly. But it seems the tusc output is bogus. I.e. why say "implicit (execve failure)". Why not show execve(2) returning a bad errno value that we can trust?

[VCS Engineering]

Hi Dennis,

The tusc output starting from fork is called.

[5509] fork() .............................................................(returning as child ...) ......... = 5508
[5508] fork() ............................................................................................... = 5509
[5509] getpid() ............................................................................................. = 5509 (5508)
[5509] lwp_self() ........................................................................................... = 4821488
[5508] exit(0) .............................................................................................. WIFEXITED(0)
[5509] set_userthreadid(1) .................................................................................. = 1
[5509] sysconf(_SC_CCNUMA_PM) ............................................................................... = 1
[5509] mpctl(MPC_GETLWP_BINDVALUE, 0, -4) ................................................................... = 104
[5509] mpctl(MPC_SETLWP, -7, -4) ............................................................................ = -6
[5509] lwp_self() ........................................................................................... = 4821488
[5509] lwp_getscheduler(LWP_SCHED_SPECIFIC, 4821488, 0x9fffffffbf50d288, 0x9fffffffbf50d28c) ................ = 0
[5509] chdir("/") ........................................................................................... = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigprocmask(SIG_SETMASK, 0x9ffffffffffec8e0, 0x9ffffffffffec900) ..................................... = 0
[5509] sigprocmask(SIG_SETMASK, 0x9ffffffffffec900, 0x9ffffffffffec8e0) ..................................... = 0
[5509] sigsetreturn(NULL, 0x6212003, 48640) ................................................................. = 0
[5509] sigprocmask(SIG_SETMASK, 0x9ffffffffffec8e0, 0x9ffffffffffec900) ..................................... = 0
[5509] sigprocmask(SIG_SETMASK, 0x9ffffffffffec900, 0x9ffffffffffec8e0) ..................................... = 0
[5509] sigvector(SIGHUP, 0x9ffffffffffec920, 0x9ffffffffffec930) ............................................ = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGINT, 0x9ffffffffffec920, 0x9ffffffffffec930) ............................................ = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGQUIT, 0x9ffffffffffec920, 0x9ffffffffffec930) ........................................... = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGILL, 0x9ffffffffffec920, 0x9ffffffffffec930) ............................................ = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGTRAP, 0x9ffffffffffec920, 0x9ffffffffffec930) ........................................... = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGABRT, 0x9ffffffffffec920, 0x9ffffffffffec930) ........................................... = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGEMT, 0x9ffffffffffec920, 0x9ffffffffffec930) ............................................ = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGFPE, 0x9ffffffffffec920, 0x9ffffffffffec930) ............................................ = 0
[5434] waitpid(5508, WIFEXITED(0), 0) ....................................................................... = 5508
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5434] close(5) ............................................................................................. = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGBUS, 0x9ffffffffffec920, 0x9ffffffffffec930) ............................................ = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGSEGV, 0x9ffffffffffec920, 0x9ffffffffffec930) ........................................... = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGSYS, 0x9ffffffffffec920, 0x9ffffffffffec930) ............................................ = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGPIPE, 0x9ffffffffffec920, 0x9ffffffffffec930) ........................................... = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGALRM, 0x9ffffffffffec920, 0x9ffffffffffec930) ........................................... = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGTERM, 0x9ffffffffffec920, 0x9ffffffffffec930) ........................................... = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGUSR1, 0x9ffffffffffec920, 0x9ffffffffffec930) ........................................... = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
5509] sigvector(SIGUSR2, 0x9ffffffffffec920, 0x9ffffffffffec930) ........................................... = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGCLD, 0x9ffffffffffec920, 0x9ffffffffffec930) ............................................ = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGPWR, 0x9ffffffffffec920, 0x9ffffffffffec930) ............................................ = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGVTALRM, 0x9ffffffffffec920, 0x9ffffffffffec930) ......................................... = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGPROF, 0x9ffffffffffec920, 0x9ffffffffffec930) ........................................... = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGIO, 0x9ffffffffffec920, 0x9ffffffffffec930) ............................................. = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGWINCH, 0x9ffffffffffec920, 0x9ffffffffffec930) .......................................... = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGTSTP, 0x9ffffffffffec920, 0x9ffffffffffec930) ........................................... = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGCONT, 0x9ffffffffffec920, 0x9ffffffffffec930) ........................................... = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGTTIN, 0x9ffffffffffec920, 0x9ffffffffffec930) ........................................... = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGTTOU, 0x9ffffffffffec920, 0x9ffffffffffec930) ........................................... = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGURG, 0x9ffffffffffec920, 0x9ffffffffffec930) ............................................ = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGLOST, 0x9ffffffffffec920, 0x9ffffffffffec930) ........................................... = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(_SIGRESERVE, 0x9ffffffffffec920, 0x9ffffffffffec930) ....................................... ERR#22 EINVAL
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(_SIGDIL, 0x9ffffffffffec920, 0x9ffffffffffec930) ........................................... ERR#22 EINVAL
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(_SIGXCPU, 0x9ffffffffffec920, 0x9ffffffffffec930) .......................................... = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(_SIGXFSZ, 0x9ffffffffffec920, 0x9ffffffffffec930) .......................................... = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(_SIGCANCEL, 0x9ffffffffffec920, 0x9ffffffffffec930) ........................................ = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(_SIGGFAULT, 0x9ffffffffffec920, 0x9ffffffffffec930) ........................................ ERR#22 EINVAL
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGRT1, 0x9ffffffffffec920, 0x9ffffffffffec930) ............................................ = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGRT2, 0x9ffffffffffec920, 0x9ffffffffffec930) ............................................ = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGRT3, 0x9ffffffffffec920, 0x9ffffffffffec930) ............................................ = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGRT4, 0x9ffffffffffec920, 0x9ffffffffffec930) ............................................ = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGRT5, 0x9ffffffffffec920, 0x9ffffffffffec930) ............................................ = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGRT6, 0x9ffffffffffec920, 0x9ffffffffffec930) ............................................ = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGRT7, 0x9ffffffffffec920, 0x9ffffffffffec930) ............................................ = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] sigvector(SIGRT8, 0x9ffffffffffec920, 0x9ffffffffffec930) ............................................ = 0
[5509] sysconf(_SC_SIGRTMAX) ................................................................................ = 44
[5509] write(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0".., 216) .................................................. = 216
[5434] read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0".., 216) ................................................... = 216
[5509] open("/dev/null", O_RDWR|0x800, 0330) ................................................................ = 3
[5434] close(3) ............................................................................................. = 0
[5509] dup2(3, 0) ........................................................................................... = 0
[5509] dup2(3, 1) ........................................................................................... = 1
[5509] dup2(3, 2) ........................................................................................... = 2
[5509] getpid() ............................................................................................. = 5509 (1)
[5509] pstat(PSTAT_PROC, 0x9ffffffffffec480, 1216, 0, 5509) ................................................. = 1
[5509] close(3) ............................................................................................. = 0
[5509] close(4) ............................................................................................. = 0
[5509] close(5) ............................................................................................. = 0
[5509] execve(0x9fffffffffffda00, 0x600000000001cfb0, 0x9ffffffffffff1f0) ................................... [entry]
argv[0] @ 0x600000000001d040: "/crsvol/bin/ohasd"
argv[1] @ 0x6000000000034390: "reboot"
env[0] @ 0x9ffffffffffff391: "_=/crsvol/bin/crsctl.bin"
env[1] @ 0x9ffffffffffff3aa: "MANPATH=/usr/share/man/%L:/usr/share/man:/usr/contrib/man/%L:/usr/cont
rib/man:/usr/local/man/%L:/usr/local/man:/opt/VRTS/vxfs5.0/man:/opt/ldapux/share/man/%L:/opt/ldapux/share/man:/opt/ipf/man:/opt/lda
pux/ypldapd/man:/opt/samba/man:/opt/samba/WTEC_Support_Tools/man:/opt/samba/cfsm_man:/opt/cifsclient/share/man:/opt/openssl/man:/op
t/openssl/prngd/man:/opt/wbem/share/man:/opt/graphics/common/man:/opt/amgr/man/%L:/opt/amgr/man:/opt/sec_mgmt/share/man:/opt/calipe
r/man/%L:/opt/caliper/man:/opt/resmon/share/man/%L:/opt/resmon/share/man:/opt/drd/share/man/%L:/opt/drd/share/man:/opt/dsau/man:/op
t/gnome/man:/opt/perf/man/%L:/opt/perf/man:/opt/propplus/share/man:/usr/contrib/kwdb/share/man:/opt/perl_32/man:/opt/perl_64/man:/o
pt/prm/man/%L:/opt/prm/man:/opt/psb/healthtest/share/man:/opt/sfm/share/man:/opt/swm/share/man/%L:/opt/swm/share/man:/opt/swa/share
/man/%L:/opt/swa/share/man:/opt/ssh/share/man:/opt/sentinel/man/%L:/opt/sentinel/man:/opt/langtools/share/man/%L:/opt/langtools/sha
re/man:/opt/wlm/share/man/%L:/opt/wlm"
env[2] @ 0x9ffffffffffff868: "LANG=C"
env[3] @ 0x9ffffffffffff86f: "VCS_LOG_RESOURCE_NAME=cssd"
env[4] @ 0x9ffffffffffff88a: "VCS_AGFW=1"
env[5] @ 0x9ffffffffffff895: "SHLIB_PATH=/crsvol/lib32:/opt/nmapi/nmapi2/lib/hpux32:"
env[6] @ 0x9ffffffffffff8cc: "VCSAG_REG_VERSION=4"
env[7] @ 0x9ffffffffffff8e0: "__ARIES_PA_AOUT_MAXSSIZ=8388608"
env[8] @ 0x9ffffffffffff900: "PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/
sbin:/usr/bin:/usr/ccs/bin:/usr/contrib/bin:/usr/contrib/Q4/bin:/opt/perl/bin:/opt/ipf/bin:/opt/gvsd/bin:/opt/nettladm/bin:/opt/fcm
s/bin:/opt/wbem/bin:/opt/wbem/sbin:/opt/sas/bin:/opt/graphics/common/bin:/opt/atok/bin:/usr/bin/X11:/usr/contrib/bin/X11:/opt/sec_m
gmt/bastille/bin:/opt/resmon/bin:/opt/drd/bin:/opt/dsau/bin:/opt/dsau/sbin:/opt/gnome/bin:/opt/perf/bin:/opt/propplus/bin:/usr/cont
rib/kwdb/bin:/opt/prm/bin:/opt/sfm/bin:/opt/swm/bin:/opt/sec_mgmt/spc/bin:/opt/ssh/bin:/opt/swa/bin:/opt/hpsmh/bin:/opt/langtools/b
in:/opt/wlm/bin:/opt/gwlm/bin:/opt/VRTS/bin:/opt/VRTSvcs/sbin:/opt/VRTSvcs/sbin:/opt/VRTSvxvm/bin:/opt/VRTSvcs/vxfen/bin:/etc/vx/bi
n:/usr/lib/vxvm/bin:/opt/VRTSvxfs/cfs/bin:/opt/caliper/bin:/opt/firefox:/opt/perl_32/bin:/opt/perl_64/bin:/opt/thunderbird:/opt/sen
tinel/bin:/sbin:/home/root"
env[9] @ 0x9ffffffffffffc73: "COLUMNS=158"
env[10] @ 0x9ffffffffffffc7f: "VCS_LOG_SCRIPT_NAME=online"
env[11] @ 0x9ffffffffffffc9a: "EDITOR=vi"
env[12] @ 0x9ffffffffffffca4: "PTHREAD_COMPAT_MODE=0"
env[13] @ 0x9ffffffffffffcba: "LOGNAME=root"
env[14] @ 0x9ffffffffffffcc7: "HOSTNAME=sliam31"
env[15] @ 0x9ffffffffffffcd8: "ERASE=^H"
env[16] @ 0x9ffffffffffffce1: "VCSONE_LOG_AGENT_NAME=Application"
env[17] @ 0x9ffffffffffffd03: "VCSONE_LOG_SCRIPT_NAME=online"
env[18] @ 0x9ffffffffffffd21: "VCSONE_LOG_RESOURCE_NAME=cssd"
env[19] @ 0x9ffffffffffffd3f: "VCS_HOME=/opt/VRTSvcs"
env[20] @ 0x9ffffffffffffd55: "__ARIES_PA_STACK_SIZE_max=8388608"
env[21] @ 0x9ffffffffffffd77: "SHLVL=2"
env[22] @ 0x9ffffffffffffd7f: "VCSONE_LOG_CATEGORY=10021"
env[23] @ 0x9ffffffffffffd99: "EAT_HOME_DIR=/opt/VRTSvcs"
env[24] @ 0x9ffffffffffffdb3: "SHELL=/sbin/sh"
env[25] @ 0x9ffffffffffffdc2: "HOSTTYPE=hppa2.0"
env[26] @ 0x9ffffffffffffdd3: "ORASYM=/crsvol/bin/crsctl.bin"
env[27] @ 0x9ffffffffffffdf1: "VCS_CONF=/etc/VRTSvcs"
env[28] @ 0x9ffffffffffffe07: "OSTYPE=hpux11.00"
env[29] @ 0x9ffffffffffffe18: "ORA_CRS_HOME=/crsvol"
env[30] @ 0x9ffffffffffffe2d: "EAT_CATALOG_DIR=/opt/VRTSvcs"
env[31] @ 0x9ffffffffffffe4a: "HOME=/"
env[32] @ 0x9ffffffffffffe51: "CLUSTER_DIAG=/var/VRTSvcs/diag//agents/Application"
env[33] @ 0x9ffffffffffffe84: "VCS_DIAG=/var/VRTSvcs"
env[34] @ 0x9ffffffffffffe9a: "LD_LIBRARY_PATH=/crsvol/lib:/opt/nmapi/nmapi2/lib/hpux64:/usr/lib:/op
t/VRTSvcs/lib:"
env[35] @ 0x9ffffffffffffeee: "TERM=xterm"
env[36] @ 0x9ffffffffffffef9: "ORACLE_HOME=/crsvol"
env[37] @ 0x9fffffffffffff0d: "VCS_LOG_AGENT_NAME=Application"
env[38] @ 0x9fffffffffffff2c: "__ARIES_PA_STACK_SIZE_cur=8388608"
env[39] @ 0x9fffffffffffff4e: "PWD=/opt/VRTSvcs/bin/Application"
env[40] @ 0x9fffffffffffff6f: "MACHTYPE=hppa2.0-hp-hpux11.00"
env[41] @ 0x9fffffffffffff8d:
env[42] @ 0x9fffffffffffff99:
env[43] @ 0x9fffffffffffffb0:
env[44] @ 0x9fffffffffffffc0:
env[45] @ 0x9fffffffffffffda:
env[46] @ 0x9fffffffffffffe3:
[5434] socket(AF_INET6, SOCK_DGRAM, 0) ...................................................................... = 3
[5434] setsockopt(3, SOL_SOCKET, SO_REUSEADDR, 0x9fffffffffffd1b8, 4) ....................................... = 0
[5434] bind(3, 0x6000000000771070, 28) ...................................................................... = 0
[5434] getsockname(3, 0x9fffffffffffd2f0, 0x9fffffffffffd1b0) ............................................... = 0
[5434] getpeername(3, 0x9fffffffffffd2f0, 0x9fffffffffffd1b0) ............................................... ERR#235 ENOTCONN
[5434] getsockopt(3, SOL_SOCKET, SO_SNDBUF, 0x9fffffffffffd444, 0x9fffffffffffd440) ......................... = 0
[5434] getsockopt(3, SOL_SOCKET, SO_RCVBUF, 0x9fffffffffffd444, 0x9fffffffffffd440) ......................... = 0
[5434] fcntl(3, F_SETFD, 1) ................................................................................. = 0
[5434] ioctl(3, FIONBIO, 0x9fffffffffffd52c) ................................................................ = 0
[5434] gettimeofday(0x9fffffffffffc520, NULL) ............................................................... = 0
[5434] socket(AF_UNIX, SOCK_STREAM, 0) ...................................................................... = 5
[5434] access("/tmp/.oracle/sOHASD_UI_SOCKET", F_OK) ........................................................ = 0
[5434] connect(5, 0x9fffffffffffbb60, 94) ................................................................... ERR#239 ECONNREFUSED
[5434] access("/tmp/.oracle/sOHASD_UI_SOCKET", F_OK) ........................................................ = 0
[5509] execve(0x9fffffffffffda00, 0x600000000001cfb0, 0x9ffffffffffff1f0) ................................... [entry]
[5509] exit(9) [implicit (execve failure)] .................................................................. WIFEXITED(9)

>Does it mean that there is some issue with FDs shared between the parent process and execed process.

Possibly. But it seems the tusc output is bogus. I.e. why say "implicit (execve failure)". Why not show execve(2) returning a bad errno value that we can trust?

Mangesh> The error displayed by tusc on terminal refers to the arguments of execve.

tusc: ttrace(TT_PROC_RDDATA, 5509, 0, 600000000001cfb0, 4, 9fffffffffffecb0): Bad addression") [32-bit] )
tusc: ttrace(TT_PROC_RDDATA, 5509, 0, 9ffffffffffff1f0, 4, 9fffffffffffecb0): Bad address

Thanks,
Mangesh

[Dennis Handly]

>[5509] exit(9) [implicit (execve failure)] . WIFEXITED(9)

I don't get this for a bad executable, I get errno 2.
What HP-UX version do you have? What tusc version?
Are you in a 32 bit process? Are you using Aries?

>The error displayed by tusc on terminal refers to the arguments of execve.
tusc: ttrace(TT_PROC_RDDATA, 5509, 0, 600000000001cfb0, 4, 9fffffffffffecb0): Bad address

Hmm, they didn't match the first time. But it appears tusc can display the parms.

[Chris Bertin]

Tusc reports an "implicit" exit when it gets an EXIT event but the syscall that resulted in the exit is not an exit syscall. If the TTS_INSYSCALL bit is set in the event flags, tusc reports which syscall caused the exit.

Another example is doing a 'tusc sleep 100' and killing the sleep command with a SIGHUP. Tusc will report:

exit(1) [implicit (sigtimedwait failure)]................ WIFSIGNALED(SIGHUP)

--
Regards, Chris (Chris Bertin, chris.bertin@hp.com)

[Chris Bertin]

Looking back at this and reading more of the thread, I should have also said, to follow up on the "bogus and why not show ..." comment, that errno not necessarily == exit value. If the process exits with an unexpected EXIT event, it's not a syscall event, i.e. there are no arguments to report, no errno. Just an exit value (which is interpreted using the encoding described in wait.h. Hence, exit(9), which doesn't necessarily have anything to do with errno 9.

Also, wrt the fact that tusc reports a ttrace error when reading some syscall arguments, it would indicate to me that the arguments are invalid. Which would somehow correlate to an exec failure in this case, IMHO.

So, either the program is passing bogus pointers to exec or the memory these pointers point to is being stepped on before the syscall.

[Dennis Handly]

>What does this indicate?

From what Chris said and the source, the problem is that execvp(2) has problems with the executable or getting memory for the process so it calls exit(SIGKILL).

Can you do this from the shell: /crsvol/bin/ohasd reboot

If you hadn't redirected stderr, you may have seen these message:
exec(2): insufficient swap or memory available.
exec(2): could not load a.out
exec(2): Could not load a.out due to swap reservation failure or due to insufficient user stack size

[Chris Bertin]

The message is a 'uprintf' and will go to the terminal, if there is one, regardless of stdout/err. 'uprintf' messages from the kernel cannot be redirected. The characters get pushed directly onto the tty queue.

The kernel message is also misleading. It will happen any time a 'copyout' fails (copyout copies data from kernel space to user space). Considering the fact that tusc can't read the exec arguments, I'd say that the issue is corruption of the binary, not lack of space.