We have installed our product in the Linux RHEL 5.5 machine. From our product we are using the system() function to spawn a different process which is passed as a command argument. The result is returned as -1 . It is currently happening only in one machine. The same code looks to be working correctly in other machines and the system() function returns the command status properly.
Is there any specific setting at the shell or kernel level because of this it can happen ? Any help on this would be appreciated.
Is /bin/sh available on the system that has the problem?
Is your application running setuid or setgid on the system that has the problem?
Maybe something in your application is causing the argument of the system() function to be NULL in that particular system?
Or maybe the system is so out of memory that it cannot create new processes at the moment?
Is SELinux enabled? If it is, is the product running in a security context that disallows creation of child processes?
(Run "ps -Zp <application_PID>" to find the SELinux security context label associated with the process.)
Back when I was on a "Programming in C 101" course at an University, the professor recommended against using the system() function in major applications. His rationale was that it had rather limited facilities for programmatic error detection, making it hard or impossible to tell exactly what went wrong. Using fork()/vfork(), execve() (or one of the other functions of the exec() family) and wait() separately would allow for better error detection and reporting.
In his opinion, system() was good for small quick-and-dirty programs, but not much else. I think your issue shows he knew what he was talking about.
