daemon start at boot vs. after boot

[John Kittel]

I'm having some trouble with some 3rd party vendor software. I have a tech support call open with the vendor, but want to post it here as well.

HP-UX 11.11.

The application runs a daemon, which accepts client connections and starts a server process for each connection. In the past it hasn't mattered whether the daemon is started via rc script at boot time, or after boot from the root account command line. After a recent upgrade it only works if started from rc script at boot. If started from the command line after boot, the client communication server processes are dying immediately after the daemon creates them.

I'm using the same script when starting from the command line that is used at boot. I also tried starting from cron after boot, thinking it would more closeley duplicate the boot environment, but have the same problem.

I modified the start script to log some environment information, then ran it both ways. The differences are few and seem completely inconsequential, - such things as environment variables for rwhod, snmp agents, sendmail, etc. in one environment but not the other, - nothing that would actually be used by the daemon.

Any suggestions about what else I can try?

- John

[A. Clay Stephenson]

Was this an upgrade of the application or the OS?

I would do this which should exactly duplicate what the rc script does:

/sbin/init.d/yourrcscriptname stop # stops it
/sbin/init.d/yourrcscriptname start # starts it

[A. Clay Stephenson]

It could be something as simple as umask.

[John Kittel]

This was an upgrade of the application.

I have tried invoking the script in the way in which you suggest, both from the command line, as as a cron job. I still have the same problem.

- John

[John Kittel]

Thanks, I'll try working with the umask...

- John

[Torsten.]

You have to read the startup script carefully line by line trying to understand what happens. Most likely there are some env. variables needed by the script. I'm pretty sure your application is logging to a file (syslog/rc.log/your_applications.log). Have a look for any information.

[A. Clay Stephenson]

This is really a vendor problem and they should be able to resolve it quickly. I suspect what is really happening is that there is already a process listening on the ports because the listeners are no longer being properly killed. Netstat (or better, lsof) should prove useful in tracking down this possible problem.

[John Kittel]

The vendor is unable to duplicate the problem on their test system.

I don't think it is a problem with listener not being properly killed, process already listening on the port, because if I remove the rc script altogether and reboot, then start the daemon from the command line, I have the same problem.

- John

[Kofi ARTHIABAH]

John:
Have you tried starting the script in debug mode? ie. sh -x /sbin/init.d/rcscript start?
maybe another thing you can try is to use tusc http://hpux.connect.org.uk/hppd/hpux/Sysadmin/tusc-7.7/ to start the daemon and see why it fails (when it fails from the command line)

[John Kittel]

I don't think running the script in debug mode will help. The script runs fine. It starts the daemon. The daemon then starts other processes to communicate with clients when clients connect. It is these other processes that are dying, not the daemon started by the startup script.

The startup script is fairly simple, just a few lines. It sets some variables and spawns the daemon process. Since it is the same script running in both cases, at boot or after boot, I think it must be the environment setup before the script runs that is different.

- John

[Kent Ostby]

Too ideas .. one is to put a -x into the script and run it that way.

The second is to check and make sure you have full path names / environmental setting configured properly.

[Kofi ARTHIABAH]

tusc might actually give you a clue as to what is happening (or not happening)
1. start it up in the normal mode, then use tusc to attach to the spawned daemon to observe it in action.
2. then try starting it up from the command prompt with tusc -f spawned_daemon (which should fail right?). You can compare the output of the tusc run from 1 - there might be a clue to what the problem is.

Good luck.

[John Kittel]

Thanks everyone. I will give set -x a try, explore umask settings, and have a look at tusc. I don't have tusc installed so I'll have to see what's involved there, how much time it will take, etc. Also, I can't reboot at the moment; it will take a while to schedule that.

- John

[John Kittel]

The script uses full path names.

- John

[John Kittel]

tusc was easy to install. I'll see what it can tell me...

- John

[John Kittel]

additional test results:

umask is the same whether the script is run as boot script or from command line after boot, 022.

output of set -x is identical whether the script is run as boot script or from command line after boot.

tusc is giving me a lot of info, but so far I haven't found the smoking gun.

- John

[A. Clay Stephenson]

Using tusc is rather tricky in that the problem doesn't seem to be with the daemon itself but rather with the child process that is fork()'ed and exec()'ed -- so that there is hardly time for tusc to be started.

You should determine if this is a multi-threaded application or does it do a fork-exec for each client connection. You can use tusc to monitor the daemon and depending upon the system calls invoked, you will know that. This will narrow down the possible resources being used.

You should also note that there is another difference when run manually or when executed by rc and that is that stdin, stdout, and stderr are different.

Yours is a perfect example of why every application, patch, and/or upgrade should first be deployeded on a Sandbox so that you know about the bugs before they actually bite you. You are really going to have to work with your vendor to replicate the problem. Compare kernel tunings and most especially compare swlist outputs with their box. I'm betting on a difference in shared libraries so a comparison of ldd outputs for both deamons would be helpful along with a cksum of each shared library listed.

If you don't have a Sandbox, now is a good time to talk with your powers-that-be to get one. Moreover, a good old slow box from the used-equipment market makes a better Sandbox than a brand-new lightning-quick box. Why? Because if it is a dog when running on a slow lightly loaded Sandbox, it's guaranteed to be a dog in Production.

[John Kittel]

Thank you for your extensive comments A. Clay. I will pursue these avenues ( kernel tunings, swlist output, etc.).

The daemon does a fork-exec for each client connection.

(BTW, I tried using the -f switch on tusc, to follow the fork, but using that in the daemon started by the boot process goofs up the forked process so that it doesn't work properly. It is unable to connect properly to the oracle database.)

Your comments about needing a sandbox system are well taken. I neglected to mention it earlier, but this problem is on our development / test system for this application. This upgrade / problem has not made it onto our production system.

- John

[John Kittel]

update: not getting too far with the vendor. They don't seem to want to seriously entertain the idea of comparing versions, tuning, etc. on our system with theirs.

Something else interesting has just come up. We've been testing and developing with some other vendor's software on our same HP-UX system. It seems to be exhibiting a similar problem. Not entirely sure yet. But it starts a daemon, which fork-execs proceeses for each client connection. The daemon seems fine, but the other processes are dying.

Now I'm wondering if this problem may be caused by something in our HP-UX installation, not specifically with the vendor's software...

- John