Operating System - HP-UX
1833420 Members
2876 Online
110052 Solutions
New Discussion

Memory fault starting process at boot level on HPUX 11.31

 
umareddy
New Member

Memory fault starting process at boot level on HPUX 11.31

Hi Experts,

I am starting couple of process at boot level from the "/sbin/rc3.d". Some reasons , process are not getting started and they are showing Memory Fault in the rc.log file. I am able to started them manually after system up and ruing.

Please give your suggesation. why they are not starting during the system reboot.? we have same type of scripts works fine on HPUX 11.21 . is there any change in 11.31 for starting the script from rc3.d?

See the error below from rc.log.

Output from "/sbin/rc3.d/S985jboss start":
----------------------------
Starting JBOSS


Output from "/sbin/rc3.d/S997rc.ugs.FSC_peocad3_imanfs start":
----------------------------
/sbin/rc3.d/S997rc.ugs.FSC_peocad3_imanfs[22]: 1944 Memory fault
"/sbin/rc3.d/S997rc.ugs.FSC_peocad3_imanfs start" FAILED

Starting the amgrd subsystem
Output from "/sbin/rc3.d/S998amgrd start":
----------------------------
"/sbin/rc3.d/S998amgrd start" SKIPPED

Starting TcEng Pool Manager
Output from "/sbin/rc3.d/S999rc.tceng.servermanager start":
----------------------------
Starting Teamcenter Servers on peocad3
/sbin/rc3.d/S999rc.tceng.servermanager[39]: 1950 Memory fault

ERROR: Failed to start server pool.
Please refer to /tmp/tcserverpool*.log for error messages.
"/sbin/rc3.d/S999rc.tceng.servermanager start" FAILED


Output from "/sbin/rc3.d/S999rc.ugs.tcfs start":
----------------------------
/sbin/rc3.d/S999rc.ugs.tcfs[22]: 1956 Memory fault
"/sbin/rc3.d/S999rc.ugs.tcfs start" FAILED

Starting PRNGD (Pseudo Random Number Generator Daemon)
Output from "/sbin/rc3.d/S99prngd.rc start":
----------------------------
"/sbin/rc3.d/S99prngd.rc start" SKIPPED

**************************************************
HP-UX run-level transition completed
Sun Mar 15 12:16:26 CDT 2009
**************************************************
10 REPLIES 10
Jov
Honored Contributor

Re: Memory fault starting process at boot level on HPUX 11.31

Hi,

Its probably due to the memory foot print the Agent can spawn a process as JBoss tend to use large chunks of memory.

Find the actual start command within the startup script and run it with 'exec' in front of it.


Jov
umareddy
New Member

Re: Memory fault starting process at boot level on HPUX 11.31

Jov,

Thanks for your reply. I am using nohup for run execuatble. You want me replace nohup with exec

nohup ${TC_ROOT}/bin/tcfs -t=tcpip -s=${SERVICE_NAME} > /tmp/${SERVICE_NAME}$$.log &

change to like below

exec ${TC_ROOT}/bin/tcfs -t=tcpip -s=${SERVICE_NAME} > /tmp/${SERVICE_NAME}$$.log &

Thanks,
Umareddy
Jeroen Peereboom
Honored Contributor

Re: Memory fault starting process at boot level on HPUX 11.31

L.S.

Better post this in the HP-UX forum http://forums11.itrc.hp.com/service/forums/categoryhome.do?categoryId=156

I.m.h.o you should not use nohup, especially if a service is needed by services started later on, because the rc script (that executes all startup scripts) will just start all Sxxx scripts without waiting for succesful startup of each individual Sxxx script.
It they are dependent, add a sleep 30 to each startup script that is a prerequisite for one of the following services. This gives the service the time needed for startup.

(If they are not depending on each other, this will not help you).

HtH,

JP.
umareddy
New Member

Re: Memory fault starting process at boot level on HPUX 11.31

JP,

Some of them are depended. I will give the wait time and try.

Thanks,
Umareddy
Jeroen Peereboom
Honored Contributor

Re: Memory fault starting process at boot level on HPUX 11.31

If adding sleep statements helps, you should try and get rid of the nohup (and the sleep too). Al least, that's the way I think it should be...

JP.
Jov
Honored Contributor

Re: Memory fault starting process at boot level on HPUX 11.31

Hi,

You can always add some logging into your start up script and have it output to a file in /tmp to troubleshoot.

Make a copy or backup your original startup scripts.

When you test the commands, are you should no other JBoss instances were running?

'nohup' tells the script not to terminal even if the parent gets killed or terminates, but 'exec' actually forks a new process. I dont think your issue is due to nohup, but the environment the OVO/OMU Agent starts the script/program.

do a man on 'set' and read '-x' options to enable shell debugging, do within a normal shell so you understand how it works before using it under an agent.

Cheers
Jov
Jeroen Peereboom
Honored Contributor

Re: Memory fault starting process at boot level on HPUX 11.31

Jov is right.
It's not the 'nohup' but the '&' that causes all scripts to be started by rc without waiting. Furthermore, when the system boots the environment variables differ from your envvars when you manually execute the scripts.

By the way, 'we have the same type of scripts'? Do they start the same daemons?

JP
Dennis Handly
Acclaimed Contributor

Re: Memory fault starting process at boot level on HPUX 11.31

>Please give your suggestion. why they are not starting during the system reboot?

I don't see any core files?
You could arm a signal handler and call U_STACK_TRACE and that may give you an idea where it is failing and why.
umareddy
New Member

Re: Memory fault starting process at boot level on HPUX 11.31

Sorry for late reply. This issue on production server So, I didn't get the window to test the your suggesation till today.

JP, not the same daemons.
I tried with exec but the same error and also changed sleep time no luck.

Dennis,
I am not having the access to the code and also nor core file dumpped.


Here is my one of the script
# Set default return status - 0 for success.
rval=0
#
[ -x $0 ] || {
# Execute permission check, if not, quietly exit.
rval=2
exit $rval
}
#
# Check for command line arguments
SWITCH=$1
case $SWITCH in
start_msg) # Issue startup message for checklist table and exit
echo "Starting TcEng Pool Manager"
exit 0 ;;
stop|stop_msg) # No action - exit status 2 is "n/a" for HP
exit 2 ;;
*) # Continue with script
break ;;
esac
#
# Set PATH environment variable
PATH=/usr/sbin:/usr/bin:/sbin:/bin:/etc ; export PATH
#
# Set IMAN_ROOT environment variable.
IMAN_ROOT=/storage/appl/tceng2007/tc2007root ; export IMAN_ROOT
# Set IMAN_DATA environment variable.
IMAN_DATA=/nfs/tce8comm/hp_unix/tceng2007cad ; export IMAN_DATA
#
# Check if the daemon is there and executable.
if [ -x ${IMAN_ROOT}/pool_manager/mgrstartProduction ] ; then
echo " Starting Teamcenter Servers on `uname -n`"
su infodba -c /bin/ksh <<-EOF
IMAN_ROOT=/storage/appl/tceng2007/tc2007root
export IMAN_ROOT
IMAN_DATA=/nfs/tce8comm/hp_unix/tceng2007cad
export IMAN_DATA
IMAN_TMP_DIR=/tmp
export IMAN_TMP_DIR
cd /tmp
nohup sh "${IMAN_ROOT}/pool_manager/mgrstartProduction > /tmp/tcserverpool_test.log &" > /dev/null
sleep 60

EOF
else
# server pool script missing
echo "\nERROR:\tUnable to locate server manager script."
rval=1
fi

# Check to see if daemon is running. If not, fail.
if [ `ps -ef|grep mgrstart|grep -v grep|wc -l` -eq 0 ] ; then
echo "\nERROR:\tFailed to start server pool."
echo "\tPlease refer to /tmp/tcserverpool*.log for error messages."
rval=1
fi

exit $rval

----------------
I am very new to unix, please help me to add the debug lines
thanks,
Umareddy
Dennis Handly
Acclaimed Contributor

Re: Memory fault starting process at boot level on HPUX 11.31

>please help me to add the debug lines

You can add "set -x" to trace the script.

>I am not having the access to the code and also nor core file dumped.

That's one problem, there isn't a core file.

>Here is my one of the script

Since there are more than 3 lines in the script, (One, two, three, many), I can't find the line in question. Is it mgrstartProduction?

/sbin/rc3.d/S997rc.ugs.FSC_peocad3_imanfs[22]: 1944 Memory fault
/sbin/rc3.d/S999rc.tceng.servermanager[39]: 1950 Memory fault
/sbin/rc3.d/S999rc.ugs.tcfs[22]: 1956 Memory fault

The bad lines are most likely the numbers in [].