topic Re: received a SIGSEGV for stack growth failure in Operating System - HP-UX

received a SIGSEGV for stack growth failure

Nick Raceu — Mon, 12 Mar 2007 09:36:25 GMT

Hello,

After installing the DST patch (PHCO_35991) and rebooting the machine, I can not use AUTOSYS software. I get the following error: "Pid 8429 received a SIGSEGV for stack growth failure. Possible causes: insufficient memory or swap space, or stack size exceeded maxssiz."
I'm new to HP-UX so please let me know what info should I provide.
I'm running HP-UX B.11.00 and Oracle 8.0.6
Thanks

Re: received a SIGSEGV for stack growth failure

A. Clay Stephenson — Mon, 12 Mar 2007 09:42:35 GMT

I very much doubt that the tztab patch had anything to do with your problem. It is probable that you either exceeded a kernel limit (maxdsiz or maxssiz) or more swap is needed.

Please post the output of these commands:

kmtune
swapinfo -tam

Re: received a SIGSEGV for stack growth failure

Pete Randall — Mon, 12 Mar 2007 09:45:50 GMT

Nick,

I have to wonder if these two aren't just coincidence. The DST patch really only affect the /usr/lib/tztab file which shouldn't have any affect on maxssize. Check the date/time stamps on /stand/system, /stand/vmunix and /stand/vmunix.prev, compared against /etc/shutdownlog. I'm wondering if someone played with maxssiz, then built a new kernel but never re-booted with that new kernel.

Pete

Re: received a SIGSEGV for stack growth failure

Nick Raceu — Mon, 12 Mar 2007 09:51:56 GMT

Hi Clay,
here is the info:
/home/root# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1024 0 1024 0% 0 - 1 /dev/vg00/lvol2
reserve - 226 -226
memory 998 102 896 10%
total 2022 328 1694 16% - 0 -

Re: received a SIGSEGV for stack growth failure

Nick Raceu — Mon, 12 Mar 2007 09:59:17 GMT

Hi Pete,

Looks like the system has been rebooted several times since the last kernel has been re-built:
root@chapters:/stand# ls -ltr
total 70166
drwxr-xr-x 2 root root 8192 Dec 20 2000 lost+found
-rw-r--r-- 1 root sys 19 Dec 20 2000 bootconf
-r--r--r-- 1 root sys 82 Dec 20 2000 kernrel
drwxr-xr-x 2 root sys 1024 Dec 21 2000 system.d
-r--r--r-- 1 root sys 1103 Dec 22 2000 system.prev2
-rw-rw-rw- 1 root sys 980 Feb 1 2001 system_chapter1
-rwxr-xr-x 1 root sys 11622256 Feb 23 2001 vmunix.prev2
-rw-rw-rw- 1 root sys 1038 Jan 19 2002 system.prev
-rwxr-xr-x 1 root sys 12117872 Jan 19 2002 vmunix.prev
drwxr-xr-x 5 root sys 1024 Jan 19 2002 dlkm.vmunix.prev
-rw-rw-rw- 1 root sys 1045 Jan 19 2002 system
-rwxr-xr-x 1 root sys 12117872 Jan 19 2002 vmunix
drwxr-xr-x 3 root sys 2048 Jan 19 2002 build

Here are some rebootes in the last 3 years:
16:38 Thu Feb 19, 2004. Reboot: (by chapters!root)
16:22 Wed Jul 21, 2004. Reboot: (by chapters!root)
11:37 Thu Sep 16, 2004. Reboot: (by chapters!root)
14:57 Tue Mar 1, 2005. Reboot: (by chapters!root)
17:40 Thu Dec 8, 2005. Reboot: (by chapters!root)
14:40 Fri Mar 9, 2007. Reboot: (by chapters!root)

Re: received a SIGSEGV for stack growth failure

A. Clay Stephenson — Mon, 12 Mar 2007 10:06:49 GMT

Your box is running with the out-of-the-box settings for maxdsiz, maxtsiz, and maxssiz.

Your current settings have maxssiz set to 8MiB and maxdsiz and maxtsiz both set to 64MiB.

I would increase maxssiz to 32MiB, maxdsiz to 512MiB, and maxtsiz to 128MiB and see if your problems go away.

Re: received a SIGSEGV for stack growth failure

Nick Raceu — Tue, 13 Mar 2007 10:40:57 GMT

Clay, I have increased maxssiz to 32MiB, maxdsiz to 512MiB, and maxtsiz to 128MiB, however i still get same error when trying to start the AutoSys event server.
Thanks
Nick

Re: received a SIGSEGV for stack growth failure

Don Morris_1 — Tue, 13 Mar 2007 12:26:05 GMT

That message is only generated when there really *is* a stack growth failure.

You raised maxssiz -- which is the most obvious thing to try [though I agree with you that the kernel settings should have been the same].

Next most obvious is swap available (since growth of any virtual object requires swap reservation), your swapinfo output shows plenty of space available... I presume you looped swapinfo or used Glance to monitor to ensure that AUTOSYS doesn't consume most of the swap just starting up (for Data, etc.) and then get the SIGSEGV for swap... and tear itself down so you see plenty of resources again.

Third in the list is lockable memory (mlock/plock interfaces) if this application is allowed to lock pages. Again, with so much available -- this shouldn't be coming into play unless the Text/Data/non-stack allocations of the application consume all/most of the memory first.

To address the points above, if you haven't already monitor the system state using Glance or other tools while you try to run AUTOSYS -- just to ensure it doesn't spike up and then return the system to previous state.

Assuming that it isn't doing this (and my gut is that it isn't), I would expect that AUTOSYS is actually having a recursion loop and truly exhausting the stack. If it appears to run longer (if the wall clock time is measurable) after raising maxssiz, that would support this theory. Additionally, you could/should check for a core file and see if the stack object within the core file is sized at or near maxssiz and what the stack trace was for the application at the time.

Re: received a SIGSEGV for stack growth failure

A. Clay Stephenson — Tue, 13 Mar 2007 13:26:21 GMT

That message is only generated when there really *is* a stack growth failure.

It would be more accurate to say that that message was probably generated on a stack growth failure but the actual message is entirely dependent upon what signal handler was in place for SIGSEGV for the process at the time.

Autosys is extremely easy to misconfigure; I would suggest that you contact Computer Associates for support.

Re: received a SIGSEGV for stack growth failure

Nick Raceu — Tue, 13 Mar 2007 15:18:16 GMT

Unfortunatley i dont have support from Ca and also, nothing has been changed on the AUTOSYS app, just the DST patch for the HP-UX and the reboot.
Nick

Re: received a SIGSEGV for stack growth failure

Dennis Handly — Tue, 13 Mar 2007 19:12:27 GMT

This error is typically a coding error caused by infinite recursion. Setting maxssiz larger will only make your core files bigger.

>Don: I would expect that AUTOSYS is actually having a recursion loop and truly exhausting the stack.

Exactly.

>Don: you could/should check for a core file and see if the stack object within the core file is sized at or near maxssiz and what the stack trace was for the application at the time.

Having the same set of function in a stack trace will show this. You need to use gdb's "bt" command:
$ gdb autosys-exec core
(gdb) bt

>Clay: but the actual message is entirely dependent upon what signal handler was in place for SIGSEGV for the process at the time.

You have to go out of your way to provide a handler for stack overflow. You have to call sigstack(2), sigaltstack(2) or sigspace(2).

So if you get the message, you have a stack overflow.

Re: received a SIGSEGV for stack growth failure

Don Morris_1 — Wed, 14 Mar 2007 10:09:54 GMT

No, I'm serious. That message is generated when a virtual fault is taken, we determine that it is a stack fault -- but the stack fault can not be satisfied. It is a kernel generated message -- it won't matter what the signal handlers do with the SIGSEGV, that message is generated first. Only an unsatisfied stack fault will generate it.

Re: received a SIGSEGV for stack growth failure

Nick Raceu — Wed, 14 Mar 2007 10:35:32 GMT

Hi All,

As I said initially, I have not much experience in HP-UX (actually Unix in general), however now I got an error from OMNIBACKUP on the same box, and there were no changes done on omnibackup nor autosys, and they were working for the last 5 years, so it looks to me more as a problem on the server instead of a bad application.

"operator@chapters:/home/operator> xomni
Starting GUI...
Please wait, this may take some time...

Pid 4150 received a SIGSEGV for stack growth failure.
Possible causes: insufficient memory or swap space,
or stack size exceeded maxssiz.
/opt/omni/bin/xomni[122]: 4150 Memory fault(coredump)"

Thanks and I appreciate all help received.

Re: received a SIGSEGV for stack growth failure

Dennis Handly — Wed, 14 Mar 2007 20:10:35 GMT

>Don: No, I'm serious.

I'm not sure who you were responding to here?

>but the stack fault can not be satisfied. It is a kernel generated message -- it won't matter what the signal handlers do with the SIGSEGV, that message is generated first.

If by this you mean that "satisfied" can be done with sigstack(2), sigaltstack(2) or sigspace(2), then there won't be that message?

>so it looks to me more as a problem on the server instead of a bad application.

As long as you have maxssiz reasonable and swapspace, you should be ok.

Pid 4150 received a SIGSEGV for stack growth failure. ...
/opt/omni/bin/xomni[122]: 4150 Memory fault(coredump)"

As suggested, you should use gdb to get a stack trace, that may suggest other solutions.

You need to look at line 122 of xomni to see what executable was being run.

Re: received a SIGSEGV for stack growth failure

Don Morris_1 — Thu, 15 Mar 2007 09:00:56 GMT

Dennis> "I'm not sure who you were responding to here?"

I was responding to Mr. Stephenson -- manual quoting is the bane of my existence.

Dennis> "If by this you mean that "satisfied" can be done with sigstack(2), sigaltstack(2) or sigspace(2), then there won't be that message?"

Nope -- I mean statisfied from VM's/the fault handlers point of view (the virtual address is legal for the process, we can grow the stack to cover it, all physical/lockable/swap reservations are satisfied, etc.). If VM can't satisfy the fault -- you get the message, and the process gets a SIGSEGV.
The sig calls you mention will relate to how the process handles the SIGSEGV (it may well handle it), but the kernel will still log that we sent it for this case.

Dennis> "As suggested, you should use gdb to get a stack trace, that may suggest other solutions."

Seconded.

Re: received a SIGSEGV for stack growth failure

Nick Raceu — Thu, 15 Mar 2007 10:02:35 GMT

After all looks like a coruption in the autosys database caused the problem.

Re: received a SIGSEGV for stack growth failure

Dennis Handly — Thu, 15 Mar 2007 17:14:23 GMT

>Don: If VM can't satisfy the fault -- you get the message, and the process gets a SIGSEGV. The sig calls you mention will relate to how the process handles the SIGSEGV (it may well handle it), but the kernel will still log that we sent it for this case.

That has not been my experience. (I just tried it.) If you call sig, you don't get the message because the stack overflow can be handled.