topic Re: Sudo memory error in Operating System - HP-UX

Sudo memory error

Mauro Livi — Fri, 17 Apr 2015 20:07:51 GMT

Hi all,

Next hurdle with my sudo configuration...seem to have a memory fault shwon below when running sudo

I haven't increased swap space yet...only because I'm thinking this is a 128GB machine (rx2800 i4) with nothing running on it yet. So I'm suspecting some kernel tuning would do the trick.

Message and swapinfo shown below. As always any input is greatly appreciated and welcomed.

Thanks

Mauro

Pid 10711 received a SIGSEGV for stack growth failure.
Possible causes: insufficient memory or swap space,
or stack size exceeded maxrsessiz
or stack size limit is set too small.
Memory fault

# swapinfo -tam
                 Mb          Mb           Mb   PCT      START/          Mb
TYPE          AVAIL        USED         FREE USED       LIMIT     RESERVE PRI
NAME
dev            8192           0         8192    0%           0           -    1
/dev/vg00/lvol2
reserve           -         554         -554
memory       124552       11687       112865    9%
total        132744       12241       120503    9%       -           0    -

Re: Sudo memory error

Mauro Livi — Fri, 17 Apr 2015 21:51:39 GMT

Well in checking the core file for this I see the following:

$ PATCH_11.31/PHCO_43464 Jan 21 2014 17:01:01 $
         $Revision: libpthread.so.1:    R11.31_HOT_BL2012_1101_1
        92453-07 linker dld HP Itanium(R) B.12.60
        92453-07 linker uld HP Itanium(R) B.12.60

And in checking PHCO_43464 I found the following Critical Warning issued by HP last Dec. (recommending the removal of the patch):

http://h20564.www2.hp.com/hpsc/doc/public/display?docId=pdb_na-PHCO_43464&sp4ts.oid=3367813

So I wonder if this is my problem...will do more digging. Stay tuned...if interested.

Mauro

Re: sudo received a SIGSEGV for stack growth failure

Dennis Handly — Fri, 17 Apr 2015 23:35:59 GMT

>stack size limit is set too small.

This is your problem. You need to increase maxssiz(5).

Use kctune(1m) to display the current value.

>I haven't increased swap space yet

No need, you have plenty free, 120 GB.

>I wonder if this is my problem

No, sudo shouldn't be using threads.

Re: Sudo memory error

RenatoMartini — Sat, 18 Apr 2015 00:53:39 GMT

Sure!

Me too... I had the same problem with sudo:

# sudo

Pid 7290 received a SIGSEGV for stack growth failure.
Possible causes: insufficient memory or swap space,
or stack size exceeded maxrsessiz
or stack size limit is set too small.
Segmentation fault (core dumped)

Re: sudo received a SIGSEGV for stack growth failure

Dennis Handly — Mon, 20 Apr 2015 07:03:11 GMT

An alternative failure cause may be in this topic:

http://h30499.www3.hp.com/t5/System-Administration/Memory-fault/td-p/6418372

Re: sudo received a SIGSEGV for stack growth failure

Mauro Livi — Mon, 20 Apr 2015 16:31:17 GMT

Thanks Dennis,

Will try increasing maxssize and will post results. In the meantime, here are my kctune results

Mauro

# kctune
Tunable                                 Value Expression Changes
NSTREVENT                                  50 Default
NSTRPUSH                                   16 Default
NSTRSCHED                                   0 Default
STRCTLSZ                                 1024 Default
STRMSGSZ                                    0 Default
acctresume                                  4 Default
acctsuspend                                 2 Default
aio_iosize_max                              0 Default     Immed
aio_listio_max                            256 Default     Immed
aio_max_ops                              2048 Default     Immed
aio_monitor_run_sec                        30 Default     Immed
aio_physmem_pct                            10 Default     Immed
aio_prio_delta_max                         20 Default     Immed
aio_proc_max                                0 Default     Immed
aio_proc_thread_pct                        70 Default     Immed
aio_proc_threads                         1024 Default     Immed
aio_req_per_thread                          1 Default     Immed
allocate_fs_swapmap                         0 Default
alwaysdump                                  0 Default     Immed
audit_memory_usage                          5 Default     Immed
audit_track_paths                           0 Default     Auto
base_pagesize                               4 Default
btree_prune_min_interval                   90 Default     Immed
close_wake_blkdflks                         0 Default
cmpt_restrict_tl                            0 Default     Immed
copy_on_write                               1 Default     Immed
core_addshmem_read                          0 Default     Immed
core_addshmem_write                         0 Default     Immed
create_fastlinks                            0 Default
default_disk_ir                             0 Default
desfree_pct                                 0 Default     Immed
diskaudit_flush_interval                    5 Default     Immed
dlpi_max_clones                          3992 Default     Immed
dlpi_max_ub_promisc                         1 Default     Immed
dma32_pool_size                     268435456 Default
dnlc_hash_locks                           512 Default
dontdump                                    0 Default     Immed
dst                                         1 Default
dump_compress_on                            1 Default     Immed
dump_concurrent_on                          1 Default     Immed
executable_stack                            0 Default     Immed
expanded_node_host_names                    0 Default     Immed
fcache_fb_policy                            0 Default     Immed
fcache_seqlimit_file                      100 Default     Immed
fcache_seqlimit_scope                       0 Default     Immed
fcache_seqlimit_system                    100 Default     Immed
fcache_vhand_ctl                            0 Default     Immed
fcd_disable_mgmt_lun                        0 Default     Immed
fclp_ifc_disable_mgmt_lun                   0 Default     Immed
fcoc_ifc_disable_mgmt_lun                   0 Default     Immed
fcq_disable_mgmt_lun                        0 Default     Immed
filecache_max                     65301360640 Default     Auto
filecache_min                      6530134016 Default     Auto
fr_rulecache                                0 Default     Immed
fr_statemax                            800000 Default     Immed
fr_tcpidletimeout                       86400 Default     Immed
fr_tcptimewait                            120 Default     Immed
frnat_tcptimewait                         120 Default     Immed
fs_async                                    0 Default
fs_symlinks                                20 Default     Immed
ftable_hash_locks                          64 Default
gvid_no_claim_dev                           0 Default
hires_timeout_enable                        0 Default     Immed
hp_hfs_mtra_enabled                         1 Default
intr_strobe_ics_pct                        80 Default     Immed
io_ports_hash_locks                        64 Default
ipf_icmp6_passthru                          0 Default     Immed
ipl_buffer_sz                            8192 Default     Immed
ipl_logall                                  0 Default     Immed
ipl_suppress                                1 Default     Immed
ipmi_watchdog_action                        0 Default     Immed
ipnat_enable                                1 Default     Immed
kmem_aggressive_caching                     0 Default     Immed
kmem_do_largepage_cache_demotion            1 Default     Immed
kmem_va_cache_enable                        1 Default
ksi_alloc_max                           33600 Default     Immed
ksi_send_max                               32 Default
lcpu_attr                                   0 0           Imm (auto disabled)
lotsfree_pct                                0 Default     Immed
max_acct_file_size                    2560000 Default     Immed
max_async_ports                          4096 Default     Immed
max_mem_window                              0 Default     Immed
max_thread_proc                          1200 1200        Immed
maxdsiz                            1073741824 Default     Immed
maxdsiz_64bit                      4294967296 Default     Immed
maxfiles                                 2048 Default
maxfiles_lim                             4096 Default     Immed
maxrsessiz                            8388608 Default
maxrsessiz_64bit                      8388608 Default
maxssiz                               8388608 Default     Immed
maxssiz_64bit                       268435456 Default     Immed
maxtsiz                             100663296 Default     Immed
maxtsiz_64bit                      1073741824 Default     Immed
maxuprc                                   256 Default     Immed
mca_recovery_on                             1 Default     Auto
mdep_reduce_rse_size                        0 Default
mpas_readonly_text                          0 Default     Immed
mprotect_reduce_protid_on                   0 Default     Immed
msgmbs                                      8 Default     Immed
msgmnb                                  16384 Default     Immed
msgmni                                    512 Default     Immed
msgtql                                   1024 Default     Immed
ncdnode                                   150 Default
nclist                                   8292 Default
ncsize                                   8976 Default
nflocks                                  4096 Default     Auto
nfs2_max_threads                            8 Default     Immed
nfs2_nra                                    4 Default     Immed
nfs3_bsize                              32768 Default     Immed
nfs3_do_readdirplus                         1 Default     Immed
nfs3_jukebox_delay                       1000 Default     Immed
nfs3_max_commit_threads                     8 Default     Immed
nfs3_max_threads                            8 Default     Immed
nfs3_max_transfer_size                1048576 Default     Immed
nfs3_max_transfer_size_cots           1048576 Default     Immed
nfs3_nra                                    4 Default     Immed
nfs4_bsize                              32768 Default     Immed
nfs4_max_threads                            8 Default     Immed
nfs4_max_transfer_size                1048576 Default     Immed
nfs4_max_transfer_size_cots           1048576 Default     Immed
nfs4_nra                                    4 Default     Immed
nfs_portmon                                 0 Default     Immed
ngroups_max                                20 Default     Immed
ninode                                   8192 Default
nkthread                                 8416 Default     Immed
nproc                                    4200 Default     Immed
npty                                       60 Default
nstrpty                                    60 Default
nstrtel                                    60 Default
nswapdev                                   32 Default
nswapfs                                    32 Default
numa_mode                                   0 Default
numa_policy                                 0 Default     Immed
numa_sched_launch                           1 Default     Immed
override_umask                              0 Default     Immed
pa_maxssiz_32bit                     83648512 Default     Immed
pa_maxssiz_64bit                    536870912 Default     Immed
pagezero_daemon_enabled                     1 Default     Immed
patch_active_text                           1 Default     Immed
pci_eh_enable                               1 Default
pci_error_tolerance_time                 1440 Default     Immed
process_id_max                          30000 Default     Auto
process_id_min                              0 Default     Auto
pwr_idle_ctl                                0 Default     Immed
remote_nfs_swap                             0 Default
rng_bitvals                        9876543210 Default
rng_sleeptime                               2 Default
rtsched_numpri                             32 Default
rusage_hires_enable                         0 Default     Immed
sched_thread_affinity                       6 Default     Immed
scroll_lines                              100 Default
secure_sid_scripts                          1 Default     Immed
semaem                                  16384 Default
semmni                                   2048 Default
semmns                                   4096 Default
semmnu                                    256 Default
semmsl                                   2048 Default     Immed
semume                                    100 Default
semvmx                                  32767 Default
shlib_debug_enable                          0 Default     Immed
shmmax                             1073741824 Default     Immed
shmmni                                    400 Default     Immed
shmseg                                    300 Default     Immed
streampipes                                 0 Default
swchunk                                  2048 Default
sysv_hash_locks                           128 Default
tcphashsz                                   0 Default
timeslice                                  10 Default
timezone                                  420 Default
uname_eoverflow                             1 Default     Immed
vnode_cd_hash_locks                       128 Default
vnode_hash_locks                          128 Default
vps_ceiling                                16 Default     Immed
vps_chatr_ceiling                     1048576 Default     Immed
vps_pagesize                               16 Default     Immed
vx_dexh_sz                             524287 Default     Immed
vx_era_nthreads                             5 Default
vx_maxlink                              32767 Default
vx_ninode                                   0 Default     Immed
vxfs_bc_bufhwm                              0 Default     Immed
vxfs_ifree_timelag                          0 Default     Immed

Re: sudo received a SIGSEGV for stack growth failure

Mauro Livi — Mon, 20 Apr 2015 17:32:09 GMT

Well I've increased maxssiz as shown below, but I still get the error below:

# kctune maxssiz
Tunable Value Expression Changes
maxssiz 134217728 134217728 Immed

Pid 4119 received a SIGSEGV for stack growth failure.
Possible causes: insufficient memory or swap space,
or stack size exceeded maxrsessiz
or stack size limit is set too small.
Memory fault

Could it be that maxressiz needs to be increased as well? I also find it interesting that if I read the core file generated it still references PHCO_43464 (below)

Has anyone been able to get this sudo to work? I downloaded sudo1.8.9p1 and it looks like there is a more recent version...so maybe I'll try that. Also wondering if there are more tuneables that should be addressed?

Always appreciative of any input.

Mauro

core:
         $ PATCH_11.31/PHCO_43464 Jan 21 2014 17:01:01 $
        92453-07 linker dld HP Itanium(R) B.12.60
        92453-07 linker uld HP Itanium(R) B.12.60

Re: sudo received a SIGSEGV for stack growth failure

Dennis Handly — Mon, 20 Apr 2015 17:28:47 GMT

>maxssiz 8388608 Default

You might want to go to 64 MB.

Re: sudo received a SIGSEGV for stack growth failure

Mauro Livi — Mon, 20 Apr 2015 17:34:11 GMT

Hi Dennis,

I've increased it (per the reply I just posted),

here is the kctune...btw many thanks for your input. It is appreciated.

Mauro

# kctune maxssiz
Tunable Value Expression Changes
maxssiz 134217728 134217728 Immed

Re: sudo received a SIGSEGV for stack growth failure

Dennis Handly — Mon, 20 Apr 2015 18:47:37 GMT

>I've increased maxssiz as shown below, but I still get the error below:

Then it is a software or config problem.

What does gdb show on the corefile for a stacktrace (bt)?

>Could it be that maxressiz needs to be increased as well?

Possible but I've only seen a RSE stackoverflow only a few times in a highly recursive program with large register frames.

>I also find it interesting that if I read the core file generated it still references PHCO_43464

If you haven't replaced PHCO_43464 why should it?

Re: sudo received a SIGSEGV for stack growth failure

Mauro Livi — Mon, 20 Apr 2015 19:23:44 GMT

Dennis,

This will definitely show how "green" I am on core debugging, but I'll give it a try. The core file generated by sudo is in / so I ran the gdb command:

gdb -c core (running it from /).

The output is below. Anything useful there? If not, what would be a more appropriate way to run gdb?

Thanks

warning: Unknown symbols for 'core'; use the 'symbol-file' command.
Core was generated by `hagent'.
Program terminated with signal 11, Segmentation fault.
SEGV_MAPERR - Address not mapped to object
#0 0x400a7f0 in <unknown_procedure> ()
(gdb) bt
#0 0x400a7f0 in <unknown_procedure> ()
warning: Attempting to unwind past bad PC 0x400a7f0
#1 0x400a700 in <unknown_procedure> ()
(gdb)

Re: sudo received a SIGSEGV for stack growth failure

Dennis Handly — Mon, 20 Apr 2015 19:57:11 GMT

>gdb -c core (running it from /).

You need to pass in the path to the sudo executable:

gdb path-to-sudo -c core

>Core was generated by `hagent'.

This says that hagent aborted.

Re: sudo received a SIGSEGV for stack growth failure

Mauro Livi — Mon, 20 Apr 2015 20:22:42 GMT

Dennis,

My apologies, but I'm more confused now. I re-ran gdb per your suggested syntax

gdb path-to-sudo -c core

However, gdb seems to be expecting a core file in the directory where gdb resides (/opt/langtools/bin). There is a core file in the sudo directory (/usr/local/bin), so I'm obvioulsy confused as to why it would be looking for it elsewhere.

# ./gdb /usr/local/bin/sudo -c core

HP gdb 6.5 for HP Itanium (32 or 64 bit) and target HP-UX 11iv2 and 11iv3.
Copyright 1986 - 2011 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 6.5 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.
..
/opt/langtools/bin/core: No such file or directory.
(gdb)

Re: sudo received a SIGSEGV for stack growth failure

Mauro Livi — Mon, 20 Apr 2015 20:34:01 GMT

Well more than one way to skin a cat :)

I just copied the core file from /usr/local/bin and re-ran gdb. Output is below

# ./gdb /usr/local/bin/sudo -c core
HP gdb 6.5 for HP Itanium (32 or 64 bit) and target HP-UX 11iv2 and 11iv3.
Copyright 1986 - 2011 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 6.5 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.
..
Core was generated by `sudo'.
Program terminated with signal 11, Segmentation fault.
SEGV_MAPERR - Address not mapped to object
#0 0x60000000c007db20:0 in LE_sym_name+0 () from /usr/lib/hpux32/dld.so
(gdb) bt
#0 0x60000000c007db20:0 in LE_sym_name+0 () from /usr/lib/hpux32/dld.so
#1 0x60000000c007de10:0 in LL_best_fit+0x130 () from /usr/lib/hpux32/dld.so
#2 0x60000000c007b4a0:0 in LL_sym_search+0x3e0 () from /usr/lib/hpux32/dld.so
#3 0x60000000c00c68a0:0 in _shl_findsym+0x400 () from /usr/lib/hpux32/dld.so
#4 0x60000000c0007680:0 in shl_findsym+0xd0 () from /usr/lib/hpux32/libdl.so.1
#5 0x404bb40:0 in sudo_dso_findsym () at ./sudo_dso.c:137
#6 0x40125b0:0 in getenv () at ./env_hooks.c:86
#7 0x40125f0:0 in getenv () at ./env_hooks.c:76
#8 0x40125f0:0 in getenv () at ./env_hooks.c:76
#9 0x40125f0:0 in getenv () at ./env_hooks.c:76
#10 0x40125f0:0 in getenv () at ./env_hooks.c:76
#11 0x40125f0:0 in getenv () at ./env_hooks.c:76
#12 0x40125f0:0 in getenv () at ./env_hooks.c:76
#13 0x40125f0:0 in getenv () at ./env_hooks.c:76
#14 0x40125f0:0 in getenv () at ./env_hooks.c:76
#15 0x40125f0:0 in getenv () at ./env_hooks.c:76
#16 0x40125f0:0 in getenv () at ./env_hooks.c:76
#17 0x40125f0:0 in getenv () at ./env_hooks.c:76
#18 0x40125f0:0 in getenv () at ./env_hooks.c:76
#19 0x40125f0:0 in getenv () at ./env_hooks.c:76
#20 0x40125f0:0 in getenv () at ./env_hooks.c:76
#21 0x40125f0:0 in getenv () at ./env_hooks.c:76
#22 0x40125f0:0 in getenv () at ./env_hooks.c:76
---Type <return> to continue, or q <return> to quit---
#23 0x40125f0:0 in getenv () at ./env_hooks.c:76

Re: sudo received a SIGSEGV for stack growth failure

Dennis Handly — Tue, 21 Apr 2015 10:55:20 GMT

>I just copied the core file from /usr/local/bin and re-ran gdb.

(You are suppose to provide the absolute path to each file.)

It looks like sudo has illegally defined its own copy of getenv and when it tries to get the address of the real getenv(3), it finds itself and is recursive!

If it's going to look for libc symbols, it should pass shl_findsym something to tell it to look only in libc.so.

sudo shouldn't be using the obsolete shl_findsym(3) but use the standard dlsym(3).

Re: sudo received a SIGSEGV for stack growth failure

Mauro Livi — Tue, 21 Apr 2015 15:00:56 GMT

Thanks Dennis,

So it sounds as though it may be something related with sudo as opposed to any more kernel tunables. At this juncture I think I might remove the current sudo install and apply the more current one from the HP-UX porting Archive center (maybe a later version will work)

http://hpux.connect.org.uk/

If that doesn't work, I might get the one from the HP depot...which I was trying to avoid since it has more things included in its package (I wanted to install sudo by itself).

https://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=HPUXIEXP1131

Re: sudo received a SIGSEGV for stack growth failure

Dennis Handly — Wed, 22 Apr 2015 06:52:07 GMT

>it sounds as though it may be something related with sudo as opposed to any more kernel tunables.

Exactly. But I'm surprised it works for anyone on HP-UX? Or it only fails with a certain configuration in sudoers?

Perhaps something to do with ignoring env vars?

Re: sudo received a SIGSEGV for stack growth failure

Mauro Livi — Wed, 22 Apr 2015 17:27:14 GMT

Well I don't know, but I just removed the 1.8.9p1 and installed the newer version of sudo (1.8.13) and it worked.

I simply installed with the downloadhelper script as before, created my sudoers file (make sure sudoers is in both /etc and /usr/local/etc). I ran the sudo command and that worked.

So in my particular environment, I was able to run 1.8.13 without any additional changes to the kernel tunables. Hope this helps someone else down the line if they run into similar problems.

Thanks again (Dennis) for all your input.

Mauro