Disconnected sessions on HP-UX

[Kevin Liquori_1]

We have two HP-UX 11.23 vPar hosts on a Superdome. One server runs an ERP application which connects to the other server running an Oracle database. We've been seeing users randomly getting dropped from the application with database termination errors. The application logs state this is an "IPC" problem.

We have not been able to find a remedy to this and we cannot prove whether the issue is with the application, database, operating system, or hardware.

On both servers I've used lanadmin to look for errors, but I have not seen any. I also don't see any errors in syslog. I've also checked with our network group and they confirmed there are not any errors on the switch ports where those cards are connected. Is there any other data I can examine related to the OS or hardware that may reveal something? Any other ideas on how to troubleshoot this?

Thanks in advance,
Kevin

[Mel Burslan]

When you said ERP, you did not actually specify which ERP application you are using. If it is SAP (which is the only one I have seen this problem on), it is notorious about not properly cleaning up shared memory segments. And longer the instance runs, worse the situation becomes. And there comes a point, you may need to shut the SAP and database , clean up shared memory segments and bring them back up.

Please see man pages for ipcs and ipcrm. Without knowing the load of your ERP system and how it is architectured, it is very hard to guess the problem or suggest a remedy. Mine is an anecdotal case. Yours might be totally different.

HTH

[Jorge Pons]

Hi
What type of connection you use? ssh, telnet, etc..

Regards, Jorge

[Kevin Liquori_1]

HTH,
The ERP we use is Baan. I looked at shared memory also as tend to have a lot of processes that hang around without correctly terminating. I've steered away from that being the problem since the issues seemed to have gotten worse the week after we rebooted these servers. But this also coincides with increased activity as we moved into a financial close period.

Jorge,
The connections from the desktop client to the application server is rexec. Then the application connects over the network to the Oracle database using it's own driver. It's not clear if the terminations are originating in the application->database connection or the desktop->application server.

Thanks

[Mel Burslan]

Kevin,
I am spit-balling here but what you said about this event taking place increasingly close to the financial year end, makes me think (because year end puts high stress on any financial and related application) that, you might have either a capacity issue, (as in servers are not powerful enough to run the database in the transaction volumes you need) or there might be a tuning issue.

Since you sounded like you know hpux better than the average sysadmin, I am assuming these points are covered. But if you are not sure, it would help to post your db server (and app servers if any) hw specs as well as your kernel tunable values list. Not myself, (as I am not familiar with BaaN) but someone might point out a gaping hole in them. Just a thought.

[Kevin Liquori_1]

Thanks, Mel. I don't think we have a capacity issue, but I could be missing something Below is the info that I think is relevant.
The problem originally started in 2009. In 2010 we increaseed the amount of physical RAM in both servers as it was tight. The terminations seem to have gotten worse in the last two weeks.

Application Server:
Superdome vPar
2 CPUs @ 1 GHz
RAM: 10GB
CPU utilization generally under 30%
memory usage generally under 80%
Load average generally under 0.5
LAN: 100Mb

I'm moving to a 1Gb LAN card this weekend though glance data does not indicate the card is reaching capacity.

Application Server kctune output:
Tunable Value Expression Changes
NSTREVENT 50 Default
NSTRPUSH 16 Default
NSTRSCHED 0 Default
STRCTLSZ 1024 Default
STRMSGSZ 65535 65535
acctresume 4 Default
acctsuspend 2 Default
acdd_adsm_bcode_len 8 Default
acdd_adsmjbcode_len 6 Default
acdd_trc_flags 536870911 Default
aio_listio_max 256 Default Immed
aio_max_ops 2048 Default Immed
aio_monitor_run_sec 30 Default Immed
aio_physmem_pct 10 Default
aio_prio_delta_max 20 Default Immed
aio_proc_thread_pct 70 Default Immed
aio_proc_threads 1024 Default Immed
aio_req_per_thread 1 Default Immed
allocate_fs_swapmap 0 Default
alternate_pathing 1 Default
alwaysdump 0 Default Immed
atdd_trc_flags 536870911 Default
chanq_hash_locks 256 Default
core_addshmem_read 0 Default Immed
core_addshmem_write 0 Default Immed
create_fastlinks 0 Default
dbc_max_pct 6 6 Immed
dbc_min_pct 3 3 Immed
default_disk_ir 0 Default
desfree_pct 0 Default Auto
disksort_seconds 0 Default
dmp_pathswitch_blks_shift 10 Default
dmp_rootdev_is_vol 0 Default
dmp_swapdev_is_vol 0 Default
dnlc_hash_locks 512 Default
dontdump 0 Default Immed
dst 1 Default
dump_compress_on 1 Default Immed
enable_idds 0 Default Immed
eqmemsize 15 Default
executable_stack 0 Default Immed
fr_statemax 800000 Default Immed
fr_tcpidletimeout 86400 Default Immed
fs_async 0 Default
fs_symlinks 20 Default Immed
ftable_hash_locks 64 Default
gvid_no_claim_dev 0 Default Immed
hdlpreg_hash_locks 128 Default
hp_hfs_mtra_enabled 1 Default
intr_strobe_ics_pct 100 Default Auto
io_ports_hash_locks 64 Default
ioforw_timeout 0 Default Auto
is_vxtrace_enabled 1 Default
ksi_alloc_max 102400 102400 Immed
ksi_send_max 32 Default
lotsfree_pct 0 Default Auto
max_acct_file_size 2560000 Default Immed
max_async_ports 50 Default
max_mem_window 0 Default
max_thread_proc 1200 1200 Immed
maxdsiz 1073741824 Default Immed
maxdsiz_64bit 4294967296 Default Immed
maxfiles 2048 Default
maxfiles_lim 4096 Default Immed
maxssiz 134217728 134217728 Immed
maxssiz_64bit 1073741824 1073741824 Immed
maxtsiz 100663296 Default Immed
maxtsiz_64bit 1073741824 Default Immed
maxuprc 4096 4096 Immed
maxvgs 32 32
mig_debug 0 Default Immed
msgmap 8194 8194
msgmax 32767 32767 Immed
msgmnb 98304 98304 Immed
msgmni 6400 6400
msgseg 32767 32767
msgssz 96 Default
msgtql 8192 8192
ncdnode 150 Default Immed
nclist 8292 Default
ncsize 119668 119668
nfile 114548 114548 Imm (auto disabled)
nflocks 20000 20000 Imm (auto disabled)
ninode 114548 114548
nkthread 22416 22416 Immed
nproc 12800 12800 Immed
npty 256 256
nstrpty 256 256
nstrtel 60 Default
nswapdev 10 Default
nswapfs 10 Default
nsysmap 25600 25600
nsysmap64 25600 25600
o_sync_is_o_dsync 0 Default
pagezero_daemon_enabled 1 Default Immed
pfdat_hash_locks 128 Default
physical_io_buffers 2048 Default Auto
pthread_condvar_prio_boost 0 Default Immed
region_hash_locks 128 Default
remote_nfs_swap 0 Default
rng_bitvals 9876543210 Default
rng_sleeptime 2 Default
rtsched_numpri 32 Default
scroll_lines 100 Default Immed
scsi_max_qdepth 8 Default Immed
scsi_maxphys 1048576 Default
secure_sid_scripts 1 Default Immed
semaem 16384 Default
semmni 20000 20000
semmns 30000 30000
semmnu 4092 4092
semmsl 2048 Default Immed
semume 100 Default
semvmx 32767 Default
sendfile_max 0 Default
shmmax 4294967296 4294967296 Immed
shmmni 512 512 Immed
shmseg 300 Default Immed
st_ats_enabled 0 Default
st_fail_overruns 0 Default
st_large_recs 0 Default
st_san_safe 0 Default Immed
streampipes 0 Default
swapmem_on 1 Default
swchunk 2048 Default
sysv_hash_locks 128 Default
tcphashsz 2048 Default
timeslice 10 Default
timezone 420 Default
unlockable_mem 0 Default
vnode_cd_hash_locks 128 Default
vnode_hash_locks 128 Default
vol_checkpt_default 10240 Default
vol_dcm_replay_size 262144 Default
vol_default_iodelay 50 Default
vol_fmr_logsz 4 Default
vol_max_bchain 32 Default
vol_max_nconfigs 20 Default
vol_max_nlogs 20 Default
vol_max_nmpool_sz 4194304 Default Immed
vol_max_prm_dgs 1024 Default
vol_max_rdback_sz 4194304 Default Immed
vol_max_vol 8388608 Default
vol_max_wrspool_sz 4194304 Default Immed
vol_maxio 1024 Default
vol_maxioctl 32768 Default
vol_maxkiocount 2048 Default
vol_maxparallelio 256 Default
vol_maxspecialio 1024 Default
vol_maxstablebufsize 256 Default
vol_min_lowmem_sz 532480 Default Immed
vol_mvr_maxround 256 Default
vol_nm_hb_timeout 10 Default
vol_rootdev_is_vol 0 Default
vol_rvio_maxpool_sz 4194304 Default Immed
vol_subdisk_num 4096 Default
vol_swapdev_is_vol 0 Default
vol_vvr_transport 1 Default
vol_vvr_use_nat 0 Default
volcvm_cluster_size 16 Default
volcvm_smartsync 1 Default
voldrl_max_drtregs 2048 Default
voldrl_min_regionsz 512 Default
voldrl_volumemax_drtregs 256 Default
voliomem_chunk_size 65536 Default
voliomem_maxpool_sz 4194304 Default
voliot_errbuf_dflt 16384 Default
voliot_iobuf_default 8192 Default
voliot_iobuf_limit 131072 Default
voliot_iobuf_max 65536 Default
voliot_max_open 32 Default
volpagemod_max_memsz 6144 Default Immed
volraid_rsrtransmax 1 Default
vps_ceiling 16 Default
vps_chatr_ceiling 1048576 Default
vps_pagesize 4 Default
vx_era_nthreads 5 Default
vx_maxlink 32767 Default
vx_ninode 0 Default Immed
vxfs_bc_bufhwm 0 Default Immed
vxfs_ifree_timelag 0 Default Immed
vxtask_max_monitors 32 Default


Database Server:
Superdome vPar
4 CPUs @ 1 GHz
RAM: 12GB
CPU utilization generally under 70%
memory usage generally under 90%
Load average generally under 1.0
LAN: 1Gb

Database Server kctune output:
Tunable Value Expression Changes
NSTREVENT 50 Default
NSTRPUSH 16 Default
NSTRSCHED 0 Default
STRCTLSZ 1024 Default
STRMSGSZ 65535 65535
acctresume 4 Default
acctsuspend 2 Default
acdd_adsm_bcode_len 8 Default
acdd_adsmjbcode_len 6 Default
acdd_trc_flags 536870911 Default
aio_listio_max 256 Default Immed
aio_max_ops 2048 Default Immed
aio_monitor_run_sec 30 Default Immed
aio_physmem_pct 10 Default
aio_prio_delta_max 20 Default Immed
aio_proc_thread_pct 70 Default Immed
aio_proc_threads 1024 Default Immed
aio_req_per_thread 1 Default Immed
allocate_fs_swapmap 0 Default
alternate_pathing 1 Default
alwaysdump 0 Default Immed
atdd_trc_flags 536870911 Default
chanq_hash_locks 256 Default
core_addshmem_read 0 Default Immed
core_addshmem_write 0 Default Immed
create_fastlinks 0 Default
dbc_max_pct 6 6 Immed
dbc_min_pct 3 3 Immed
default_disk_ir 0 Default
desfree_pct 0 Default Auto
disksort_seconds 0 Default
dmp_pathswitch_blks_shift 10 Default
dmp_rootdev_is_vol 0 Default
dmp_swapdev_is_vol 0 Default
dnlc_hash_locks 512 Default
dontdump 0 Default Immed
dst 1 Default
dump_compress_on 1 Default Immed
enable_idds 0 Default Immed
eqmemsize 15 Default
executable_stack 0 Default Immed
fr_statemax 800000 Default Immed
fr_tcpidletimeout 86400 Default Immed
fs_async 0 Default
fs_symlinks 20 Default Immed
ftable_hash_locks 64 Default
gvid_no_claim_dev 0 Default Immed
hdlpreg_hash_locks 128 Default
hp_hfs_mtra_enabled 1 Default
intr_strobe_ics_pct 100 Default Auto
io_ports_hash_locks 64 Default
ioforw_timeout 0 Default Auto
is_vxtrace_enabled 1 Default
ksi_alloc_max 102400 102400 Immed
ksi_send_max 32 Default
lotsfree_pct 0 Default Auto
max_acct_file_size 2560000 Default Immed
max_async_ports 50 Default
max_mem_window 0 Default
max_thread_proc 1200 1200 Immed
maxdsiz 1073741824 Default Immed
maxdsiz_64bit 4294967296 Default Immed
maxfiles 2048 Default
maxfiles_lim 4096 Default Immed
maxssiz 134217728 134217728 Immed
maxssiz_64bit 1073741824 1073741824 Immed
maxtsiz 100663296 Default Immed
maxtsiz_64bit 1073741824 Default Immed
maxuprc 4096 4096 Immed
maxvgs 32 32
mig_debug 0 Default Immed
msgmap 8194 8194
msgmax 32767 32767 Immed
msgmnb 98304 98304 Immed
msgmni 6400 6400
msgseg 32767 32767
msgssz 96 Default
msgtql 8192 8192
ncdnode 150 Default Immed
nclist 8292 Default
ncsize 8976 Default
nfile 114548 114548 Imm (auto disabled)
nflocks 20000 20000 Imm (auto disabled)
ninode 114548 114548
nkthread 22416 22416 Immed
nproc 12800 12800 Immed
npty 256 256
nstrpty 256 256
nstrtel 60 Default
nswapdev 10 Default
nswapfs 10 Default
nsysmap 25600 25600
nsysmap64 8400 Default
o_sync_is_o_dsync 0 Default
pagezero_daemon_enabled 1 Default Immed
pfdat_hash_locks 128 Default
physical_io_buffers 2048 Default Auto
pthread_condvar_prio_boost 0 Default Immed
region_hash_locks 128 Default
remote_nfs_swap 0 Default
rng_bitvals 9876543210 Default
rng_sleeptime 2 Default
rtsched_numpri 32 Default
scroll_lines 100 Default Immed
scsi_max_qdepth 8 Default Immed
scsi_maxphys 1048576 Default
secure_sid_scripts 1 Default Immed
semaem 16384 Default
semmni 20000 20000
semmns 30000 30000
semmnu 4092 4092
semmsl 2048 Default Immed
semume 100 Default
semvmx 32767 Default
sendfile_max 0 Default
shmmax 4294967296 4294967296 Immed
shmmni 512 512 Immed
shmseg 300 Default Immed
st_ats_enabled 0 Default
st_fail_overruns 0 Default
st_large_recs 0 Default
st_san_safe 0 Default Immed
streampipes 0 Default
swapmem_on 1 Default
swchunk 2048 Default
sysv_hash_locks 128 Default
tcphashsz 2048 Default
timeslice 10 Default
timezone 420 Default
unlockable_mem 0 Default
vnode_cd_hash_locks 128 Default
vnode_hash_locks 128 Default
vol_checkpt_default 10240 Default
vol_dcm_replay_size 262144 Default
vol_default_iodelay 50 Default
vol_fmr_logsz 4 Default
vol_max_bchain 32 Default
vol_max_nconfigs 20 Default
vol_max_nlogs 20 Default
vol_max_nmpool_sz 4194304 Default Immed
vol_max_prm_dgs 1024 Default
vol_max_rdback_sz 4194304 Default Immed
vol_max_vol 8388608 Default
vol_max_wrspool_sz 4194304 Default Immed
vol_maxio 1024 Default
vol_maxioctl 32768 Default
vol_maxkiocount 2048 Default
vol_maxparallelio 256 Default
vol_maxspecialio 1024 Default
vol_maxstablebufsize 256 Default
vol_min_lowmem_sz 532480 Default Immed
vol_mvr_maxround 256 Default
vol_nm_hb_timeout 10 Default
vol_rootdev_is_vol 0 Default
vol_rvio_maxpool_sz 4194304 Default Immed
vol_subdisk_num 4096 Default
vol_swapdev_is_vol 0 Default
vol_vvr_transport 1 Default
vol_vvr_use_nat 0 Default
volcvm_cluster_size 16 Default
volcvm_smartsync 1 Default
voldrl_max_drtregs 2048 Default
voldrl_min_regionsz 512 Default
voldrl_volumemax_drtregs 256 Default
voliomem_chunk_size 65536 Default
voliomem_maxpool_sz 4194304 Default
voliot_errbuf_dflt 16384 Default
voliot_iobuf_default 8192 Default
voliot_iobuf_limit 131072 Default
voliot_iobuf_max 65536 Default
voliot_max_open 32 Default
volpagemod_max_memsz 6144 Default Immed
volraid_rsrtransmax 1 Default
vps_ceiling 16 Default
vps_chatr_ceiling 1048576 Default
vps_pagesize 4 Default
vx_era_nthreads 5 Default
vx_maxlink 32767 Default
vx_ninode 0 Default Immed
vxfs_bc_bufhwm 0 Default Immed
vxfs_ifree_timelag 0 Default Immed
vxtask_max_monitors 32 Default

[Kevin Liquori_1]

This weekend I switched the physical and virtual IP to a 1 Gb LAN card. Previously it was using a 100 Mb card. So It's a new card, new cable, and new port on the switch. Users are still experiencing disconnects, but again I don't see errors on the card in lanadmin or syslog. We are configuring traces on the end users' clients so hopefully that will reveal something related to the application.

Kevin