No buffer space

[Jannik]

I have looked at the kernel, gpm and swap, and i can't see any shortage PLEASE HELP:

Error:
Fri Aug 13 15:26:34 2004
Can't open /dev/ixe ppa_num = 2
No buffer space available.

Info:
swapinfo -ta
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4194304 106212 4088092 3% 0 - 1 /dev/vg00/lvol2
reserve - 1133752 -1133752
memory 4703044 4420696 282348 94%
total 8897348 5660660 3236688 64% - 0 -

kmtune
Parameter Current Dyn Planned Module Version
===============================================================================
NSTRBLKSCHED - - 2
NSTREVENT 50 - 50
NSTRPUSH 16 - 16
NSTRSCHED 0 - 0
STRCTLSZ 1024 - 1024
STRMSGSZ 65535 - 65535
acctresume 4 - 4
acctsuspend 2 - 2
aio_listio_max 256 - 256
aio_max_ops 2048 - 2048
aio_physmem_pct 10 - 10
aio_prio_delta_max 20 - 20
allocate_fs_swapmap 0 - 0
alwaysdump 1 - 1
bcvmap_size_factor 2 - 2
bootspinlocks - - 256
bufcache_hash_locks 128 - 128
bufpages 0 - (NBUF*2)
chanq_hash_locks 256 - 256
core_addshmem_read 0 Y 0
core_addshmem_write 0 Y 0
create_fastlinks 1 - 1
dbc_max_pct 50 - 50
dbc_min_pct 5 - 5
default_disk_ir 0 - 0
desfree - - 0
disksort_seconds 0 - 0
dmp_rootdev_is_vol 0 - 0
dmp_swapdev_is_vol 0 - 0
dnlc_hash_locks 512 - 512
dontdump 0 - 0
dskless_node - - 0
dst 1 - 1
effective_maxpid - - ((NPROC<=30000)?30000:(NPROC*5/4))
eisa_io_estimate - - 0x300
enable_idds 0 - 0
eqmemsize 15 - 15
executable_stack 1 - 1
fcp_large_config 0 - 0
file_pad - - 10
fs_async 0 - 0
ftable_hash_locks 64 - 64
hdlpreg_hash_locks 128 - 128
hfs_max_ra_blocks 8 - 8
hfs_max_revra_blocks 8 - 8
hfs_ra_per_disk 64 - 64
hfs_revra_per_disk 64 - 64
hp_hfs_mtra_enabled 1 - 1
hpux_aes_override - - 0
initmodmax 50 - 50
io_ports_hash_locks 64 - 64
iomemsize - - 40000
ksi_alloc_max 33312 - (NPROC*8)
ksi_send_max 32 - 32
lotsfree - - 0
max_async_ports 50 - 50
max_fcp_reqs 512 - 512
max_mem_window 0 - 0
max_thread_proc 256 - 256
maxdsiz 0x40000000 - 0X40000000
maxdsiz_64bit 0x80000000 - 0X80000000
maxfiles 600 - 600
maxfiles_lim 1024 Y 1024
maxqueuetime - - 0
maxssiz 0x8000000 - 0X8000000
maxssiz_64bit 0x40000000 - 0X40000000
maxswapchunks 16384 - 16384
maxtsiz 0x4000000 Y 0X4000000
maxtsiz_64bit 0x40000000 Y 0X40000000
maxuprc 3747 Y ((NPROC*9)/10)
maxusers 512 - 512
maxvgs 10 - 10
mesg 1 - 1
minfree - - 0
modstrmax 500 - 500
msgmap 4166 - (2+MSGTQL)
msgmax 8192 Y 8192
msgmnb 16384 Y 16384
msgmni 4164 - (NPROC)
msgseg 32767 - 32767
msgssz 8 - 8
msgtql 4164 - (NPROC)
nbuf 0 - 0
ncallout 7367 - (64+NKTHREAD)
ncdnode 150 - 150
nclist 8292 - (100+16*MAXUSERS)
ncsize 36384 - ((8*NPROC+2048)+VX_NCSIZE)
ndilbuffers 30 - 30
netisr_priority - - -1
netmemmax - - 0
nfile 64508 - (15*NPROC+2048)
nflocks 4164 - (NPROC)
nhtbl_scale 0 - 0
ninode 35360 - ((8*NPROC)+2048)
nkthread 7303 - (((NPROC*7)/4)+16)
nni - - 2
no_lvm_disks 0 - 0
nproc 4164 - (68+8*MAXUSERS)
npty 60 - 60
nstrpty 60 - 60
nstrtel 60 - 60
nswapdev 10 - 10
nswapfs 10 - 10
nsysmap 8328 - ((NPROC)>800?2*(NPROC):800)
nsysmap64 8328 - ((NPROC)>800?2*(NPROC):800)
o_sync_is_o_dsync 0 - 0
page_text_to_local - - 0
pfdat_hash_locks 128 - 128
public_shlibs 1 - 1
region_hash_locks 128 - 128
remote_nfs_swap 0 - 0
rtsched_numpri 32 - 32
scroll_lines 100 - 100
scsi_max_qdepth 8 Y 8
scsi_maxphys 1048576 - 1048576
sema 1 - 1
semaem 16384 - 16384
semmap 8930 - (SEMMNI+2)
semmni 8928 - ((NPROC*2)+600)
semmns 17856 - (SEMMNI*2)
semmnu 4160 - (NPROC-4)
semmsl 2048 Y 2048
semume 50 - 50
semvmx 32768 - 32768
sendfile_max 0 - 0
shmem 1 - 1
shmmax 0x40000000 Y 0X40000000
shmmni 912 - 912
shmseg 120 Y 120
st_ats_enabled 0 - 0
st_fail_overruns 0 - 0
st_large_recs 0 - 0
st_san_safe 0 - 0
streampipes 1 - 1
swapmem_on 1 - 1
swchunk 2048 - 2048
sysv_hash_locks 128 - 128
tcphashsz 0 - 0
timeslice 10 - (100/10)
timezone 420 - 420
unlockable_mem 0 - 0
vas_hash_locks 128 - 128
vnode_cd_hash_locks 128 - 128
vnode_hash_locks 128 - 128
vol_checkpt_default 10240 - 10240
vol_dcm_replay_size 262144 - (256*1024)
vol_default_iodelay 50 - 50
vol_fmr_logsz 4 - 4
vol_max_bchain 32 - 32
vol_max_nconfigs 20 - 20
vol_max_nlogs 20 - 20
vol_max_nmpool_sz 4194304 - (4*1024*1024)
vol_max_prm_dgs 1024 - 1024
vol_max_rdback_sz 4194304 - (4*1024*1024)
vol_max_vol 8388608 - (8*1024*1024)
vol_maxio 256 - 256
vol_maxioctl 32768 - 32768
vol_maxkiocount 2048 - 2048
vol_maxparallelio 256 - 256
vol_maxspecialio 256 - 256
vol_maxstablebufsize 256 - 256
vol_min_lowmem_sz 524288 - (512*1024)
vol_mvr_maxround 256 - 256
vol_nm_hb_timeout 10 - (10)
vol_subdisk_num 4096 - 4096
vol_vvr_transport 1 - (1)
vol_vvr_use_nat 0 - (0)
volcvm_cluster_size 16 - 16
volcvm_smartsync 1 - 1
voldrl_max_drtregs 2048 - 2048
voldrl_min_regionsz 512 - 512
voliomem_chunk_size 65536 - (64*1024)
voliomem_maxpool_sz 4194304 - (4*1024*1024)
voliot_errbuf_dflt 16384 - 16384
voliot_iobuf_default 8192 - 8192
voliot_iobuf_limit 131072 - 131072
voliot_iobuf_max 65536 - 65536
voliot_max_open 32 - 32
volraid_rsrtransmax 1 - 1
vps_ceiling 64 - 64
vps_chatr_ceiling 1048576 - 1048576
vps_pagesize 4 - 4
vx_fancyra_enable 0 - 0
vx_maxlink 32767 - 32767
vx_ncsize 1024 - 1024
vx_ninode 0 - 0
vxfs_max_ra_kbytes 1024 - 1024
vxfs_ra_per_disk 1024 - 1024
vxtask_max_monitors 32 - 32
x25_setup_pvc_timer - - 0

[RAC_1 · Accepted solution]

You are using all of your swap space. Check if you are swapping out or not.

vmstat 5 5
Also you are using dynamic buffer cahe(settings dbc_max_pct and dbc_min_pct) and the values are at default. dbc_max_pct is at 50 % and min at 5 %.

what this means is you are using half of your RAM as buffer cache. Are you effectively sing it??? sar -b 5 5

If write cache rate is not constantly more than 80 %, you can certainly lower it.

Tuning these parameters will require a reboot. If you can not reboot it right away, try adding additional swap area. Turn it on, may be the error will go away.
lvcreate -L size_in_mb -C y /dev/vgxx/lvolx
entry in /etc/fstab
/dev/vgxx/lvolx ... swap pri=1 0 0

swapon -a

Anil
Anil

[Jeff Schussele]

Hi Jannik,

That appears to me to be a TCP connection resource problem.
Might want to look at the ndd value tcp_conn_request_max
You can query TCP values with
ndd -get /dev/tcp tcp_status
or check netstat outputs as well.

HTH,
Jeff

[piotr wienckiewicz]

hey
my opinion you must increase parametr dbc_max_pct

[Jannik]

vmstat 5 5
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
1 1 0 129360 61249 116 29 1 0 0 0 1 2166 8082 606 30 2 67
2 0 0 126160 64005 38 38 0 0 0 0 0 2088 2380 570 13 1 85
2 0 0 126160 60616 135 42 1 0 0 0 0 2087 6161 1044 28 4 69
1 0 0 125698 61709 226 72 1 0 0 0 0 2279 9875 920 19 3 78
1 0 0 125698 64512 87 41 0 0 0 0 0 2181 4394 674 14 1 85

#sar -b 5 5

HP-UX tm025 B.11.11 U 9000/800 08/13/04

16:06:15 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
16:06:20 0 2439 100 162 602 73 0 0
16:06:25 0 28043 100 186 943 80 0 0
16:06:30 0 52872 100 92 483 81 0 0
16:06:35 0 47530 100 90 408 78 0 0
16:06:40 0 622 100 94 349 73 0 0

Average 0 26323 100 125 557 78 0 0

[Jannik]

I don't think it's the lan:
Class I H/W Path Driver S/W State H/W Type Description
==========================================================================
lan 0 0/0/8/0/0/4/0 igelan CLAIMED INTERFACE HP A6794-60001 PCI 1000Base-T
lan 1 1/0/1/0/0/4/0 igelan CLAIMED INTERFACE HP A6794-60001 PCI 1000Base-T

[RAC_1]

You are paging out. And buffer cahe is also being used heavily.

How much RAM you have on this machine??? I would decrease buffer cahe to 10-15 %(dbc_max_pct) This would require a reboot. If that can not be done right away, other option that I see is adding secondary swap as mentioned in previoud reply. But in this case this would work/may not work

It is better to lower the buffer cache and check.

Anil

[Jeff Schussele]

Hi (again) Jannik,

Well, it may not be TCP - but I believe it is comm related. Do you have X25 on this system?
/dev/ixe is an X25 comm device.
See the following thread:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=150275

It could be with the system paging out X25 cannot get the resources it needs to communicate.

Also here's the latest X25 patch - PHNE_30745

http://www1.itrc.hp.com/service/cki/patchDocDisplay.do?patchId=PHNE_30745

It may help, but it's more likely that you need to free up some memory.

HTH,
Jeff

[Jannik]

Jeff I think you may be right. What is the newest X25 patch for 11.11?
I have 6GB of memory on the system.
I have add extra filesystem swap.

I will submit point when the issue is solved... :-) thanks for the help!

[Jeff Schussele]

Jannik,

It would be PHNE_29089 - avail here:

http://www1.itrc.hp.com/service/patch/patchDetail.do?BC=patch.breadcrumb.main|patch.breadcrumb.search|&patchid=PHNE_29089&context=hpux:800:11:11

Rgds,
Jeff

[Jannik]

That was what i feared :-)

swlist -l patch | grep PHNE_29089
# PHNE_29089 1.0 J2793B X.25 SX25-HPerf/SYNC-WAN
# PHNE_29089.COM-64SLIB 1.0 SX25-HPerf.COM-64SLIB
# PHNE_29089.COM-ALIB 1.0 SX25-HPerf.COM-ALIB
# PHNE_29089.PA-ALIB 1.0 SX25-HPerf.PA-ALIB
# PHNE_29089.SX25-HPERF-COM 1.0 SX25-HPerf.SX25-HPERF-COM
# PHNE_29089.SX25-HPERF-IP 1.0 SX25-HPerf.SX25-HPERF-IP
# PHNE_29089.SX25-HPERF-MAN 1.0 SX25-HPerf.SX25-HPERF-MAN
# PHNE_29089.SX25-HPERF-PAD 1.0 SX25-HPerf.SX25-HPERF-PAD
# PHNE_29089.SX25-HPERF-SAM 1.0 SX25-HPerf.SX25-HPERF-SAM
# PHNE_29089.SX25-SNMP 1.0 SX25-HPerf.SX25-SNMP
# PHNE_29089.SYNC-ALIB 1.0 SYNC-WAN.SYNC-ALIB
# PHNE_29089.SYNC-COM 1.0 SYNC-WAN.SYNC-COM

I think i will by more memory.