SWAP error - Plz help

[Yogeeraj_1]

hi,

Also, can you check if these processes are still there?

ps -ef|grep sshd

ps -ef|grep telnetd


Also, repost the output from:

kmtune


revert!
kind regards
yogeeraj

[Gops_1]

I can't change that maxswapchunks and swchunk to 16384. Because am not getting any difference when i changed that maxswapchunks to 2048. I really doubt whether i can set this parameter to this much high. I am using 11.0 and my total hardisk is 30GB.

hai yogee
sshd is there as i come into that machine by ssh.

kmtune output as follows.
Parameter Value
===============================================================================
NSTRBLKSCHED 2
NSTREVENT 50
NSTRPUSH 16
NSTRSCHED 0
STRCTLSZ 1024
STRMSGSZ 65535
acctresume 4
acctsuspend 2
aio_listio_max 256
aio_max_ops 2048
aio_physmem_pct 10
aio_prio_delta_max 20
allocate_fs_swapmap 0
alwaysdump 0
bootspinlocks 256
bufcache_hash_locks 128
bufpages (NBUF*2)
chanq_hash_locks 256
create_fastlinks 0
dbc_max_pct 50
dbc_min_pct 5
default_disk_ir 1
desfree 0
disksort_seconds 0
dnlc_hash_locks 64
dontdump 0
dskless_node 0
dst 1
eisa_io_estimate 0x300
eqmemsize 15
fcp_large_config 0
file_pad 10
fs_async 0
ftable_hash_locks 64
hdlpreg_hash_locks 128
hfs_max_ra_blocks 8
hfs_ra_per_disk 64
hpux_aes_override 0
initmodmax 50
io_ports_hash_locks 64
iomemsize 40000
km_disable 0
ksi_alloc_max 11680
ksi_send_max 32
lotsfree 0
max_async_ports 50
max_fcp_reqs 512
max_mem_window 0
max_thread_proc 256
maxdsiz 0x04000000
maxdsiz_64bit 0x0000000040000000
maxfiles 120
maxfiles_lim 1024
maxqueuetime 0
maxssiz 0x00800000
maxssiz_64bit 0x00800000
maxswapchunks 2048
maxtsiz 0x04000000
maxtsiz_64bit 0x0000000040000000
maxuprc 75
maxusers 180
maxvgs 10
mesg 1
minfree 0
modstrmax 500
msgmap (2+MSGTQL)
msgmax 8192
msgmnb 16384
msgmni 50
msgseg 2048
msgssz 8
msgtql 40
nbuf 0
ncallout 2064
ncdnode 150
nclist 2980
ncsize 2832
ndilbuffers 30
netisr_priority -1
netmemmax 0
nfile 21101
nflocks 200
nhtbl_scale 0
ninode 1808
nkthread 2571
nni 2
no_lvm_disks 0
nproc 1460
npty 60
nstrpty 60
nstrtel 60
nswapdev 10
nswapfs 10
nsysmap ((NPROC)>800?2*(NPROC):800)
nsysmap64 ((NPROC)>800?2*(NPROC):800)
num_tachyon_adapters 0
o_sync_is_o_dsync 0
page_text_to_local 0
pfdat_hash_locks 128
public_shlibs 1
region_hash_locks 128
remote_nfs_swap 0
rtsched_numpri 32
scroll_lines 100
scsi_max_qdepth 8
scsi_maxphys 1048576
sema 1
semaem 16384
semmap (SEMMNI+2)
semmni 64
semmns 128
semmnu 30
semmsl_override 2048
semume 10
semvmx 32767
sendfile_max 0
shmem 1
shmmax 0X4000000
shmmni 200
shmseg 120
st_ats_enabled 1
st_fail_overruns 0
st_large_recs 0
st_san_safe 0
streampipes 0
swapmem_on 1
swchunk 2048
sysv_hash_locks 128
tcphashsz 0
timeslice (100/10)
timezone 420
unlockable_mem 0
vnode_cd_hash_locks 128
vnode_hash_locks 128
vps_ceiling 16
vps_chatr_ceiling 65536
vps_pagesize 4
vx_maxlink 32767
vx_ncsize 1024
vx_ninode 0
vx_noifree 0
vx_nothrottle 1
vxfs_max_ra_kbytes 1024
vxfs_ra_per_disk 1024

[Don Morris_1]

Ok, let's take a step back.

You raised maxswapchunks (and swapmem_on, I take it). This is good -- but we need to be sure they took effect.

On 11.0, SAM is the easiest way to change these things, but you earlier said that it wasn't working. Did that problem get resolved, or did you use kmtune (or just edit the system file) to change maxswapchunks?

The important part here is that 11.0 and 11.11 for non-dynamic tunables require a kernel rebuild to have tunable changes take effect. SAM will do all the steps for you, if you aren't using SAM we have to be sure prepare the new kernel and reboot to it to have an effect.

(My suspicion is that this hasn't happened since virtual memory exhaustion [by swap exhaustion] causes fork failures, which fits your ssh and other new process difficulties).

Your sequence needs to be something like:
#cd /stand
#/usr/lbin/sysadm/system_prep -s system
#kmtune -s maxswapchunks=2048 -S /stand/system
#cp system build/system
#mk_kernel -o /stand/vmunix
#kmupdate
#shutdown -r now

Obviously you'll need to get the load down and the swap down to be able to run these commands at all.

One obvious check -- if you boot the kernel and you still get the warning that you need to increase maxswapchunks, your kernel doesn't have the updated value. You want that message gone and to have swapinfo show the full 4096 post-reboot.

[Yogeeraj_1]

hi,

Yes. Did you follow the steps as mentioned by Don above?
(i.e. manually change these parameters)

or did you make the changes using SAM?
(this will automatically rebuild the kernel)


revert!

kind regards
yogeeraj

[Gops_1]

Hi Morris,
That was good command for me. I did the same and rebooted the machine. Now am not getting that error.

ba
8 memory
160 processor
btlan3: Initializing 10/100BASE-TX card at 0/0/0/0....

System Console is on the Built-In Serial Interface
btlan5: Initializing 10/100BASE-TX card at 0/2/0/0....
Logical volume 64, 0x3 configured as ROOT
Logical volume 64, 0x2 configured as SWAP
Logical volume 64, 0x2 configured as DUMP
Swap device table: (start & size given in 512-byte blocks)
entry 0 - major is 64, minor is 0x2; start = 0, size = 8388608
Dump device table: (start & size given in 1-Kbyte blocks)
entry 0 - major is 31, minor is 0x1f000; start = 519008, size = 4194304
Starting the STREAMS daemons-phase 1
Create STCP device files
B2352B/9245XB HP-UX (B.11.00) #1: Wed Nov 5 22:38:19 PST 1997

Memory Information:
physical page size = 4096 bytes, logical page size = 4096 bytes
Physical: 1048576 Kbytes, lockable: 827332 Kbytes, available: 861716 Kbytes


But still my ping utility doesn't work. It still hangs. Also all my NNM services are still in unstarable mode. I dont know what happened.

What else i need to do from my end.

You are really amazing...

Thanks
gV

[Gops_1]

Whenever i ping am getting this output

ping: cannot fork: too many processes

What should i do ?

Thanks
gV

[Don Morris_1]

fork(2) is giving EAGAIN instead of ENOMEM then. (Good sign - that's hitting nproc or maxuprc instead of swap limits).

You'll have to either wait for processes to go away or try hard to kill them. Then you need to determine what spawned all these processes such that you hit the limits (nproc is total processes on the system, maxuprc is max per user). See:

http://docs.hp.com/en/TKP-90202/re60.html

and

http://docs.hp.com/en/TKP-90202/re40.html

for details. Once you understand what's hitting the limits, you'll have to decide if the limits need to be higher or the application needs to be fixed to not hit them (this may be an application which is doing something akin to the classic "fork bomb" -- hence it consumes lots of swap and lots of process table entries -- stopping it from doing that is the right solution. Getting the swap configured properly moved the failure point from the application getting failures on new reservations to running out the process table).

[YAQUB_1]

U can try to change maxfiles, maxfiles_lim, maxusers, nfiles & nproc

maxfiles---120(existing)---1024(new)
maxfiles_lim---1024(existing)---2048(new)
maxusers---180(existing)---512(new)
nfile---21101(existing)---65536(new)
nproc---1460(existing)---4096(new)

[YAQUB_1]

Hi,

Not new "Proposed Value". BTW, if U wants then U changes this value. After change this value U mast be reboot Ur system.

[Venkatesh BL]

Yes, your swap is fine now. The problem now is other kernel parameters. I think you may have to specifically increase the following:
nproc: Total no. of processes on the system.
maxuprc: Max no. of user processes.

[Gops_1]

Hi youb, yogeeraj and don,
Ok i go ahead and change the kernel parameters as yoyub suggested me. So i want to change the kernel parmeter which are all the steps i should follow,

Is it ok if i just use the command


#kmtune -s

and a restart or

#cd /stand
#/usr/lbin/sysadm/system_prep -s system
#kmtune -s -S /stand/system
#cp system build/system
#mk_kernel -o /stand/vmunix
#kmupdate
#shutdown -r now


Do I need to do all the steps suggested by Don Or just kmtune -s kl_parameter=Desired value

and a restart...

Thanks
gV

[Gops_1]

Hi all thank for your great support but stil i need ur help,

I have changed the kernel parameter in SAM as you instructed,

But still am getting the same ping problem,

(/opt/home/x57750)$ ping
ping: cannot fork: too many processes
ping[2]: cannot fork: too many processes

FYI my kernel parameter is as follows,

maxfiles=1024
maxfiles_lim=2048
maxusers=512
nfile=65536
nproc=4096
ncallout=4096 (ncallout >= nproc)
swapmem_on=1

What else we are missing ?

Thanks
gV

[Venkatesh BL]

Did you restart the system?

[Gops_1]

Hi,

Yes we have restarted the system, after we changed the parameters.

Thanks
gV

[TTr]

Please type the following commands and post the output

1. which ping

2. ll /usr/sbin/ping

3. ps -ef

[Gops_1]

(/opt/home/x57750)$ which ping
/usr/sbin/ping
(/opt/home/x57750)$ ll /usr/sbin/ping
-r-sr-xr-x 1 root bin 52 Apr 15 18:19 /usr/sbin/ping
(/opt/home/x57750)$ ps -ef
UID PID PPID C STIME TTY TIME COMMAND
root 0 0 0 15:52:37 ? 0:05 swapper
root 1 0 0 15:52:37 ? 0:01 init
root 2 0 0 15:52:37 ? 0:02 vhand
root 3 0 0 15:52:37 ? 0:18 statdaemon
root 4 0 0 15:52:37 ? 0:00 unhashdaemon
root 8 0 0 15:52:37 ? 0:00 supsched
root 9 0 0 15:52:37 ? 0:00 strmem
root 10 0 0 15:52:37 ? 0:00 strweld
root 11 0 0 15:52:37 ? 0:00 strfreebd
root 12 0 0 15:52:37 ? 0:00 ttisr
root 18 0 0 15:52:37 ? 0:00 lvmkd
root 19 0 0 15:52:37 ? 0:00 lvmkd
root 20 0 0 15:52:37 ? 0:00 lvmkd
root 21 0 0 15:52:37 ? 0:00 lvmkd
root 22 0 0 15:52:37 ? 0:01 lvmkd
root 23 0 0 15:52:37 ? 0:03 lvmkd
root 24 0 0 15:52:37 ? 0:00 lvmschedd
root 1468 1 0 15:54:58 console 0:00 /usr/sbin/getty console console
root 783 1 0 15:54:25 ? 0:00 /usr/sbin/biod 4
root 350 1 0 15:52:56 ? 0:18 /usr/sbin/syncer
root 29 0 0 15:52:37 ? 0:18 vxfsd
root 1240 1 0 15:54:39 ? 0:00 /usr/sbin/envd
root 439 1 0 15:53:15 ? 0:00 /usr/sbin/syslogd -D
root 442 1 0 15:53:15 ? 0:00 /usr/sbin/ptydaemon
root 737 1 0 15:54:24 ? 0:00 /opt/ssh/sbin/sshd
root 455 1 0 15:53:16 ? 0:00 /usr/lbin/nktl_daemon 0 0 0 0 0 1 -2
root 757 1 0 15:54:24 ? 0:00 /usr/sbin/rpcbind
root 465 1 0 15:53:19 ? 0:00 /usr/lbin/ntl_reader 0 1 1 1 100 0 /var/adm/nettl /var/adm/co
root 466 465 0 15:53:19 ? 0:02 /usr/sbin/netfmt -C -F -f /var/a dm/nettl.LOG00 -c /var/adm/c
root 762 0 0 15:54:24 ? 0:00 nfskd
root 830 1 0 15:54:26 ? 0:00 /usr/sbin/inetd
root 782 1 0 15:54:25 ? 0:00 /usr/sbin/biod 4
root 781 1 0 15:54:25 ? 0:00 /usr/sbin/biod 4
root 784 1 0 15:54:25 ? 0:00 /usr/sbin/biod 4
root 1094 1 0 15:54:28 ? 0:01 /usr/sbin/snmpdm
root 790 1 0 15:54:26 ? 0:00 /usr/sbin/rpc.statd
root 796 1 0 15:54:26 ? 0:00 /usr/sbin/rpc.lockd
root 1105 1 0 15:54:33 ? 0:00 /usr/sbin/hp_unixagt
root 1113 1 0 15:54:33 ? 0:16 /usr/sbin/mib2agt
root 1121 1 0 15:54:33 ? 0:00 /usr/sbin/trapdestagt
root 1170 1 0 15:54:38 ? 0:00 /usr/bin/X11/xfs -config /etc/X1 1/fs/config -port 7000 -daem
root 1155 1 0 15:54:34 ? 0:00 /opt/dce/sbin/rpcd
root 1179 1 0 15:54:38 ? 0:01 /usr/sbin/xntpd
root 1189 1 0 15:54:38 ? 0:25 /usr/sbin/pwgrd
root 1676 1233 0 15:55:07 ? 0:06 cclogd
root 1675 1325 0 15:55:07 ? 0:01 /opt/perf/bin/agdbserver -t alar mgen /var/opt/perf/datafiles
root 19611 737 0 10:25:03 ? 0:00 sshd: x57750 [priv]
root 1540 1466 0 15:55:00 ? 0:00 /usr/dt/bin/dtlogin
root 1202 1 0 15:54:39 ? 0:01 /usr/sbin/cron
root 1477 1 0 15:54:58 ? 0:01 /etc/opt/resmon/lbin/p_client
root 1245 1 0 15:54:39 ? 0:00 /usr/bin/X11/SLSd_daemon
root 1233 1 0 15:54:39 ? 0:07 /usr/sbin/stm/uut/bin/sys/diagmo nd
root 1237 1 0 15:54:39 ? 0:00 /opt/pred/bin/psmond
root 1377 1 0 15:54:49 ? 0:00 ./windu_registryd50 -d /var//opt /omni/windu/ -vers 2 -prog 8
root 1379 1 0 15:54:49 ? 0:00 /usr/sbin/swagentd -r
root 1677 1233 0 15:55:07 ? 0:00 diaglogd
root 1466 1 0 15:54:58 ? 0:00 /sbin/sh /usr/dt/bin/dtrc
root 1285 1 0 15:54:40 ? 0:00 /opt/perf/bin/ttd
root 1347 1 0 15:54:48 ? 0:36 /opt/perf/bin/midaemon
root 1441 1439 0 15:54:50 ? 0:02 opcacta
root 1443 1439 0 15:54:50 ? 0:05 opcmona
root 1678 1233 0 15:55:07 ? 0:00 memlogd
root 1325 1 0 15:54:44 ? 0:00 /opt/perf/bin/perflbd
root 1390 1 0 15:54:49 ? 0:00 /etc/opt/resmon/lbin/emsagent
root 1442 1439 0 15:54:50 ? 0:04 opcle
root 1665 1325 0 15:55:05 ? 0:04 /opt/perf/bin/rep_server -t SCOP E /var/opt/perf/datafiles/lo
root 1439 1 0 15:54:50 ? 0:01 opcctla -start
root 1440 1439 0 15:54:50 ? 0:02 opcmsga
root 1444 1439 0 15:54:50 ? 0:00 opcmsgi
root 1684 1675 0 15:55:07 ? 0:04 /opt/perf/bin/alarmgen -svr 1675 -t alarmgen /var/opt/perf/d
root 1679 1233 0 15:55:07 ? 0:14 psmctd
root 1686 830 0 15:55:08 ? 0:00 /etc/opt/resmon/lbin/registrar
root 1449 1 0 15:54:51 ? 0:23 /opt/perf/bin/scopeux
root 1804 1 0 15:55:41 ? 0:03 /usr/sbin/stm/uut/bin/tools/moni tor/dm_core_hw
root 1732 1 0 15:55:41 ? 0:00 /usr/sbin/stm/uut/bin/tools/moni tor/disk_em
root 1951 1 0 15:56:00 ? 0:03 /usr/sbin/stm/uut/bin/tools/moni tor/lpmc_em
root 16360 16357 0 10:00:00 ? 0:00 /usr/lbin/sa/sadc 600 6 /var/adm /sa/sa29
root 1861 1 0 15:55:43 ? 0:00 /usr/sbin/stm/uut/bin/tools/moni tor/dm_memory
root 16357 1202 0 10:00:00 ? 0:00 sh -c /usr/lib/sa/sa1 600 6
root 2059 1 0 15:56:01 ? 0:00 /usr/sbin/stm/uut/bin/tools/moni tor/sysstat_em
x57750 19631 19611 0 10:25:09 ? 0:00 sshd: x57750@pts/0
x57750 19633 19631 0 10:25:09 pts/0 0:00 -sh
x57750 19653 19633 1 10:25:36 pts/0 0:00 ps -ef
(/opt/home/x57750)$

Thanks
gV

[TTr]

It is still not clear what is preventing you from starting new processes.

There are some processes and services that are running that you probably do not need but they are not many.

Please post the file /var/adm/syslog/syslog.log

[Gops_1]

My ping tool which is in /usr/sbin/ping was corrupted. So the problem. I just copied the ping from other HP UX server. Now everything is fine and working good.

Hope i wasted all your times. But i really a ppreciate this great support from all of you. Long live HP Forums...

Thanks
gV