topic Re: fork: Not enough space in Operating System - HP-UX

fork: Not enough space

Paul Sperry — Tue, 25 May 2004 11:30:27 GMT

I have a C3000 running hp-ux 11.11. This morning anything I typed (except pwd) All I would get is fork: Not enough space. I had to power cycle the system in order to get it back up and running. One hour later I am back to where I started, nothing works. There is nothing in the syslogs or dmesg which I check after the power cycle.
The fork: Not enough space to me seems to be a swapspace issue but Iam not sure what's sucking it all up. I have 1024MB of memory and 2048 MB of Swap. It is running oracle but has been for years. The /var: I/O error to me seems to be a disk error but I had just replaced the disk about a month ago. Any ideas? I hate to keep power cycling it.

# pwd
/
# ls
bash: fork: Not enough space
# bdf
bash: fork: Not enough space
# shutdown -r now
bash: fork: Not enough space
# uname -a
bash: fork: Not enough space
# cd /var/adm/syslog
bash: cd: /var/adm/syslog: I/O error
# cd /var
bash: cd: /var: I/O error
# more /etc/fstab
bash: fork: Not enough space

TIA

Re: fork: Not enough space

Steven E. Protter — Tue, 25 May 2004 11:56:17 GMT

Things to look at:

Is the /var filesystem even mounted

Space on /var and / /tmp

/etc/rc.log

I'd also be wondering if Your kernel is currupt.

Obviously the box boots, but its sure acting like there aren't enough processes or room in the process table.

SEP

Re: fork: Not enough space

Paul Sperry — Tue, 25 May 2004 12:02:34 GMT

SEP

Thanks for the input

plenty of space on /var and /tmp.

# bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 147456 61898 80241 44% /
/dev/vg00/lvol1 111637 31414 69059 31% /stand
/dev/vg00/lvol8 2564096 394434 2034072 16% /var
/dev/vg00/lvol7 5120000 1460654 3430647 30% /usr
/dev/vg01/datavol 26861568 19564436 7183772 73% /u02
/dev/vg00/orabinvol
15360000 5748694 9010658 39% /u01
/dev/vg00/lvol4 106496 61384 42322 59% /tmp
/dev/vg00/lvol6 1024000 511895 480135 52% /opt
/dev/vg00/ifasvol 6144000 2746221 3185470 46% /ifas
/dev/vg00/lvol5 24576 13154 10726 55% /home

When the system is running /var is mounted.
But when it's in it's "messed up" stage
commands like bdf, df, mount just return the fork error. So I don't know if it's mounted or not

Re: fork: Not enough space

Bill Hassell — Tue, 25 May 2004 12:02:53 GMT

Not enough space in this context means swapspace. You must have either a runaway process or your DBA decided to change the size of SGA (shared memory) without telling you. Boot up your system, then put these commands into a script:

UNIX95=1 ps -eo vsz,ruser,pid,args | sort -rn | head -20
ipcs -bmop|sort -rnk8
/usr/sbin/swapinfo -tam
echo

Now run this script from cron every 5 minutes:

5,10,15,20,25,30,35,40,45,50,55 * * * * /root/bin/myscript >> /var/tmp/monitormem

The ps listing will show the biggest memory hogs (sorted by size), ipcs will show segments in use sorted by size and swapinfo will show actual memory and swap usage.

Re: fork: Not enough space

Rita C Workman — Tue, 25 May 2004 12:04:04 GMT

fork and swap are basically saying there are no resources left to run anything.
You have swap allocated, but you could try to add a little more. It is possible that something has happened to your disk, your kernel ....or..... somebody has created a process that is chewing up your box.

I'd reboot, not start anything and begin by checking the hardware (disk). Eliminate the basics first. Then I'd first find out what I could about who was running what -and- what changes were recently made to apps. Cause I'm thinking -if ain't that disk- it's a job gone goofy.
Restart work one thing at a time......
and..... "speak softly but carry a big stick i.e. kill -9..ha ha"

Just my thoughts,
Rita

Re: fork: Not enough space

KapilRaj — Tue, 25 May 2004 12:15:57 GMT

u ran out of swap ! I have seen a similier error before.

Kaps

Re: fork: Not enough space

Paul Sperry — Tue, 25 May 2004 12:33:44 GMT

Ok

Thanks guys,

I'll restart the system in single user mode
then init it one level at a time. Then I'll start all of the "custom" stuff. Bill I'll use your script to moniter the memory usage and if nessary I'll use Rita's big stick
to stop the memory sucker. Kapil thanks for the input I think we'll have a solution soon.

I'll post what I find.

Thanks again folks.

Re: fork: Not enough space

Ganesh Babu — Tue, 25 May 2004 16:30:12 GMT

Just last week had this fork issue, our issue was not "not enough space" but instead "too many processes"..

we found that there was a monitoring process changed last week, which was creating run away processes and went to the max..

Ganesh

Re: fork: Not enough space

Paul Sperry — Wed, 26 May 2004 09:46:54 GMT

Still a not sure where the memory leak is
but it is leaking.

Tue May 25 16:20:00 MDT 2004
30780 ias9 2958 /u01/ias9/jdk/bin/../bin/PA_RISC2.0/native_threads/java -Xmx128
30736 oracle8 1855 ora_dbw0_TRAIN
30736 oracle8 1853 ora_pmon_TRAIN
30480 oracle8 1865 ora_arc0_TRAIN
30288 oracle8 1937 oracleTRAIN (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
30224 oracle8 1939 oracleTRAIN (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
30224 oracle8 1863 ora_reco_TRAIN
30224 oracle8 1861 ora_smon_TRAIN
30224 oracle8 1859 ora_ckpt_TRAIN
30224 oracle8 1857 ora_lgwr_TRAIN
9612 oracle8 1930 dbsnmp
8460 oracle8 1907 dbsnmp
7692 oracle8 1903 /u01/oracle8/bin/tnslsnr tcpip -inherit
7692 oracle8 1899 /u01/oracle8/bin/tnslsnr listener -inherit
4228 ias9 2035 /u01/ias9/opmn/bin/opmn -ds
4068 root 1384 /opt/dce/sbin/rpcd
3580 root 2166 /usr/local/sbin/named -c /etc/named.conf
3524 daemon 2310 /usr/bin/X11/X :0 -auth /var/dt/dragoAAAb02283
3036 root 2136 /u01/ias9/Apache/Apache/bin/httpd -d /u01/ias9/Apache/Apache -D
2984 root 1688 /usr/sbin/swagentd -r
m 4101 0xe11cfa18 --rw-r----- oracle8 dba 9 231923712 1839 9981
m 520 0x431c0638 --rw-rw-rw- daemon daemon 1 5767168 2481 2310
m 17926 0x00000000 D-rw------- ias9 oinstall 8 3948548 2081 2081
m 3 0x301c8843 --rw-rw-rw- root root 3 1048576 1422 1433
m 4 0x00000000 --rw------- root root 2 65536 1636 1637
m 1 0x4e0c0002 --rw-rw-rw- root root 1 61760 673 673
m 2 0x41203904 --rw-rw-rw- root root 1 8192 673 685
m 7 0x435dce60 --rw-rw-rw- root root 0 8024 2293 2293
m 0 0x411c27ea --rw-rw-rw- root root 0 348 673 673
T ID KEY MODE OWNER GROUP NATTCH SEGSZ CPID LPID
Shared Memory:
IPC status from /dev/kmem as of Tue May 25 16:20:00 2004
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 2048 0 2048 0% 0 - 1 /dev/vg00/lvol2
reserve - 502 -502
memory 705 42 663 6%
total 2753 544 2209 20% - 0 -

Wed May 26 08:35:00 MDT 2004
30780 ias9 2958 /u01/ias9/jdk/bin/../bin/PA_RISC2.0/native_threads/java -Xmx128
30736 oracle8 1855 ora_dbw0_TRAIN
30736 oracle8 1853 ora_pmon_TRAIN
30480 oracle8 1865 ora_arc0_TRAIN
30288 oracle8 1937 oracleTRAIN (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
30224 oracle8 1939 oracleTRAIN (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
30224 oracle8 1863 ora_reco_TRAIN
30224 oracle8 1861 ora_smon_TRAIN
30224 oracle8 1859 ora_ckpt_TRAIN
30224 oracle8 1857 ora_lgwr_TRAIN
9612 oracle8 1930 dbsnmp
8460 oracle8 1907 dbsnmp
7692 oracle8 1903 /u01/oracle8/bin/tnslsnr tcpip -inherit
7692 oracle8 1899 /u01/oracle8/bin/tnslsnr listener -inherit
4228 ias9 2035 /u01/ias9/opmn/bin/opmn -ds
4068 root 1384 /opt/dce/sbin/rpcd
3580 root 2166 /usr/local/sbin/named -c /etc/named.conf
3524 daemon 2310 /usr/bin/X11/X :0 -auth /var/dt/dragoAAAb02283
3036 root 2136 /u01/ias9/Apache/Apache/bin/httpd -d /u01/ias9/Apache/Apache -D
3036 ias9 3038 /u01/ias9/Apache/Apache/bin/httpd -d /u01/ias9/Apache/Apache -D
m 4101 0xe11cfa18 --rw-r----- oracle8 dba 9 231923712 1839 8208
m 520 0x431c0638 --rw-rw-rw- daemon daemon 1 5767168 2481 2310
m 17926 0x00000000 D-rw------- ias9 oinstall 8 3948548 2081 2081
m 3 0x301c8843 --rw-rw-rw- root root 3 1048576 1422 1433
m 4 0x00000000 --rw------- root root 2 65536 1636 1637
m 1 0x4e0c0002 --rw-rw-rw- root root 1 61760 673 673
m 2 0x41203904 --rw-rw-rw- root root 1 8192 673 685
m 7 0x435dce60 --rw-rw-rw- root root 0 8024 2293 2293
m 0 0x411c27ea --rw-rw-rw- root root 0 348 673 673
T ID KEY MODE OWNER GROUP NATTCH SEGSZ CPID LPID
Shared Memory:
IPC status from /dev/kmem as of Wed May 26 08:35:00 2004
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 2048 52 1996 3% 0 - 1 /dev/vg00/lvol2
reserve - 464 -464
memory 705 434 271 62%
total 2753 950 1803 35% - 0 -

note that memory usage has gone from 6% to 62% over night. Nobody is using this system
so I just don't get it

Re: fork: Not enough space

Sridhar Bhaskarla — Wed, 26 May 2004 10:03:40 GMT

Hi Paul,

As you see the total "reserved" column didn't increase. So, I don't believe there is any memory leak on the system. It is due to more dynamic allocation of memory by the kernel which is particularly seen on 11i. Since you are getting an IO error on /var, probably nothing is getting logged into syslog once the problem starts. Look at "sar -v 2 20" regularly and see if the nproc/nfile kernel parameters are reaching their limits.

Do you know what changed recently?. Like oracle increasing SGA size, SAs adding patches etc.,?

As an option, When this is happening again, instead of hard booting the system, try to produce a crash dump and open a call with HP.

-Sri

Re: fork: Not enough space

Paul Sperry — Wed, 26 May 2004 10:24:57 GMT

Sri,

Thaks for the input. The DBAs swear that nothing has changed. The nproc, nfile
parameters seem fine.

# sar -v 2 20

HP-UX dragon B.11.11 U 9000/785 05/26/04

09:17:05 text-sz ov proc-sz ov inod-sz ov file-sz ov
09:17:07 N/A N/A 132/4096 0 0/34816 0 709/63498 0
09:17:09 N/A N/A 132/4096 0 0/34816 0 709/63498 0
09:17:11 N/A N/A 132/4096 0 0/34816 0 710/63498 0
09:17:13 N/A N/A 132/4096 0 0/34816 0 710/63498 0
09:17:15 N/A N/A 132/4096 0 0/34816 0 710/63498 0
09:17:17 N/A N/A 132/4096 0 0/34816 0 710/63498 0
09:17:19 N/A N/A 132/4096 0 0/34816 0 709/63498 0
09:17:21 N/A N/A 135/4096 0 0/34816 0 718/63498 0
09:17:23 N/A N/A 135/4096 0 0/34816 0 718/63498 0
09:17:25 N/A N/A 135/4096 0 0/34816 0 718/63498 0
09:17:27 N/A N/A 132/4096 0 0/34816 0 709/63498 0
09:17:29 N/A N/A 132/4096 0 0/34816 0 709/63498 0
09:17:31 N/A N/A 132/4096 0 0/34816 0 710/63498 0
09:17:33 N/A N/A 132/4096 0 0/34816 0 710/63498 0
09:17:35 N/A N/A 132/4096 0 0/34816 0 710/63498 0
09:17:37 N/A N/A 132/4096 0 0/34816 0 710/63498 0
09:17:39 N/A N/A 132/4096 0 0/34816 0 710/63498 0
09:17:41 N/A N/A 132/4096 0 0/34816 0 710/63498 0
09:17:43 N/A N/A 132/4096 0 0/34816 0 710/63498 0
09:17:45 N/A N/A 132/4096 0 0/34816 0 710/63498 0

Iam beging to think I may have a disk problem even though I changed it only about a month ago. It would be just my luck.
Thanks again.

Re: fork: Not enough space

Bill Hassell — Wed, 26 May 2004 14:16:21 GMT

The kernel parameters nfile and nproc seem excessively high (63000 and 4096) for the actual files and processes opened at the same time. They could be reduced to about half that size unless you expect a lot of growth in the near future. Since syslog doesn't seem to be working (/var I/O error), use this to check the lvol:

dd if=/dev/vg00/rlvol5 of=/dev/null bs=64k

If it reports an error, your root/boot disk is bad. This assumes lvol5 is /var.

Re: fork: Not enough space

Paul Sperry — Wed, 26 May 2004 14:47:42 GMT

Bill,

The kernel parameters nfile and nproc are set that high per oracle reconmendation.
I decided that /var might be on a corrupt part of the disk. So I :

1. added new lvm var2
2. cd var
3. find. -print | cpio -pcxvdmu /var2
4. edited fstab commented out var and changed var2 to var
5. rebooted
6. removed unused volumes.

Right now I am running
dd if=/dev/rdsk/c3t6d0 of=/dev/null bs=512

So far no errors

Re: fork: Not enough space

Bill Hassell — Wed, 26 May 2004 15:06:52 GMT

The Oracle recommendation is fine if you have 10x more users and processes. It won't hurt anything to have them so high--they just reserve a small amount of RAM.

Copying /var:

/var is the most active (and critical) directory on the system, so copying it while the system is running may not result in a 100% accurate copy. It's best to boot into single user mode, mount /usr /tmp and /var and then do the work there.

A note about cpio options: NEVER use -x. Although there aren't any device files on /var (there better not be), a lot of admins are in the habit of using -x without thinking. -c is meaningless with a disk-disk transfer. Add the -l option to make sure you have links created correctly. The easy-to-remember option list is puddle-move or cpio -pudlmv.

A note about dd. bs=512 is the default and for your 50meg disks, it works well. But it will take an agonizingly long time to copy hundreds of megs. bs=64k or even bs=128k will speed things up about 20x to 50x faster.

Re: fork: Not enough space

Paul Sperry — Wed, 26 May 2004 15:34:59 GMT

Bill,

Thanks for the cpio info. Old habits die hard. But the /var copy seemed to work ok
(so far). The dd worked without errors.

# dd if=/dev/rdsk/c3t6d0 of=/dev/null bs=512
# 71687369+0 records in
# 71687369+0 records out

It did take a couple of hours. I am now running

# dd if=/dev/vg00/lvol9 of=/dev/null bs=128

just for good measure /var is lvol9.

This is a training system and it is possible to have 10 to 15 people connected to it at once so I'll leave the kernel parms for now.

Thanks again

Re: fork: Not enough space

Hai Nguyen_1 — Wed, 26 May 2004 15:43:43 GMT

Paul,

what is your kernel param dbc_max_pct set to? And please post the output of "ipcs -bam".

Hai

Re: fork: Not enough space

Paul Sperry — Wed, 26 May 2004 16:25:19 GMT

Hai,

dbc_max_pct = 50
Max Dynamic Buffer Cache Size as Percent of System RAM size

ipcs -bam
IPC status from /dev/kmem as of Wed May 26 15:21:40 2004
T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME
Shared Memory:
m 0 0x411c27ea --rw-rw-rw- root root root root 0 348 668 668 11:18:57 11:18:57 11:18:50
m 1 0x4e0c0002 --rw-rw-rw- root root root root 1 61760 668 668 11:18:53 11:18:57 11:18:50
m 2 0x412c36a2 --rw-rw-rw- root root root root 1 8192 668 680 11:18:53 11:18:50 11:18:50
m 3 0x301c8843 --rw-rw-rw- root root root root 3 1048576 1398 1408 11:20:45 no-entry 11:19:15
m 4 0x00000000 --rw------- root root root root 2 65536 1611 1612 11:19:25 no-entry 11:19:25
m 4101 0xe11cfa18 --rw-r----- oracle8 dba oracle8 dba 9 231923712 1815 16708 15:17:33 15:17:35 11:19:39
m 17926 0x00000000 D-rw------- ias9 oinstall root oinstall 8 3948548 2064 2064 11:20:34 no-entry 11:20:34
m 7 0x435dce60 --rw-rw-rw- root root root root 0 8024 2269 2269 11:20:39 no-entry 11:20:39
m 520 0x431c0638 --rw-rw-rw- daemon daemon daemon daemon 1 5767168 2393 2283 11:20:42 no-entry 11:20:42

Re: fork: Not enough space

Michael Tully — Wed, 26 May 2004 16:39:09 GMT

The ideal amount of buffer cache is in between 300-500Mb. Anything more will cause some performance issues.

Re: fork: Not enough space

Ross Barton — Wed, 26 May 2004 19:53:06 GMT

If you are sure the problem is a memory leak and if you have the HP Measureware suite installed you will have access to glance.
Running glance will enable you to easily track down processes using lots of memory - or showing signs of memory leaks (memory usage growth over time). This only helps with the identification though.

Can you do without the process? Is it a process that you have written? If you have access to a tool like Purify you could use this to help with debugging the code and actually eliminate the problem (assuming it's code you own and can change).

As other people have suggested - I'd be checking what has changed. Kernel paramaters, patch levels, new software, new versions of existing products etc. Once you've got the culprit that is chewing up your resources - you can take further steps.
Are you up to date with patches?

Cheers,
Ross

Re: fork: Not enough space

Joseph Loo — Wed, 26 May 2004 20:48:06 GMT

hi,

wow, 1024MB physical RAM and 2048MB swap space running Oracle 8i with a SGA of 232MB?

may i know your shmmax, "swapinfo -tam" and vmstat 5 20?

u may have only 2 choices, buy more memory or add more swap space.

regards.