fork: Not enough space

[Paul Sperry]

I have a C3000 running hp-ux 11.11. This morning anything I typed (except pwd) All I would get is fork: Not enough space. I had to power cycle the system in order to get it back up and running. One hour later I am back to where I started, nothing works. There is nothing in the syslogs or dmesg which I check after the power cycle.
The fork: Not enough space to me seems to be a swapspace issue but Iam not sure what's sucking it all up. I have 1024MB of memory and 2048 MB of Swap. It is running oracle but has been for years. The /var: I/O error to me seems to be a disk error but I had just replaced the disk about a month ago. Any ideas? I hate to keep power cycling it.


# pwd
/
# ls
bash: fork: Not enough space
# bdf
bash: fork: Not enough space
# shutdown -r now
bash: fork: Not enough space
# uname -a
bash: fork: Not enough space
# cd /var/adm/syslog
bash: cd: /var/adm/syslog: I/O error
# cd /var
bash: cd: /var: I/O error
# more /etc/fstab
bash: fork: Not enough space


TIA

[Steven E. Protter]

Things to look at:

Is the /var filesystem even mounted

Space on /var and / /tmp

/etc/rc.log

I'd also be wondering if Your kernel is currupt.

Obviously the box boots, but its sure acting like there aren't enough processes or room in the process table.

SEP

[Paul Sperry]

SEP

Thanks for the input

plenty of space on /var and /tmp.

# bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 147456 61898 80241 44% /
/dev/vg00/lvol1 111637 31414 69059 31% /stand
/dev/vg00/lvol8 2564096 394434 2034072 16% /var
/dev/vg00/lvol7 5120000 1460654 3430647 30% /usr
/dev/vg01/datavol 26861568 19564436 7183772 73% /u02
/dev/vg00/orabinvol
15360000 5748694 9010658 39% /u01
/dev/vg00/lvol4 106496 61384 42322 59% /tmp
/dev/vg00/lvol6 1024000 511895 480135 52% /opt
/dev/vg00/ifasvol 6144000 2746221 3185470 46% /ifas
/dev/vg00/lvol5 24576 13154 10726 55% /home

When the system is running /var is mounted.
But when it's in it's "messed up" stage
commands like bdf, df, mount just return the fork error. So I don't know if it's mounted or not

[Bill Hassell]

Not enough space in this context means swapspace. You must have either a runaway process or your DBA decided to change the size of SGA (shared memory) without telling you. Boot up your system, then put these commands into a script:

UNIX95=1 ps -eo vsz,ruser,pid,args | sort -rn | head -20
ipcs -bmop|sort -rnk8
/usr/sbin/swapinfo -tam
echo

Now run this script from cron every 5 minutes:

5,10,15,20,25,30,35,40,45,50,55 * * * * /root/bin/myscript >> /var/tmp/monitormem

The ps listing will show the biggest memory hogs (sorted by size), ipcs will show segments in use sorted by size and swapinfo will show actual memory and swap usage.

[Rita C Workman]

fork and swap are basically saying there are no resources left to run anything.
You have swap allocated, but you could try to add a little more. It is possible that something has happened to your disk, your kernel ....or..... somebody has created a process that is chewing up your box.

I'd reboot, not start anything and begin by checking the hardware (disk). Eliminate the basics first. Then I'd first find out what I could about who was running what -and- what changes were recently made to apps. Cause I'm thinking -if ain't that disk- it's a job gone goofy.
Restart work one thing at a time......
and..... "speak softly but carry a big stick i.e. kill -9..ha ha"

Just my thoughts,
Rita

[KapilRaj]

u ran out of swap ! I have seen a similier error before.

Kaps

[Paul Sperry]

Ok

Thanks guys,

I'll restart the system in single user mode
then init it one level at a time. Then I'll start all of the "custom" stuff. Bill I'll use your script to moniter the memory usage and if nessary I'll use Rita's big stick
to stop the memory sucker. Kapil thanks for the input I think we'll have a solution soon.

I'll post what I find.

Thanks again folks.

[Ganesh Babu]

Just last week had this fork issue, our issue was not "not enough space" but instead "too many processes"..

we found that there was a monitoring process changed last week, which was creating run away processes and went to the max..

Ganesh

[Paul Sperry]

Still a not sure where the memory leak is
but it is leaking.


Tue May 25 16:20:00 MDT 2004
30780 ias9 2958 /u01/ias9/jdk/bin/../bin/PA_RISC2.0/native_threads/java -Xmx128
30736 oracle8 1855 ora_dbw0_TRAIN
30736 oracle8 1853 ora_pmon_TRAIN
30480 oracle8 1865 ora_arc0_TRAIN
30288 oracle8 1937 oracleTRAIN (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
30224 oracle8 1939 oracleTRAIN (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
30224 oracle8 1863 ora_reco_TRAIN
30224 oracle8 1861 ora_smon_TRAIN
30224 oracle8 1859 ora_ckpt_TRAIN
30224 oracle8 1857 ora_lgwr_TRAIN
9612 oracle8 1930 dbsnmp
8460 oracle8 1907 dbsnmp
7692 oracle8 1903 /u01/oracle8/bin/tnslsnr tcpip -inherit
7692 oracle8 1899 /u01/oracle8/bin/tnslsnr listener -inherit
4228 ias9 2035 /u01/ias9/opmn/bin/opmn -ds
4068 root 1384 /opt/dce/sbin/rpcd
3580 root 2166 /usr/local/sbin/named -c /etc/named.conf
3524 daemon 2310 /usr/bin/X11/X :0 -auth /var/dt/dragoAAAb02283
3036 root 2136 /u01/ias9/Apache/Apache/bin/httpd -d /u01/ias9/Apache/Apache -D
2984 root 1688 /usr/sbin/swagentd -r
m 4101 0xe11cfa18 --rw-r----- oracle8 dba 9 231923712 1839 9981
m 520 0x431c0638 --rw-rw-rw- daemon daemon 1 5767168 2481 2310
m 17926 0x00000000 D-rw------- ias9 oinstall 8 3948548 2081 2081
m 3 0x301c8843 --rw-rw-rw- root root 3 1048576 1422 1433
m 4 0x00000000 --rw------- root root 2 65536 1636 1637
m 1 0x4e0c0002 --rw-rw-rw- root root 1 61760 673 673
m 2 0x41203904 --rw-rw-rw- root root 1 8192 673 685
m 7 0x435dce60 --rw-rw-rw- root root 0 8024 2293 2293
m 0 0x411c27ea --rw-rw-rw- root root 0 348 673 673
T ID KEY MODE OWNER GROUP NATTCH SEGSZ CPID LPID
Shared Memory:
IPC status from /dev/kmem as of Tue May 25 16:20:00 2004
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 2048 0 2048 0% 0 - 1 /dev/vg00/lvol2
reserve - 502 -502
memory 705 42 663 6%
total 2753 544 2209 20% - 0 -








Wed May 26 08:35:00 MDT 2004
30780 ias9 2958 /u01/ias9/jdk/bin/../bin/PA_RISC2.0/native_threads/java -Xmx128
30736 oracle8 1855 ora_dbw0_TRAIN
30736 oracle8 1853 ora_pmon_TRAIN
30480 oracle8 1865 ora_arc0_TRAIN
30288 oracle8 1937 oracleTRAIN (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
30224 oracle8 1939 oracleTRAIN (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
30224 oracle8 1863 ora_reco_TRAIN
30224 oracle8 1861 ora_smon_TRAIN
30224 oracle8 1859 ora_ckpt_TRAIN
30224 oracle8 1857 ora_lgwr_TRAIN
9612 oracle8 1930 dbsnmp
8460 oracle8 1907 dbsnmp
7692 oracle8 1903 /u01/oracle8/bin/tnslsnr tcpip -inherit
7692 oracle8 1899 /u01/oracle8/bin/tnslsnr listener -inherit
4228 ias9 2035 /u01/ias9/opmn/bin/opmn -ds
4068 root 1384 /opt/dce/sbin/rpcd
3580 root 2166 /usr/local/sbin/named -c /etc/named.conf
3524 daemon 2310 /usr/bin/X11/X :0 -auth /var/dt/dragoAAAb02283
3036 root 2136 /u01/ias9/Apache/Apache/bin/httpd -d /u01/ias9/Apache/Apache -D
3036 ias9 3038 /u01/ias9/Apache/Apache/bin/httpd -d /u01/ias9/Apache/Apache -D
m 4101 0xe11cfa18 --rw-r----- oracle8 dba 9 231923712 1839 8208
m 520 0x431c0638 --rw-rw-rw- daemon daemon 1 5767168 2481 2310
m 17926 0x00000000 D-rw------- ias9 oinstall 8 3948548 2081 2081
m 3 0x301c8843 --rw-rw-rw- root root 3 1048576 1422 1433
m 4 0x00000000 --rw------- root root 2 65536 1636 1637
m 1 0x4e0c0002 --rw-rw-rw- root root 1 61760 673 673
m 2 0x41203904 --rw-rw-rw- root root 1 8192 673 685
m 7 0x435dce60 --rw-rw-rw- root root 0 8024 2293 2293
m 0 0x411c27ea --rw-rw-rw- root root 0 348 673 673
T ID KEY MODE OWNER GROUP NATTCH SEGSZ CPID LPID
Shared Memory:
IPC status from /dev/kmem as of Wed May 26 08:35:00 2004
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 2048 52 1996 3% 0 - 1 /dev/vg00/lvol2
reserve - 464 -464
memory 705 434 271 62%
total 2753 950 1803 35% - 0 -





note that memory usage has gone from 6% to 62% over night. Nobody is using this system
so I just don't get it

[Sridhar Bhaskarla]

Hi Paul,

As you see the total "reserved" column didn't increase. So, I don't believe there is any memory leak on the system. It is due to more dynamic allocation of memory by the kernel which is particularly seen on 11i. Since you are getting an IO error on /var, probably nothing is getting logged into syslog once the problem starts. Look at "sar -v 2 20" regularly and see if the nproc/nfile kernel parameters are reaching their limits.

Do you know what changed recently?. Like oracle increasing SGA size, SAs adding patches etc.,?

As an option, When this is happening again, instead of hard booting the system, try to produce a crash dump and open a call with HP.

-Sri

[Paul Sperry]

Sri,

Thaks for the input. The DBAs swear that nothing has changed. The nproc, nfile
parameters seem fine.

# sar -v 2 20

HP-UX dragon B.11.11 U 9000/785 05/26/04

09:17:05 text-sz ov proc-sz ov inod-sz ov file-sz ov
09:17:07 N/A N/A 132/4096 0 0/34816 0 709/63498 0
09:17:09 N/A N/A 132/4096 0 0/34816 0 709/63498 0
09:17:11 N/A N/A 132/4096 0 0/34816 0 710/63498 0
09:17:13 N/A N/A 132/4096 0 0/34816 0 710/63498 0
09:17:15 N/A N/A 132/4096 0 0/34816 0 710/63498 0
09:17:17 N/A N/A 132/4096 0 0/34816 0 710/63498 0
09:17:19 N/A N/A 132/4096 0 0/34816 0 709/63498 0
09:17:21 N/A N/A 135/4096 0 0/34816 0 718/63498 0
09:17:23 N/A N/A 135/4096 0 0/34816 0 718/63498 0
09:17:25 N/A N/A 135/4096 0 0/34816 0 718/63498 0
09:17:27 N/A N/A 132/4096 0 0/34816 0 709/63498 0
09:17:29 N/A N/A 132/4096 0 0/34816 0 709/63498 0
09:17:31 N/A N/A 132/4096 0 0/34816 0 710/63498 0
09:17:33 N/A N/A 132/4096 0 0/34816 0 710/63498 0
09:17:35 N/A N/A 132/4096 0 0/34816 0 710/63498 0
09:17:37 N/A N/A 132/4096 0 0/34816 0 710/63498 0
09:17:39 N/A N/A 132/4096 0 0/34816 0 710/63498 0
09:17:41 N/A N/A 132/4096 0 0/34816 0 710/63498 0
09:17:43 N/A N/A 132/4096 0 0/34816 0 710/63498 0
09:17:45 N/A N/A 132/4096 0 0/34816 0 710/63498 0



Iam beging to think I may have a disk problem even though I changed it only about a month ago. It would be just my luck.
Thanks again.

[Bill Hassell]

The kernel parameters nfile and nproc seem excessively high (63000 and 4096) for the actual files and processes opened at the same time. They could be reduced to about half that size unless you expect a lot of growth in the near future. Since syslog doesn't seem to be working (/var I/O error), use this to check the lvol:

dd if=/dev/vg00/rlvol5 of=/dev/null bs=64k

If it reports an error, your root/boot disk is bad. This assumes lvol5 is /var.

[Paul Sperry]

Bill,

The kernel parameters nfile and nproc are set that high per oracle reconmendation.
I decided that /var might be on a corrupt part of the disk. So I :

1. added new lvm var2
2. cd var
3. find. -print | cpio -pcxvdmu /var2
4. edited fstab commented out var and changed var2 to var
5. rebooted
6. removed unused volumes.

Right now I am running
dd if=/dev/rdsk/c3t6d0 of=/dev/null bs=512

So far no errors

[Bill Hassell]

The Oracle recommendation is fine if you have 10x more users and processes. It won't hurt anything to have them so high--they just reserve a small amount of RAM.

Copying /var:

/var is the most active (and critical) directory on the system, so copying it while the system is running may not result in a 100% accurate copy. It's best to boot into single user mode, mount /usr /tmp and /var and then do the work there.

A note about cpio options: NEVER use -x. Although there aren't any device files on /var (there better not be), a lot of admins are in the habit of using -x without thinking. -c is meaningless with a disk-disk transfer. Add the -l option to make sure you have links created correctly. The easy-to-remember option list is puddle-move or cpio -pudlmv.

A note about dd. bs=512 is the default and for your 50meg disks, it works well. But it will take an agonizingly long time to copy hundreds of megs. bs=64k or even bs=128k will speed things up about 20x to 50x faster.

[Paul Sperry]

Bill,

Thanks for the cpio info. Old habits die hard. But the /var copy seemed to work ok
(so far). The dd worked without errors.

# dd if=/dev/rdsk/c3t6d0 of=/dev/null bs=512
# 71687369+0 records in
# 71687369+0 records out

It did take a couple of hours. I am now running

# dd if=/dev/vg00/lvol9 of=/dev/null bs=128

just for good measure /var is lvol9.

This is a training system and it is possible to have 10 to 15 people connected to it at once so I'll leave the kernel parms for now.

Thanks again

[Hai Nguyen_1]

Paul,

what is your kernel param dbc_max_pct set to? And please post the output of "ipcs -bam".

Hai

[Paul Sperry]

Hai,

dbc_max_pct = 50
Max Dynamic Buffer Cache Size as Percent of System RAM size



ipcs -bam
IPC status from /dev/kmem as of Wed May 26 15:21:40 2004
T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME
Shared Memory:
m 0 0x411c27ea --rw-rw-rw- root root root root 0 348 668 668 11:18:57 11:18:57 11:18:50
m 1 0x4e0c0002 --rw-rw-rw- root root root root 1 61760 668 668 11:18:53 11:18:57 11:18:50
m 2 0x412c36a2 --rw-rw-rw- root root root root 1 8192 668 680 11:18:53 11:18:50 11:18:50
m 3 0x301c8843 --rw-rw-rw- root root root root 3 1048576 1398 1408 11:20:45 no-entry 11:19:15
m 4 0x00000000 --rw------- root root root root 2 65536 1611 1612 11:19:25 no-entry 11:19:25
m 4101 0xe11cfa18 --rw-r----- oracle8 dba oracle8 dba 9 231923712 1815 16708 15:17:33 15:17:35 11:19:39
m 17926 0x00000000 D-rw------- ias9 oinstall root oinstall 8 3948548 2064 2064 11:20:34 no-entry 11:20:34
m 7 0x435dce60 --rw-rw-rw- root root root root 0 8024 2269 2269 11:20:39 no-entry 11:20:39
m 520 0x431c0638 --rw-rw-rw- daemon daemon daemon daemon 1 5767168 2393 2283 11:20:42 no-entry 11:20:42

[Michael Tully]

The ideal amount of buffer cache is in between 300-500Mb. Anything more will cause some performance issues.

[Ross Barton]

If you are sure the problem is a memory leak and if you have the HP Measureware suite installed you will have access to glance.
Running glance will enable you to easily track down processes using lots of memory - or showing signs of memory leaks (memory usage growth over time). This only helps with the identification though.

Can you do without the process? Is it a process that you have written? If you have access to a tool like Purify you could use this to help with debugging the code and actually eliminate the problem (assuming it's code you own and can change).

As other people have suggested - I'd be checking what has changed. Kernel paramaters, patch levels, new software, new versions of existing products etc. Once you've got the culprit that is chewing up your resources - you can take further steps.
Are you up to date with patches?

Cheers,
Ross

[Joseph Loo]

hi,

wow, 1024MB physical RAM and 2048MB swap space running Oracle 8i with a SGA of 232MB?

may i know your shmmax, "swapinfo -tam" and vmstat 5 20?

u may have only 2 choices, buy more memory or add more swap space.

regards.

[Bill Hassell]

As mentioned, 1Gb of RAM is crippling Oracle and the other applications. For a small (dozen or less users), you might get away with 2Gb of RAM and a 700meg SGA for Oracle with good performance, but 3-4Gb RAM is more useful. As mentioned, the DBC-max is 50% and that's 500 megs. Now that won't stay at 50% since you have so little RAM available (the DBC will shrink to the minimum size if necessary). However, that may only be 50 megs (5% or RAM) which will badly impact file access for ALL programs and users.

As far as the disk goes, don't bother with individual filesystems--check the whole disk:

dd if=/dev/rdsk/c0t5d0 of=/dev/null bs=64k

(change c0t5d0 to your physical disk address) This will take just a couple of minutes. Change bs=64k to bs=128 and it may take a couple of days (seriously!)

[Hai Nguyen_1]

Paul,

1)
dbc_max_pct = 50
is way too high especially for system running Oracle since Oracle has already reserved SGA memory for itself (In your case, the SGA is almost 231MB as seen from the output of ipcs -bam.) You should reduce your dbc_min_pct to 4 and dbc_max_pct to 8. Then your issue should disappear.

Hai

Note: It is dbc_max_pct which gradually ate up your RAM and made your RAM utilization run up to 60 something % as you had seen.

[Paul Sperry]

Hey guys,

Thanks for all of the kernel param tips,
I'll give them a try when I get the chance.

But this system has been working for years now. Nothing has changed.

dd if=/dev/rdsk/c3t6d0 of=/dev/null bs=128
reported no errors

The system has been up for 2 days but just now when I checked it I again have
fork: Not enough space

Unfortunatly this system is off site

Whats the possibility of bad memory in this case? I am at my wits end.

[Bill Hassell]

Bad memory will cause much worse symptoms than what you are seeing. There will be entries in syslog.log (unless you did not install the online diags) and possible system crashes. With the remote system, the least intrusive is to double or triple the size of swap. Find some empty space on a VG, create an lvol then use swapon to add the swap space. Using swapon, the space is temporary and must be re-enabled on reboot unless you add the entry to fstab. By adding a lot of swap, you should eliminate the problem or at least make the culprit more visible.

[Paul Sperry]

Hey guys,

Ok so I changed dbc_min_pct to 4 and dbc_max_pct to 8 also more than doubled my swap. The system was up for 3 days the all of a sudden fork: Not enough space error reappered!! output from

UNIX95=1 ps -eo vsz,ruser,pid,args | sort -rn | head -20
ipcs -bmop | sort -rnk8
/usr/sbin/swapinfo -tam

just before error:


Sun Jun 6 23:35:00 MDT 2004
38972 ias9 13572 /u01/ias9/jdk/bin/../bin/PA_RISC2.0/native_threads/java -Xmx128
30992 oracle8 10613 oracleTRAIN (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
30736 oracle8 1933 ora_dbw0_TRAIN
30736 oracle8 1931 ora_pmon_TRAIN
30480 oracle8 1946 ora_arc0_TRAIN
30288 oracle8 2080 oracleTRAIN (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
30224 oracle8 2078 oracleTRAIN (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
30224 oracle8 1944 ora_reco_TRAIN
30224 oracle8 1940 ora_smon_TRAIN
30224 oracle8 1937 ora_ckpt_TRAIN
30224 oracle8 1935 ora_lgwr_TRAIN
9612 oracle8 2071 dbsnmp
8460 oracle8 2044 dbsnmp
7692 oracle8 2040 /u01/oracle8/bin/tnslsnr tcpip -inherit
7692 oracle8 2030 /u01/oracle8/bin/tnslsnr listener -inherit
4236 daemon 2702 /usr/bin/X11/X :0 -auth /var/dt/dragoAAAa02698
4228 ias9 2246 /u01/ias9/opmn/bin/opmn -ds
4068 root 1448 /opt/dce/sbin/rpcd
3676 root 1756 /usr/sbin/swagentd -r
3580 root 2581 /usr/local/sbin/named -c /etc/named.conf
m 4613 0xe11cfa18 --rw-r----- oracle8 dba 10 231923712 1910 10671
m 520 0x431c3ce7 --rw-rw-rw- daemon daemon 1 5767168 2706 2702
m 17414 0x00000000 D-rw------- ias9 oinstall 8 3948548 2495 2495
m 3 0x301c8843 --rw-rw-rw- root root 3 1048576 1469 1486
m 4 0x00000000 --rw------- root root 2 65536 1706 1707
m 1 0x4e0c0002 --rw-rw-rw- root root 1 61760 708 708
m 2 0x412c016d --rw-rw-rw- root root 1 8192 708 720
m 7 0x435dce60 --rw-rw-rw- root root 0 8024 2687 2687
m 0 0x411c27ea --rw-rw-rw- root root 0 348 708 708
T ID KEY MODE OWNER GROUP NATTCH SEGSZ CPID LPID
Shared Memory:
IPC status from /dev/kmem as of Sun Jun 6 23:35:01 2004
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 2048 0 2048 0% 0 - 1 /dev/vg00/lvol2
dev 3056 0 3056 0% 0 - 0 /dev/vg00/lvol10
reserve - 556 -556
memory 713 216 497 30%
total 5817 772 5045 13% - 0 -



Any ideas?? This is a remote system (about 7 miles away) and Iam headed out to go get the system to bring it to my location since it is so ubstable.

Thanks for all the input