No space left on device error

[Vasu_1]

Hi,

Would greatly appreciate it if someone could help me here. Jobs are at stake.. :-(

Our release runs on an HP 3600 box. After a few hours of operation, we get an Oracle error 12500 that says no space left on device. When I try to bring up SAM it says the same. BUT, there is GOBS of disk space left - several gigs on both the disks we have.

A point of note is that we recently increased several kernel parameters whose values I show below to solve an OutOfMemory error we used to get earlier (these were recommended by HPJconfig). I am thinking that has got to something to do with this problem, although the experts can correct me. Any help will be greatly appreciated.

max_thread_proc 512
maxdsiz 2063835136
maxfiles 2048
maxfiles_lim 2048
ncallout 6000
nkthread 6000
nproc 2068

[Michael Tully]

Please post a copy of the 'bdf' output and any messages relating to 'no space' left from either the output of 'dmesg' or entries in /var/adm/syslog/syslog.log

[A. Clay Stephenson]

I'm a little confused,ORA-12500 is related to the listener. Did you also get a smaller errno value. That is probably more helpful. What you may be doing is running out of virtual address space (not enough swap).

[James R. Ferguson]

Hi:

The "no space left on device" would equate to errno 28. This might suggest insufficient shared memory.

Regards!

...JRF...

[Vasu_1]

Here is the bdf output and the exact error message. (We had to reboot the machine so the latest log doesn't show errors):
/dev/vg00/lvol3 16625664 4313884 11927892 27% /
/dev/vg00/lvol1 99669 34931 54771 39% /stand
/dev/dsk/c3t5d0 17783240 833284 16685308 5% /disk2

Problem releasing Db Connections java.sql.SQLException: Io exception: Connection r
efused(DESCRIPTIO
N=(TMP=)(VSNNUM=135295744)(ERR=12500)(ERROR_STACK=(ERROR=(CODE=12500)(EMFI=4))(E
RROR=(CODE=12560)(EMFI=4))(ERROR=(CODE=530)(EMFI=4))(ERROR=(BUF='HPUX Error: 28:
No space left on device'))))

We had to reboot the machine because it was needed.

[A. Clay Stephenson]

Jim may be on the right track. This appears to be errno 28 [ENOSPC]. System calls that can fail for this reason include shmget(), msgget(), and semget().

You may be reaching the system-wide limits for shared memory identifiers, message queue identifiers, and semaphore identifiers respectively. You should look at shmmni, semmni, and msgmni.

A good way to track down this sort of problem is to do a man 2 for each system call and look for those which can fail with the indicated errno value. The above list is not complete but is a very good place to start.

[Sandip Ghosh]

You can look at the following link for java related problem.

http://h21007.www2.hp.com/dspp/tech/tech_TechDocumentDetailPage_IDX/1,1701,1618,00.html

Sandip

[Vasu_1]

Here are those other kernel parameters:
shmmni 200
semmni 256
msgmni 50

Would nproc have anything to do with it? We noticed another machine that had 4096 as the value and we've not seen the problem there (We have 2048).

[A. Clay Stephenson]

To answer your question about nproc, you should examine the system calls that nproc might effect. Two come to mind immediately, fork() and exec(). You should man 2 fork and man 2 exec and see if those guys can set errno to ENOSPC. The answer is no.

There is no way to know what the "correct" values for these tunables are without knowing a lot more about your system. I can say that those are on the small side. For medium to large databases msgmni might be ~ 1060, shmmni ~ 1024, and semmni might be ~ 5000 but again it depends.

[James R. Ferguson]

Hi:

'nproc' limits the number of system-wide processes. You would see errors like EAGAIN (no more proccesses; errno-11) or perhaps ENOMEM (not enough core; errno-12) during fork()'s if this limit were being hit.

ENOSPC [errno-28] (no space left on device) points to 'shmmni' on shmget() calls and this is the error you seem to be having.

ENOSPC also occurs when 'semmni' is too small (for trying to get semaphores).

ENOSPC errors are also returned for 'msgget()' system calls limited by too small a 'msgmni' kernel value.

I thing you have identified at least some of your boundries.

Regards!

...JRF...

[Vasu_1]

Just an update on this problem. I compared all the parameters to a working machine and found that maxswapchunks had to be increased to 1025 (from 512) and nproc had to be increased to 4096. After that, it worked consistently. Interestingly, along with these modifications, when I increased msgmni to 1068, it gave the same error! I had to bring it down to its earlier value of 50 to make it work. Thanks to all for your suggestions.

[Patrick Wallek]

maxswapchunks should have NO effect on how Oracle runs.

maxswapchunks only effect how much swap space you can configure. The only thing increaseing the maxswapchunks parameter bought you was the ability to now configure up to 8GB of swap space.

If you didn't add anymore swap space, then that was not part of your problem.

More info on maxswapchunks: http://docs.hp.com/hpux/onlinedocs/939/KCParms/KCparam.MaxSwapChunks.html

[Patrick Wallek]

Oops....used the incorrect value to figure how much swap you can now configure.

Wit maxswapchunks at 1025 you can have up to 2GB of swap space configured.

The rest of my previous comment still applies, however.

[Vasu_1]

Hi Patrick,

Interesting. Swap space was also added before. It was increased to 2GB from 1GB. I don't know if SAM automatically suggests increasing maxswapchunks or not. The swap was added by a verification guy and I am not sure if the option to increase maxswapchunks was offered and it was declined by him. I am hunting for another machine where I can try and replicate this..