File System constantly filling up

[TWBSupport]

I have a file system that my oracle dba has logs written to and if this fills up to 100%, of course the database will stop running. It's 10G and I can't see why it's showing 87% full at this point. The file sizes do not add up.

Can anyone clue me in on what's going? The file is attached. Thank you!

[Devender Khatana · Accepted solution]

Hi,

Sometimes there are files open by applications and remiving them do not free up the space unless allication is restarted. Have you removed some such files recently? If yes then restarting the application will release the actual space.

HTH,
Devender

[Andy Torres]

My first guess is that someone deleted a very large file that was in use (open).
If you do a fuser -u /dev/vg03/lvol38 you will see the process IDs using that filesystem and you can whittle down which process you need to stop in order to "unlock" the space again.

[Devender Khatana]

Hi Again,

As the file system is in use by oracle. Stopping or killing an process ID is not the recommended way. I again say if some such files were removed recently then restart applciation.

HTH,
Devender

[Mark Nieuwboer]

Hi denda,

I agree with denver.
Probebly an actieve file was removed.
To free this space up you have to restart the application.

grtz. Mark

[TWBSupport]

Well, I'm not the dba and the dba has all control over stopping and starting the database. I'll ask but I asked him to do that yesterday and he made it sound as though this is my fault (hardware config) although I go through this everytime there's a space issue, I like to make sure I have hit all the spots and haven't missed anything that why I ask you experts! :o) All users are the oracle users, so that is more than likely what is happening. He has archives being copied to this directory and then moved somewhere or deleted, I'm waiting for a response from him on that one.

[Yogeeraj_1]

hi,

Check your top processes. Do you see any oracle processes which could have gone astray?

you can check if the oracle process is still active at the database level using the following sql statement:

select b.sid SID,b.serial# "Serial#", c.spid "srvPID", b.osuser, b.username, b.status, b.client_info from v$session b, v$process c where b.paddr = c.addr and c.sPID = &OSPID

you may also change the location of your files (arch, udump, bdump and cdump)

kind regards
yogeeraj

[Andy Torres]

Don't get me wrong, I don't usually kill processes arbitrarily. When I said "stop" the process, I meant "Go find the DBA, present your case, go through all the downtime blabbidy-blah, etc.", of course.

You can use lsof to pinpoint the open file to solidify your point to the DBA.

Left unchecked, it's likely this will cause other problems and the process will die in time with an "out of space" error or something, so.... stop it now or deal with possible DB problems when it dies, his choice.

[Kent Ostby]

Denda --

The idea of apps with temporary files is very possible.

Do you mount any filesystems onto the oracle file system?

At times a file could get written to that mount point if the sub-filesystem wasn't mounted at the time.

Generally, I like to do:

find . -size +10000000c -print

which will show you files of 10 meg or more ... if that's too big of a list then add a zero and find the 100 meg files.

[TWBSupport]

Thanks to you all.. The dba stopped and started the process that he had writing to this directory and the percentage immediately went down to 63% from 87%. We're watching it closely to make sure nothing else could be causing issues. Thank you all for helping me so quickly. As soon as the system starts reponding properly all of you will receive your points!

[TWBSupport]

Well, back to the drawing board... After putting all the numbers together after stopping and starting the processes, they still do not jive. Now the process that is happening is a move of an archive file to /u104, then a gzip is done on the file in the /u104 directory. I know gzip needs more room to zip than the actual file size, but would it be over double? Attached is the new file.

[Andy Torres]

I'd say stop the database again, then run your checks again (fuser, lsof, find...) to isolate a process that may still be using a deleted file. The database may not be the only offender in the mix.

[TWBSupport]

Weeeellllll... I asked the dba to stop the database for me again so I can see what proceses are still hanging around on that specific directory to rule out anything I have going and the database just crashed seconds prior to me asking. I have a feeling this was something that was happening sooner

[Andy Torres]

Sounds like a perfect opportunity to check for other processes holding deleted file handles, and maybe even consider a reboot if the database has to be completely restarted.