topic Re: deleted file still occupying space in Operating System - Linux

deleted file still occupying space

iinfi1 — Tue, 09 Nov 2010 18:46:16 GMT

I think this question would have appeared many times in several forums.
I read that rm'ing a file actually deletes only the name of the file if it is open and the file inode is still present.
it can be removed by killing processes which lsof lists with empty names.
we have 2 nodes of RAC n I use one ocfs moutpoint for export dumps. the export dumps are 4-5G big and sometimes i do a TAR.GZ of the exp files and remove the original ones to save space.
today i did the same and deleted 10 .EXP files immediately after tarring them. I found that the space around 25G occupied by the files did not get released.

user@dbs1:~> df -h | grep u10
/dev/sdi1 200G 164G 37G 82% /u10
user@dbs1:~> du -sh /u10/
113G /u10/

this is how my system shows as of now. I am lucky it didnt happen to a production mountpoint which is running out of space.
I would like to know how i should handle this in future.
>> If I leave it as is, will the new files automatically write to the disk without running out space.

lsof does not show anything for me. I mean there is no file which has empty names.

any inputs on how to deal with such situations and what I should do now to recover this space (apart from rebooting).

Re: deleted file still occupying space

Patrick Wallek — Tue, 09 Nov 2010 19:48:11 GMT

What lsof command did you use?

The best way to find deleted files is with 'lsof +L 1'

Try that and see what you find.

Re: deleted file still occupying space

Patrick Wallek — Tue, 09 Nov 2010 19:51:21 GMT

Here's an example of output from a quick test I did:

# lsof +L 1 | grep afile
more 19812 root 3r REG 8,2 110079 0 558341 /root/afile (deleted)

This shows that /root/afile has been deleted but is still open by the 'more' command from pid 19812.

Re: deleted file still occupying space

Steven E. Protter — Tue, 09 Nov 2010 20:48:56 GMT

Shalom,

The deleted file will occupy space so long as a process maintains an open file handle on it.

Lets use /tmp as an example

fuser -cu /tmp
# displays process with open file handles on the file system

fuser -cuk /tmp

Kills them all. Very dangerous. I warned you.

SEP

Re: deleted file still occupying space

iinfi1 — Wed, 10 Nov 2010 03:58:56 GMT

hi..thanks Steven,Patrik for your inputs.
the command i was trying all this while was
lsof /u10
shows no output

lsof +L 1
does not show the files which i deleted yesterday. how do i deal with this.

and further, if I now copy a 50G file into the partition /u10, will it run out of space at 37G or will it overwrite the open handles?

Re: deleted file still occupying space

iinfi1 — Thu, 11 Nov 2010 04:22:34 GMT

i am afraid i hit a roadblock here.
diff between df and dh is as below. I am losing over 50G

user1@dbs1:~> du -sh /u10/
114G /u10/
user1@dbs1:~> df -h | grep u10
/dev/sdi1 200G 166G 35G 83% /u10

Re: deleted file still occupying space

taylan kurtkaya — Fri, 12 Nov 2010 14:21:41 GMT

hi iinfi,
run "lsof |grep deleted" command and then kill that processes
taylan.

Re: deleted file still occupying space

iinfi1 — Sat, 13 Nov 2010 19:10:56 GMT

this doesnt show the files which i had deleted.

the command lsof | grep deleted shows a whole bunch of files ... none actually show a size greater than zero.
my partition is a ocfs partition which is shared on two nodes

Re: deleted file still occupying space

Dennis Handly — Sun, 14 Nov 2010 00:01:30 GMT

>this doesn't show the files which I had deleted.

How can you tell, wrong filesystem?

>none actually show a size greater than zero.

Why not clean them up too?

Re: deleted file still occupying space

iinfi1 — Sun, 14 Nov 2010 04:45:15 GMT

the lsof | grep deleted shows files in the /u01 partition. this is where oracle home is installed.
i deleted files. backup exp dumps from /u10 which is an ocfs partition.
current status
du -sh /u10/
128G /u10/
Filesystem Size Used Avail Use% Mounted on
/dev/sdi1 200G 180G 21G 90% /u10

52G loss.
there are about 110 entries when i run lsof | grep deleted. is it safe to kill those processes.

Re: deleted file still occupying space

Dennis Handly — Sun, 14 Nov 2010 11:57:10 GMT

>there are about 110 entries when I run lsof | grep deleted. is it safe to kill those processes?

Which is more important, those processes or the space?

Re: deleted file still occupying space

iinfi1 — Sun, 14 Nov 2010 19:15:55 GMT

@Dennis: Kindly explain what you want to convey.
the lsof command does not show any entries from /u10
So will deleting the other processes help. For sure space is imp, but i dont want to screw something.

Re: deleted file still occupying space

Dennis Handly — Sun, 14 Nov 2010 20:00:08 GMT

>Kindly explain what you want to convey.

You don't know what each file has, only the filesystem. But you do know the process and the owner. That the info you can use to figure out what's more important.

>the lsof command does not show any entries from /u10

Then there is nothing you can do. Somebody is lying to you. You might want to see if you regain that space after some future reboot.

>So will deleting the other processes help. space is imp, but I don't want to screw something.

No, not if it is a different filesystem.

Re: deleted file still occupying space

Patrick Wallek — Mon, 15 Nov 2010 22:46:46 GMT

>>is an ocfs partition.

Have you checked ALL nodes that use the OCFS partition? Could a node other than the one you are logged into be holding the files open?

Re: deleted file still occupying space

iinfi1 — Tue, 16 Nov 2010 10:40:08 GMT

well, future reboot is an option.
i had checked on both nodes of my cluster. my partition /u10 is not seen in
lsof | grep deleted

the current partition doesnt contain anything related to production. i can try to umount it and remount it.... provided it allows me to.
my concern is, if this happens in my production mountpoint tomorrow, i sud be able to handle it. :)

Re: deleted file still occupying space

bullz — Wed, 17 Nov 2010 13:59:36 GMT

Hi

not sure, u might have alrady tried all these, but just for your information.

If you use just lsof, it will show lot of entries, please try to use something like this

#lsof /u01 | grep -i deleted

as you stated, yes i understand, no deleted open files. can u also check the list of process using /u01 filesystem? by fuser, and try to understand which process can be killed.

Such issues, shuld be solved without a server reboot... good luck.... please post the solution without fail.

Re: deleted file still occupying space

Earl_Crowder — Wed, 17 Nov 2010 15:26:55 GMT

I suggest reviewing
http://www.idevelopment.info/data/Oracle/DBA_tips/OCFS2/OCFS2_5.shtml

It may shed some light on the problems you are having.

Re: deleted file still occupying space

iinfi1 — Wed, 17 Nov 2010 19:04:51 GMT

thank you ...
looks like the idevelopment.info like mite help me.

rpm -qa shows ocfs2 version as 1.4.0
the partition in question is not in production.
so shall i umount it from all nodes and do a fsck.ocfs2 -fy /dev/sdi1 without stopping the clusterware?

Re: deleted file still occupying space

Earl_Crowder — Fri, 19 Nov 2010 19:06:34 GMT

I would run the debugfs.ocfs2 commands first, then the fsck if it turns out there are orphaned files.

Re: deleted file still occupying space

Mohamed Saleem_1 — Mon, 22 Nov 2010 08:24:10 GMT

Hey Mate...

Try this example.. it should work. kill all the process which has the nlink count as 0.

lsof +L1 /u300 | sort -nr +7

Cheers,
Saleem