deleted file still occupying space

[iinfi1]

I think this question would have appeared many times in several forums.
I read that rm'ing a file actually deletes only the name of the file if it is open and the file inode is still present.
it can be removed by killing processes which lsof lists with empty names.
we have 2 nodes of RAC n I use one ocfs moutpoint for export dumps. the export dumps are 4-5G big and sometimes i do a TAR.GZ of the exp files and remove the original ones to save space.
today i did the same and deleted 10 .EXP files immediately after tarring them. I found that the space around 25G occupied by the files did not get released.

user@dbs1:~> df -h | grep u10
/dev/sdi1 200G 164G 37G 82% /u10
user@dbs1:~> du -sh /u10/
113G /u10/

this is how my system shows as of now. I am lucky it didnt happen to a production mountpoint which is running out of space.
I would like to know how i should handle this in future.
>> If I leave it as is, will the new files automatically write to the disk without running out space.

lsof does not show anything for me. I mean there is no file which has empty names.

any inputs on how to deal with such situations and what I should do now to recover this space (apart from rebooting).

[Patrick Wallek]

What lsof command did you use?

The best way to find deleted files is with 'lsof +L 1'

Try that and see what you find.

[Patrick Wallek]

Here's an example of output from a quick test I did:

# lsof +L 1 | grep afile
more 19812 root 3r REG 8,2 110079 0 558341 /root/afile (deleted)


This shows that /root/afile has been deleted but is still open by the 'more' command from pid 19812.

[Steven E. Protter]

Shalom,

The deleted file will occupy space so long as a process maintains an open file handle on it.

Lets use /tmp as an example

fuser -cu /tmp
# displays process with open file handles on the file system

fuser -cuk /tmp

Kills them all. Very dangerous. I warned you.

SEP

[iinfi1]

hi..thanks Steven,Patrik for your inputs.
the command i was trying all this while was
lsof /u10
shows no output

lsof +L 1
does not show the files which i deleted yesterday. how do i deal with this.

and further, if I now copy a 50G file into the partition /u10, will it run out of space at 37G or will it overwrite the open handles?

[iinfi1]

i am afraid i hit a roadblock here.
diff between df and dh is as below. I am losing over 50G

user1@dbs1:~> du -sh /u10/
114G /u10/
user1@dbs1:~> df -h | grep u10
/dev/sdi1 200G 166G 35G 83% /u10

[taylan kurtkaya]

hi iinfi,
run "lsof |grep deleted" command and then kill that processes
taylan.

[iinfi1]

this doesnt show the files which i had deleted.

the command lsof | grep deleted shows a whole bunch of files ... none actually show a size greater than zero.
my partition is a ocfs partition which is shared on two nodes

[Dennis Handly]

>this doesn't show the files which I had deleted.

How can you tell, wrong filesystem?

>none actually show a size greater than zero.

Why not clean them up too?

[iinfi1]

the lsof | grep deleted shows files in the /u01 partition. this is where oracle home is installed.
i deleted files. backup exp dumps from /u10 which is an ocfs partition.
current status
du -sh /u10/
128G /u10/
Filesystem Size Used Avail Use% Mounted on
/dev/sdi1 200G 180G 21G 90% /u10

52G loss.
there are about 110 entries when i run lsof | grep deleted. is it safe to kill those processes.

[Dennis Handly]

>there are about 110 entries when I run lsof | grep deleted. is it safe to kill those processes?

Which is more important, those processes or the space?

[iinfi1]

@Dennis: Kindly explain what you want to convey.
the lsof command does not show any entries from /u10
So will deleting the other processes help. For sure space is imp, but i dont want to screw something.

[Dennis Handly]

>Kindly explain what you want to convey.

You don't know what each file has, only the filesystem. But you do know the process and the owner. That the info you can use to figure out what's more important.

>the lsof command does not show any entries from /u10

Then there is nothing you can do. Somebody is lying to you. You might want to see if you regain that space after some future reboot.

>So will deleting the other processes help. space is imp, but I don't want to screw something.

No, not if it is a different filesystem.

[Patrick Wallek]

>>is an ocfs partition.

Have you checked ALL nodes that use the OCFS partition? Could a node other than the one you are logged into be holding the files open?

[iinfi1]

well, future reboot is an option.
i had checked on both nodes of my cluster. my partition /u10 is not seen in
lsof | grep deleted

the current partition doesnt contain anything related to production. i can try to umount it and remount it.... provided it allows me to.
my concern is, if this happens in my production mountpoint tomorrow, i sud be able to handle it. :)

[bullz]

Hi

not sure, u might have alrady tried all these, but just for your information.

If you use just lsof, it will show lot of entries, please try to use something like this

#lsof /u01 | grep -i deleted


as you stated, yes i understand, no deleted open files. can u also check the list of process using /u01 filesystem? by fuser, and try to understand which process can be killed.

Such issues, shuld be solved without a server reboot... good luck.... please post the solution without fail.

[Earl_Crowder]

I suggest reviewing
http://www.idevelopment.info/data/Oracle/DBA_tips/OCFS2/OCFS2_5.shtml

It may shed some light on the problems you are having.

[iinfi1]

thank you ...
looks like the idevelopment.info like mite help me.

rpm -qa shows ocfs2 version as 1.4.0
the partition in question is not in production.
so shall i umount it from all nodes and do a fsck.ocfs2 -fy /dev/sdi1 without stopping the clusterware?

[Earl_Crowder]

I would run the debugfs.ocfs2 commands first, then the fsck if it turns out there are orphaned files.

[Mohamed Saleem_1]

Hey Mate...

Try this example.. it should work. kill all the process which has the nlink count as 0.


lsof +L1 /u300 | sort -nr +7

Cheers,
Saleem

[iinfi1]

hi all,
sorry for the delay in replying.
i am disappointed with the turn of events at my end.
yday we had some serious issues with the storage eva4400 and both nodes of the RAC restarted automatically.
luckily the data was safe! And space in the partition for which i started the thread came back to its normal state.

i m disappointed that i cud not handle it without the restart ... :(
wonder wat i wud do if it happens in a partition used in prod! :(