Removed a file and space not returned. Never seen this before.

[Ray Humpage]

I removed a 10 GB file and bdf shows the same.
How come the space didn't come back? What?

Before
/dev/vg02/psindx01 24576000 24154006 408808 98% /u02

-rw-rw---- 1 oinstall 10066337792 Nov 2 15:37 /u02/oradata/prd88/psindex.dbf
-rw-rw---- 1 oinstall 629153792 Nov 2 15:44 /u02/oradata/prd88/psindex1.dbf
-rw-rw---- 1 oinstall 686825472 Nov 2 15:44 /u02/oradata/prd88/psindex2.dbf
-rw-rw---- 1 oinstall 734011392 Nov 2 14:32 /u02/oradata/prd88/psindex5.dbf
-rw-rw---- 1 oinstall 896540672 Nov 2 15:46 /u02/oradata/prd88/psindex3.dbf
-rw-rw---- 1 oinstall 943726592 Nov 2 15:37 /u02/oradata/prd88/psindex4.dbf

After
/dev/vg02/psindx01 24576000 24154006 408808 98% /u02

-rw-rw---- 1 oinstall 629153792 Nov 2 15:44 /u02/oradata/prd88/psindex1.dbf
-rw-rw---- 1 oinstall 686825472 Nov 2 15:44 /u02/oradata/prd88/psindex2.dbf
-rw-rw---- 1 oinstall 734011392 Nov 2 14:32 /u02/oradata/prd88/psindex5.dbf
-rw-rw---- 1 oinstall 896540672 Nov 2 15:46 /u02/oradata/prd88/psindex3.dbf
-rw-rw---- 1 oinstall 943726592 Nov 2 15:37 /u02/oradata/prd88/psindex4.dbf

[Muthukumar_5]

Is there sub-level logical volumes under /u02 mount point? Post full bdf result. Some other logical volume's file is deleted and /u02 mount point is not touched for that.

-Muthu

[Arunvijai_4]

Is that file being created dynamically ? Stop all Oracle related processes and try deleting it.

-Arun

[Ray Humpage]

No logical volumes below. Full bdf below.
The index was dropped at the Oracle level and the file was removed at the OS level. The space remained. I've done this before and never had to bounce Oracle.

nmhhrprd(/)# bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 143360 98004 42652 70% /
/dev/vg00/lvol1 251285 35897 190259 16% /stand
/dev/vg00/lvol8 1024000 577091 424729 58% /var
/dev/vg00/lvol7 1433600 1233234 189229 87% /usr
/dev/vg00/lvol4 65536 4741 57377 8% /tmp
/dev/vg00/lvol6 5632000 5451614 169120 97% /opt
/dev/vg00/lvol5 20480 1275 18065 7% /home
nmhptprd.nmh.org:/hrarch
4096000 2720 3837520 0% /hrarch
/dev/vg06/pstemp01 6144000 4559410 1485558 75% /u06
/dev/vg04/psredo01 6144000 2053362 3835040 35% /u04
/dev/vg03/pssys01 9216000 7266747 1827554 80% /u03
/dev/vg02/psindx01 24576000 24154006 408808 98% /u02
/dev/vg01/psdata01 28672000 26509860 2128392 93% /u01
/dev/vg06/psstage 18432000 10810170 7383818 59% /stage
/dev/vg07/psarch01 12288000 3566662 8448894 30% /u07
/dev/vg07/pshome 5120000 4611048 497050 90% /home1
/dev/vg03/wstage 12288000 4842733 6979989 41% /wstage
/dev/vg02/psredo03 4096000 1232441 2684591 31% /u99

[Devender Khatana]

Hi,

I have seen this behaviour where the data deleted take some time to reflect in free space with FC array. But the data in the case was around 1 TB and took around 5-6 minutes to reflect.

Another option is that the oracle process might be using the file and will not release it untill restarted.

HTH,
Devender

[RAC_1]

That is most proably becuase you deleted a file that was in use by a process.
lsof +aL1 /mount_point
Not the file that has link count less than 1.You will have to stop the process that was using it.

[Michael Jorgensen · Accepted solution]

If a file is open by a process and you try to remove it the file listed in the directory will disappear because the OS will remove the link to the underlying inode from the directory.

However the OS cannot delete a file that is in use by a process so the underlying inode for the file still exists and is still holding the disk blocks used by the inode.

The only way to reclaim the space is for the process that has the inode open to be stopped. In this case it looks like an Oracle process of some description might have the file/inode open.

The quickest/easiest way to reclaim the space is to reboot but that's not always the easiest thing to arrange.

[Mark Nieuwboer]

Hi Ray,

Look at the last 2 posts from RAV and jorgensen.
You have removed i file that is used by a program. Restart the program and you have your space back.

grtz. mark

[Cheryl Griffin]

No need to reboot at this point just bounce the process.

In addition to lsof, uli can also be used. Look for it attached to Bruno's post on
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=42677

[Matti_Kurkela]

Looks like Oracle database index files.
You definitely should have asked your Database Administrator to "drop" the file first before you removed it.

Or maybe you just removed something you should not be removing?

If the Oracle is still running, it is not yet too late: if your DBA would not have wanted you to remove that file, he/she can still fix the situation. The Oracle is hanging on to the file, so the OS does not actually remove the file just yet.

If the Oracle database is stopped now, the index file will really be gone. If there were active database indexes in that file, your DBA might not like you. You may have just slowed his/her database to a crawl by breaking the database indexes. :-(

The DBA can re-create the indexes, but the re-creation will take some time and processing power.

[A. Clay Stephenson]

Do a man 2 unlink (the system call that underlies the rm command) and read it and never ask this question again.

[James R. Ferguson]

Hi Ray:

To add to Clay's comments, the concept of creating and immediately unlinking a file is actually common. You have the benefit of the file with "automatic" removal whenever (or however) the program terminates without adding epilog code.

Regards!

...JRF...

[Ray Humpage]

Thanks everyone for all your help.

Many of you stated the same thing - that stopping Oracle would release the space - but added some additional info and commands that will be helpful in the future.

I come away from this with a solution and increased knowledge.

Thanks again.