I can't recover disk space after rm of raw disk files

[Brenda Vanshellenbeck_1]

I was trying to delete my Oracle database which has raw datafiles. I used rm
and I think maybe I should have used lvremove.

When I created the database, first I created logical volumes in sam with a
filetype of 'none'. Then the Oracle create database and create tablespace
commands actually put the datafiles in these logical volumes.

When I deleted them, I used the rm command and removed the block and character
files from the /dev/vg01 directory. I did not use sam to remove the logical
volumes.

Now when I go back into LVM, this space deleted above is not showing to be
available (nor is it showing to be allocated anywhere).

When I do pvdisplay -v /dev/dsk/c0t3d0, these extents I thought I deleted are
showing to be 'current' and used by logical volume with a name of '???'

How do I recover this disk space?

Thanks.

[Anthony Goonetilleke_1]

Removing the device files are not the way to delete any type of volume even if
they are managed by LVM or are simple RAW filesystems There are system
commands to remove logical volumes and volume groups.
You should use SAM or commands like lvremove and vgremove. Deleting special
device filenames is a big no no and should never be done except for a few
special circumstances.

[Brenda Vanshellenbeck_1]

Thank you for your prompt reply! Please excuse my ignorance...I'm a rookie.

Any advice on how I can recover from this situation?

There were 3 disks in the volume group. I have removed everything from 2 of
them and then tried 'vgreduce /dev/vg01 /dev/dsk/c04td0' ...I get a message
'physical volume cound not be removed because some of its extents are still in
use'.

Thanks for your help.

[Anthony Goonetilleke_1]

Do the following

pvdisplay -v /dev/dsk/c04td0

towards the end you should see something like
--- Physical extents ---
PE Status LV LE
0000 free 0000

They should all be free. If you see anything like the line below that means
there is still parts of the disk in use.

0000 current /dev/vgtest/testlv 0000

[Brenda Vanshellenbeck_1]

When I do the pvdisplay -v /dev/dsk/ct04d0
not all of the extents are free. The problem extents look like this in the
pvdisplay output:

1000 current ??? 0121

It shows the logical volume name as ???

Thanks again..

[Yu-hsin Seah_2]

If you can remember what the minor number(s) of the logical volume(s) you
deleted was/were eg:

brw-r----- 1 root sys 64 0x010001 Jun
^^^^^^^^
minor #
8 1999 /dev/vg01/lvol1

You can then recreate the lvol's device file using:

mknod c 64 0x0?000? /dev/vgxx/rlvolxx
mknod b 64 0x0?000? /dev/vgxx/lvolxx

where the ?s represent the correct minor number. This will clear the ???
problem in your pvdisplay. After that, you should be able to remove the logical
volume(s) using lvremove.

If you can't recall what the minor numbers were, try doing this:

1. vgreduce -f /dev/vgxx
This should give you a message similar to:

Not all extents are free. i.e. Out of 1023 PEs, only 1013 are free.
You must free all PEs using lvreduce/lvremove before the PV can be removed.
Example: lvreduce -A n -m 0 /dev/vg01/lvol1.
lvremove -A n /dev/vg01/lvol1.
Here's the map of used PEs

--- Logical extents ---
LE LV PE Status 1
0000 lvol4 0000 ???
0001 lvol4 0001 ???
0002 lvol4 0002 ???
0003 lvol4 0003 ???
0004 lvol4 0004 ???
0005 lvol4 0005 ???
0006 lvol4 0006 ???
0007 lvol4 0007 ???
0008 lvol4 0008 ???
0009 lvol4 0009 ???
^^^^^
note this lv number here
it's actually the minor #

All you need to do is to create the device files lvolx with the minor number x
i.e.

# mknod lvol4 b 64 0x040004
# mknod rlvol4 c 64 0x040004

This should clear the pvdisplay problem, and you should be able to
lvremove/vgreduce.

Don't worry if your lvol was not previously lvol4 etc. That's just the way LVM
represents the minor number when you do a vgreduce -f.

Good luck!

[Neil Curtis_1]

Hi Brenda,
If you just want to blitz the devices do a vgexport. Make sure the group file
exists in the volume group directory :

mknod group c 64 0xnn0000 (where nn is a unique number associated with the
volume group. By default vg00 is 0x000000, vg01 is 0x010000 etc etc)

Then change the availability of the volume group with :

vgchange -a n vgxx

This will remove them from the /etc/lvmtab.

When you use the disks again initialise them with a :

pvcreate -f


On the other hand if you want to recover your logical volumes you could try is
a vgimport. For this you need to know your physical devices which were in the
volume group.
If you removed the volume group directory under /dev then recreate it with :

mkdir vgxx (where xx is your vg name)

Then recreate the group file with :

mknod group c 64 0xnn0000 (where nn is a unique number associated with the
volume group. By default vg00 is 0x000000, vg01 is 0x010000 etc etc)

From your first post you mentioned you removed just the lvol and rlvol files.
If this is still the case try :

vgimport -v vgname pvpath (e.g. vgimport -v vg01 /dev/dsk/c5t4d0
/dev/dsk/c5t3d0, note the block device and that you can specify multiple
physical devices).

This should get your logical volumes back so that you can work with them.

Hope this is of some help,

Regards,

Neil.

[Neil Curtis_1]

Brenda,
Sorry - missed a bit - after the vgchange do the vgexport - thats the important
bit. Doh.

Neil.