Changing the nature of a file

[ConnieK]

I have "inherited" 2 HPUX 11.23 rp4640 clustered servers running Oracle 10. The DBA needs to add a new database file to an existing VG. He states that his attempt errored out saying that the nature of the file was not a RAW device.

I would assume that the previous SA built the VG's and the LV's and ran "newfs" on the LV's - am I right? If so, how can I change this?

Here is what I see for the specific vg/lvol:

lvdisplay -v /dev/vg12:

LV Name /dev/vg12/lvol30
LV Status available/syncd
LV Size (Mbytes) 2048
Current LE 256
Allocated PE 256
Used PV 4


lvdisplay -v /dev/vg12/lvol30
--- Logical volumes ---
LV Name /dev/vg12/lvol30
VG Name /dev/vg12
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 2048
Current LE 256
Allocated PE 256
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation PVG-strict/distributed
IO Timeout (Seconds) default
--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c5t3d3 64 64
/dev/dsk/c5t3d4 64 64
/dev/dsk/c5t3d5 64 64
/dev/dsk/c5t3d6 64 64


/dev/vg12 > ll

crw------- 1 oracle dba 64 0x0c001d Mar 26 2006 rlvol29
crw------- 1 oracle dba 64 0x0c0003 Mar 26 2006 rlvol3
-rw------- 1 oracle dba 63963136 Feb 8 17:13 rlvol30
-rw------- 1 oracle dba 235929600 Feb 8 17:13 rlvol31
crw------- 1 oracle dba 64 0x0c0020 Mar 26 2006 rlvol32
crw------- 1 oracle dba 64 0x0c0021 Mar 26 2006 rlvol33
crw------- 1 oracle dba 64 0x0c0022 Mar 26 2006 rlvol34
crw------- 1 oracle dba 64 0x0c0023 Mar 26 2006 rlvol35
crw------- 1 oracle dba 64 0x0c0024 Mar 26 2006 rlvol36
crw------- 1 oracle dba 64 0x0c0025 Mar 26 2006 rlvol37
crw------- 1 oracle dba 64 0x0c0026 Mar 26 2006 rlvol38
-rw------- 1 oracle dba 80740352 Feb 8 17:17 rlvol39

As you can (maybe) see above in the LL output of /dev/vg12 indicates that the character device file (crw) was not created.

If the previous SA did a "newfs" after creating the lvols (30, 31 and 39) - how can I reverse these to be usable for the DBA's?

Please help me!

Thanks,

Connie

[Pete Randall · Accepted solution]

I'll say those aren't a RAW device. Thhey aren't device files at all, but regular files that someone has been writing into. Look at the sizes of rlvol30, 31, and 39. Is your / filesystem filling up by any chance?

You need to rm these files and re-create the proper device files.


Pete

[Geoff Wild]

If you are indeed using "raw" lvols, then you do NOT want to newfs them. THat is, they need to stay RAW and not have a filesystem on them.

What is in the control file as far as vg's and LV's(if any) of the Servieguard package?

Rgds...Geoff

[A. Clay Stephenson]

Pete has nailed it and because rlvol30-31 are regular (cooked) files in / and because 30-38 are device nodes, there is a very strong probability that someone (that XXXXX DBA) rm'ed the device node and then created new replacements which had the effect of replacing raw device files with cooked files. You need to do a vgdisplay -v and see if the number of LVOL's matches what you expect. You can recover this mess by cp'ing the regular files to a new location. Next rm these regular files and use mknod to recreate the block and character device nodes for each one. Finally use dd to copy the saved regular file back to the corresponding device node. To prevent this happening in the future, you need to remove write permission from oracle on the /dev/vg12
directory for user oracle. The directory should be owned by root but readable and searchable by oracle. The device nodes under this directory should be owned by oracle, group dba and have 600 or 660 mode. This configuration will allow oracle to read and write to any of the device nodes but not remove or create device nodes.

[TwoProc]

Connie,
First issue: rlvol30, rlvol31 and rlvol39 are not actually lvols, they are just files named to to look like lvols. My guess is that a dba just added/created a file to a tablespace that didn't exist as an lvol at all.

See that they are "c..." devices? Notice that they don't have major numbers of 64, and a nice hexadecimal minor number? They are just files, not lvols.

I'm not sure how to stick the toothpaste back into the tube on this one.

Step one would be to clone this whole mess over to another server for testing.

Then you begin testing your scenario on the test server. You begin by doing a shutdown on your database. Make sure to backup the WHOLE database before making the next move. Next, backup to another filesystem for handy use in the next steps rlvols 30, 31, and 39. Delete the three rlvols 30,31, and 39 from /dev/vg12/lvol30, 31, and 39. Then actually create those lvols from the real vg, vg12, making sure tha that the lvols are large enough to at least hold the size of the data in the files that you see. DO NOT run mkfs (these are raw, and by that term - they don't have a file system, so you DON'T want to to put one on there. Then "dd" the file contents onto the drive. An example for rlvol30 would be:

dd if=/backuplocation/rlvol30 of=/dev/vg12/rlvol30 bs=64k

Then bring up your up database.
Now run "dbv" on each of the lvols to make sure all is well. Afterward run some programs known to use data from those datafiles, hold breath. Run a whole set of stuff, say a whole night run of production work against your database, lots and lots of processing.

If all is well, you're probably OK. If not restore EVERYTHING from your backup.

Also, if the three files are type "temp", then in you're in luck, then all you have to do is delete those three temp files, and recreate on the lvols as they are supposed to be. Ditto for undo (rollback) files.

[TwoProc]

re: my comment above on "temp" and "undo" spaces. Of course, what I mean by "delete the files" I mean go through the standard procedure of taking that stuff offline, etc, etc. and recreating - not just using the Unix "rm" command to delete the file.

[ConnieK]

Geoff,

I am not sure how to get into the SG control file to see what's in there. I'm a newbie at clustering and SG. I looked at the cmcluster.conf file and really sam nothing related to VG's & LV's in there as below:

SGCONF=/etc/cmcluster
SGSBIN=/usr/sbin
SGLBIN=/usr/lbin
SGLIB=/usr/lib
SGRUN=/var/adm/cmcluster
SGAUTOSTART=/etc/rc.config.d/cmcluster
SGFFLOC=/opt/cmcluster/cmff
CMSNMPD_LOG_FILE=/var/adm/SGsnmpsuba.log
CMGMSD_LOG_FILE=/var/adm/cmcluster/cmgmsd/cmgmsd.log

John,

I cannot re-create this scenario as I do not have an available test server, but I do appreciate your input - it helped too!

Pete & Clay - your valuable input has helped me understand what I need to do. Here's my plan to fix rlvol30: (the others will be done later)

1. Get a tape back up of rlvol30.
2. cp /dev/vg12/rlvol30 /tmp/space/rlvol30.
By the way, is this figure in KB or Bytes?
-rw------- 1 oracle dba 63963136 Feb 8 17:13 rlvol30
3. rm /dev/vg12/rlvol30
4. mknod rlvol30 b 64 0x0c0030
5. mknod rlvol30 c 64 0x0c0030
5. dd if=/tmp/space/rlvol30 of /dev/vg12/rlvol30 bs=64k

Does this look right?

[Geoff Wild]

If you cd /etc/cmcluster and do a ll you should see a directory for each package that is configured.

cmviewcl
will show you which packages are running or not.

cd into that dir, there choud be a packagename.conf and a packagename.cntl

Example, on my test cluster:

root@sha1 [ /etc/cmcluster ]
# ll
total 224
drwxr-xr-x 2 root sys 8192 Oct 7 2005 MISC
drwxr-xr-x 2 root sys 8192 Sep 15 2004 ORACLE
drwxr-xr-x 2 root sys 8192 Nov 22 2005 SAMBA
drwxr-xr-x 2 bin bin 8192 Apr 18 10:23 cfs
drwxr-xr-x 2 bin bin 8192 Feb 13 2006 cfs.old
-rw------- 1 root root 9044 Jul 10 10:20 cmclconfig
-rw------- 1 root root 11344 Feb 13 2006 cmclconfig.old
-rw------- 1 root root 0 Jul 10 10:20 cmclconfig.tmp
-r-------- 1 bin bin 0 Mar 15 2004 cmknowncmds
-rw-r--r-- 1 root sys 25 Mar 29 2004 cmnodelist.1114
---------- 1 root root 0 Sep 28 2004 config.lck
dr-xr-xr-x 2 bin bin 96 Feb 13 2006 cvm
-r--r--r-- 1 root root 7887 Jan 4 2006 sha1.ascii
-rw-r--r-- 1 root sys 4162 Mar 29 2004 sha1.ascii.1114
-rw-r--r-- 1 root sys 7886 Sep 16 2004 sha1.ascii.1116
-r--r--r-- 1 root root 327 Sep 13 2004 sha1.ascii.bak
drwxr-xr-x 2 bin bin 96 Feb 13 2006 sharedtape
drwxr-xr-x 2 root sys 96 Mar 7 2005 testpkg


# cmviewcl

CLUSTER STATUS
sha1 up

NODE STATUS STATE
sha1 up running

PACKAGE STATUS STATE AUTO_RUN NODE
sambapkg up running enabled sha1
testpkg up running enabled sha1


Basically, look in the cntl files and see if you have any LV's defined:

LV[0]="/dev/vg01/lvora"; FS[0]="/app/oracle"


Rgds...Geoff

[TwoProc]

Hey Steve, what's with outposting me by 10 seconds??? :-)

[ConnieK]

Geoff,

I checked the ServiceGuard cluster control file and it has no reference to VG's or LV's. Just examples - still commented out as below:
# Example: If a package uses two JFS filesystems, pkg01a and pkg01b,
# which are mounted on LVM logical volumes lvol1 and lvol2 for read and
# write operation, you would enter the following:
# LV[0]=/dev/vg01/lvol1; FS[0]=/pkg01a; FS_MOUNT_OPT[0]="-o rw";
# FS_UMOUNT_OPT[0]=""; FS_FSCK_OPT[0]=""; FS_TYPE[0]="vxfs"
#
# LV[1]=/dev/vg01/lvol2; FS[1]=/pkg01b; FS_MOUNT_OPT[1]="-o rw"
# FS_UMOUNT_OPT[1]=""; FS_FSCK_OPT[1]=""; FS_TYPE[1]="vxfs"
#
#LV[0]=""; FS[0]=""; FS_MOUNT_OPT[0]=""; FS_UMOUNT_OPT[0]=""; FS_FSCK_OPT[0]=""
#FS_TYPE[0]=""
#

Do I need to do anything here? And, does it have anything to help me fix the problem I already have - the deleted character device file and how to put it right?