Changing the nature of a file

[ConnieK]

I have "inherited" 2 HPUX 11.23 rp4640 clustered servers running Oracle 10. The DBA needs to add a new database file to an existing VG. He states that his attempt errored out saying that the nature of the file was not a RAW device.

I would assume that the previous SA built the VG's and the LV's and ran "newfs" on the LV's - am I right? If so, how can I change this?

Here is what I see for the specific vg/lvol:

lvdisplay -v /dev/vg12:

LV Name /dev/vg12/lvol30
LV Status available/syncd
LV Size (Mbytes) 2048
Current LE 256
Allocated PE 256
Used PV 4


lvdisplay -v /dev/vg12/lvol30
--- Logical volumes ---
LV Name /dev/vg12/lvol30
VG Name /dev/vg12
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 2048
Current LE 256
Allocated PE 256
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation PVG-strict/distributed
IO Timeout (Seconds) default
--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c5t3d3 64 64
/dev/dsk/c5t3d4 64 64
/dev/dsk/c5t3d5 64 64
/dev/dsk/c5t3d6 64 64


/dev/vg12 > ll

crw------- 1 oracle dba 64 0x0c001d Mar 26 2006 rlvol29
crw------- 1 oracle dba 64 0x0c0003 Mar 26 2006 rlvol3
-rw------- 1 oracle dba 63963136 Feb 8 17:13 rlvol30
-rw------- 1 oracle dba 235929600 Feb 8 17:13 rlvol31
crw------- 1 oracle dba 64 0x0c0020 Mar 26 2006 rlvol32
crw------- 1 oracle dba 64 0x0c0021 Mar 26 2006 rlvol33
crw------- 1 oracle dba 64 0x0c0022 Mar 26 2006 rlvol34
crw------- 1 oracle dba 64 0x0c0023 Mar 26 2006 rlvol35
crw------- 1 oracle dba 64 0x0c0024 Mar 26 2006 rlvol36
crw------- 1 oracle dba 64 0x0c0025 Mar 26 2006 rlvol37
crw------- 1 oracle dba 64 0x0c0026 Mar 26 2006 rlvol38
-rw------- 1 oracle dba 80740352 Feb 8 17:17 rlvol39

As you can (maybe) see above in the LL output of /dev/vg12 indicates that the character device file (crw) was not created.

If the previous SA did a "newfs" after creating the lvols (30, 31 and 39) - how can I reverse these to be usable for the DBA's?

Please help me!

Thanks,

Connie

[Pete Randall · Accepted solution]

I'll say those aren't a RAW device. Thhey aren't device files at all, but regular files that someone has been writing into. Look at the sizes of rlvol30, 31, and 39. Is your / filesystem filling up by any chance?

You need to rm these files and re-create the proper device files.


Pete

[Geoff Wild]

If you are indeed using "raw" lvols, then you do NOT want to newfs them. THat is, they need to stay RAW and not have a filesystem on them.

What is in the control file as far as vg's and LV's(if any) of the Servieguard package?

Rgds...Geoff

[A. Clay Stephenson]

Pete has nailed it and because rlvol30-31 are regular (cooked) files in / and because 30-38 are device nodes, there is a very strong probability that someone (that XXXXX DBA) rm'ed the device node and then created new replacements which had the effect of replacing raw device files with cooked files. You need to do a vgdisplay -v and see if the number of LVOL's matches what you expect. You can recover this mess by cp'ing the regular files to a new location. Next rm these regular files and use mknod to recreate the block and character device nodes for each one. Finally use dd to copy the saved regular file back to the corresponding device node. To prevent this happening in the future, you need to remove write permission from oracle on the /dev/vg12
directory for user oracle. The directory should be owned by root but readable and searchable by oracle. The device nodes under this directory should be owned by oracle, group dba and have 600 or 660 mode. This configuration will allow oracle to read and write to any of the device nodes but not remove or create device nodes.

[TwoProc]

Connie,
First issue: rlvol30, rlvol31 and rlvol39 are not actually lvols, they are just files named to to look like lvols. My guess is that a dba just added/created a file to a tablespace that didn't exist as an lvol at all.

See that they are "c..." devices? Notice that they don't have major numbers of 64, and a nice hexadecimal minor number? They are just files, not lvols.

I'm not sure how to stick the toothpaste back into the tube on this one.

Step one would be to clone this whole mess over to another server for testing.

Then you begin testing your scenario on the test server. You begin by doing a shutdown on your database. Make sure to backup the WHOLE database before making the next move. Next, backup to another filesystem for handy use in the next steps rlvols 30, 31, and 39. Delete the three rlvols 30,31, and 39 from /dev/vg12/lvol30, 31, and 39. Then actually create those lvols from the real vg, vg12, making sure tha that the lvols are large enough to at least hold the size of the data in the files that you see. DO NOT run mkfs (these are raw, and by that term - they don't have a file system, so you DON'T want to to put one on there. Then "dd" the file contents onto the drive. An example for rlvol30 would be:

dd if=/backuplocation/rlvol30 of=/dev/vg12/rlvol30 bs=64k

Then bring up your up database.
Now run "dbv" on each of the lvols to make sure all is well. Afterward run some programs known to use data from those datafiles, hold breath. Run a whole set of stuff, say a whole night run of production work against your database, lots and lots of processing.

If all is well, you're probably OK. If not restore EVERYTHING from your backup.

Also, if the three files are type "temp", then in you're in luck, then all you have to do is delete those three temp files, and recreate on the lvols as they are supposed to be. Ditto for undo (rollback) files.

[TwoProc]

re: my comment above on "temp" and "undo" spaces. Of course, what I mean by "delete the files" I mean go through the standard procedure of taking that stuff offline, etc, etc. and recreating - not just using the Unix "rm" command to delete the file.

[ConnieK]

Geoff,

I am not sure how to get into the SG control file to see what's in there. I'm a newbie at clustering and SG. I looked at the cmcluster.conf file and really sam nothing related to VG's & LV's in there as below:

SGCONF=/etc/cmcluster
SGSBIN=/usr/sbin
SGLBIN=/usr/lbin
SGLIB=/usr/lib
SGRUN=/var/adm/cmcluster
SGAUTOSTART=/etc/rc.config.d/cmcluster
SGFFLOC=/opt/cmcluster/cmff
CMSNMPD_LOG_FILE=/var/adm/SGsnmpsuba.log
CMGMSD_LOG_FILE=/var/adm/cmcluster/cmgmsd/cmgmsd.log

John,

I cannot re-create this scenario as I do not have an available test server, but I do appreciate your input - it helped too!

Pete & Clay - your valuable input has helped me understand what I need to do. Here's my plan to fix rlvol30: (the others will be done later)

1. Get a tape back up of rlvol30.
2. cp /dev/vg12/rlvol30 /tmp/space/rlvol30.
By the way, is this figure in KB or Bytes?
-rw------- 1 oracle dba 63963136 Feb 8 17:13 rlvol30
3. rm /dev/vg12/rlvol30
4. mknod rlvol30 b 64 0x0c0030
5. mknod rlvol30 c 64 0x0c0030
5. dd if=/tmp/space/rlvol30 of /dev/vg12/rlvol30 bs=64k

Does this look right?

[Geoff Wild]

If you cd /etc/cmcluster and do a ll you should see a directory for each package that is configured.

cmviewcl
will show you which packages are running or not.

cd into that dir, there choud be a packagename.conf and a packagename.cntl

Example, on my test cluster:

root@sha1 [ /etc/cmcluster ]
# ll
total 224
drwxr-xr-x 2 root sys 8192 Oct 7 2005 MISC
drwxr-xr-x 2 root sys 8192 Sep 15 2004 ORACLE
drwxr-xr-x 2 root sys 8192 Nov 22 2005 SAMBA
drwxr-xr-x 2 bin bin 8192 Apr 18 10:23 cfs
drwxr-xr-x 2 bin bin 8192 Feb 13 2006 cfs.old
-rw------- 1 root root 9044 Jul 10 10:20 cmclconfig
-rw------- 1 root root 11344 Feb 13 2006 cmclconfig.old
-rw------- 1 root root 0 Jul 10 10:20 cmclconfig.tmp
-r-------- 1 bin bin 0 Mar 15 2004 cmknowncmds
-rw-r--r-- 1 root sys 25 Mar 29 2004 cmnodelist.1114
---------- 1 root root 0 Sep 28 2004 config.lck
dr-xr-xr-x 2 bin bin 96 Feb 13 2006 cvm
-r--r--r-- 1 root root 7887 Jan 4 2006 sha1.ascii
-rw-r--r-- 1 root sys 4162 Mar 29 2004 sha1.ascii.1114
-rw-r--r-- 1 root sys 7886 Sep 16 2004 sha1.ascii.1116
-r--r--r-- 1 root root 327 Sep 13 2004 sha1.ascii.bak
drwxr-xr-x 2 bin bin 96 Feb 13 2006 sharedtape
drwxr-xr-x 2 root sys 96 Mar 7 2005 testpkg


# cmviewcl

CLUSTER STATUS
sha1 up

NODE STATUS STATE
sha1 up running

PACKAGE STATUS STATE AUTO_RUN NODE
sambapkg up running enabled sha1
testpkg up running enabled sha1


Basically, look in the cntl files and see if you have any LV's defined:

LV[0]="/dev/vg01/lvora"; FS[0]="/app/oracle"


Rgds...Geoff

[TwoProc]

Hey Steve, what's with outposting me by 10 seconds??? :-)

[ConnieK]

Geoff,

I checked the ServiceGuard cluster control file and it has no reference to VG's or LV's. Just examples - still commented out as below:
# Example: If a package uses two JFS filesystems, pkg01a and pkg01b,
# which are mounted on LVM logical volumes lvol1 and lvol2 for read and
# write operation, you would enter the following:
# LV[0]=/dev/vg01/lvol1; FS[0]=/pkg01a; FS_MOUNT_OPT[0]="-o rw";
# FS_UMOUNT_OPT[0]=""; FS_FSCK_OPT[0]=""; FS_TYPE[0]="vxfs"
#
# LV[1]=/dev/vg01/lvol2; FS[1]=/pkg01b; FS_MOUNT_OPT[1]="-o rw"
# FS_UMOUNT_OPT[1]=""; FS_FSCK_OPT[1]=""; FS_TYPE[1]="vxfs"
#
#LV[0]=""; FS[0]=""; FS_MOUNT_OPT[0]=""; FS_UMOUNT_OPT[0]=""; FS_FSCK_OPT[0]=""
#FS_TYPE[0]=""
#

Do I need to do anything here? And, does it have anything to help me fix the problem I already have - the deleted character device file and how to put it right?

[A. Clay Stephenson]

First, before doing anything make sure that you shutdown the database.

The file length are listed in bytes (as opposed to KB, KiB, MB, or MiB) --- they are just 'B'.

Because you are using raw devices for your database, there are no LVOL related commands (unless you have come cooked files for control files or unless you are moving the Oracle binaries with your packages). If no "cooked" file (ie those within a filesystem) are in use by your package then all you will have are commands to activate the volume group. You actually had a ticking time bomb here because if the package shifted to another node, those "cooked" files under /dev/vg12, being on /, would not have moved with the package. At best, you would have the old data on the LVOL's before the rm was done (which would have actually left the data intact) and is thus very out of date --- an instantly corrupt database.

Finally, never, ever allow a DBA to have write permission on the /dev/vgxx directory itself. This will not impair in any way his ability to read and write to files within that directory.

[Geoff Wild]

Do I need to do anything here? And, does it have anything to help me fix the problem I already have - the deleted character device file and how to put it right?

Nope - it just proves that you are indeed using Raw logical volumes.

So, to me, it sounds like the DBA did something wrong or maybe there are no free disks in the vg for them to add another raw database file.

Rgds...Geoff

[ConnieK]

Well - so okay - the permissions to that directory are not correct.
drwx------ 2 oracle dba 8192 Aug 7 13:49 vg12

I will fix that issue after I fix this one. Regardless of who did this - the DBA actually admitted the mistake, but the previous SA should have not allowed the permissions in the first place.

Is my plan, as above, to fix /dev/vg12/rlvol30 correct?

Except for not including bringing the DB down?

[A. Clay Stephenson]

Yes, everything else is fine. My one (minor) quibble is that you are using /tmp for temporary file. /tmp should really only be used for system-related temporary files whereas /var/tmp is intended for user temporary files. However, suppose that you have a system crash while doing this. It's very common to have automatic cleanup scripts run by rc upon boot to clean /tmp and /var/tmp ---- so, if you have sufficient room for these files in a directory that will not be subject to possible automated cleanup, that would be my choice. When doing things like this, I tend to be very careful and not give Mr. Murphy a chance.

[Geoff Wild]

I don't think it is - someone correct me if I'm wrong but:

4. mknod rlvol30 b 64 0x0c0030
5. mknod rlvol30 c 64 0x0c0030
6. dd if=/tmp/space/rlvol30 of /dev/vg12/rlvol30 bs=64k

Shouldn't the steps be:

lvcreate -L -n lvol30 /dev/vg12

then the dd command?

Rgds...Geoff

[A. Clay Stephenson]

Not necessarily, Geoff. What I think happened was that the device node was rm'ed by that !@XXX%? DBA rather than lvremove'ed. In this case, doing an lvcreate will waste disk. That is why I asked Connie to do a vgdisplay -v because I think the LVOL's are actually there --- just not the device nodes. I also suspect that the /dev/vg12/lxxx block device nodes may still be there because that !@XXX%? DBA didn't know about them.

[ConnieK]

Clay,

DANKE!!! Your help is, and always has been invaluable!

Many thnaks to all of you - Geoff, for giving me insight on the cluster control file. Pete, for being right on target and getting the ball rolling. John, for your input that makes sense, but cannot be performed due to lack of a test server.

I hope this will help the next poor SA that gets involved in a similar situation is able to help from this Q&A.

Thanks again!

Connie

[ConnieK]

Update & more to this saga. I just HAD to reopen this thread to give an update!!

Firstly, I only have to create the character dev, the block device was there - Clay was correct.

Get this: Now the DBA wants me only to create the char device and then he wants to bring the DB up and test some things. I asked him about the data that I intend to save off. He said that instead of using the data from this server, he wanted me to copy the data from the other server in this cluster. (It appears to be correct)

I told him I wasn't sure that I can actually do a "dd" across the network - never having done it - but I didn't think I would be able to. Am I right?

So, now I have to come up with some other way to get the supposed uncorrupted data from the other server in this cluster to the corrected char device (rlvol) on the first server.

Any suggestions? And "smite the DBA" does not count!

And the saga continues......

I love my job, I love my job, I LOVE my job!

Connie

[A. Clay Stephenson]

That's ok. I just wish Geoff could spell Jeff.

[Geoff Wild]

Okay - if this is truly a cluster - then there should be only one location for the data. Period.

You don't copy data between nodes in a cluster.

All you do is create the files on one node, then create a map file of the vg, export in preview mode, then import on the alternate node(s).

On node with active vg12:

vgexport -s -v -p -m /tmp/vg12.map /dev/vg12

Then on alternate node(s).

copy the map file to /tmp
vgexport vg12
mkdir /dev/vg12
mknod /dev/vg12/group 64 0x0c0000

vgimport -s -v -m /tmp/vg12.map /dev/vg12

Rgds...Geoff

[TwoProc]

I've reviewed this section of Connie's steps once again today, and it bugs me:

4. mknod rlvol30 b 64 0x0c0030
5. mknod rlvol30 c 64 0x0c0030

shouldn't this be:

4. mknod lvol30 b 64 0x0c0030
5. mknod rlvol30 c 64 0x0c0030

???

I don't think you can make two special files (or any files) with the same name in the same directory - and, the character (c) devices have the "r" in the front - not the block devices.

Connie, how did you accomplish this portion and end up with a working solution?

Can you list the contents of /dev/vg12/ as it stands now?

[A. Clay Stephenson]

If these data are as described then you technically have what is referred to as "a mess". There is really no way that if this is one package consisting of one Oracle instance that the "other" data can be correct. The data is either activated on one node on the cluster or the other --- unless this is also an Oracle RAC. The problem is that you have (or at least had) those cooked files on one node and the raw files on the other. At least one of those attempts to start the database should be inconsistant.

The very first thing that I would check is if the sizes of those "cooked" files are no larger than the LVOL's which should house them. A vgdisplay -v compared to ls -l's will tell you that. If the "cooked" files are larger than their raw counterparts, then you will need to lvextend those LVOL's before doing a dd.

It is possible to dd across the network using a remsh pipeline; something like this:

dd if=/aaa/bbb/myfile bs=8k | remsh mickey dd bs=8k of=/dev/vg12/rlvol45

You should note that if this is one package then you shouldn't have to move the data at all but you may still need to move those cooked files back to their original LVOL's.

But before you do much of anything, you and the DBA need to sitdown and carefully discuss your plan and both of you need a better understanding of how SG works because otherwise it's very easy to get into a situation from which recovery, at best, would be extremely difficult. Before I did much of anything, I would get a cold backup and the trick will be deciding which of these (the one with the cooked files or the other) is "correct" at this point in time -- if either is.

[ConnieK]

Geoff - True cluster. One location for data - true. Just need to create three character device files on one node of the cluster - true. The other node has what looks like the correct device and block files.

Clay - you are correct - this is an Oracle RAC cluster.

Sizes as below:

ll /dev/vg12:

-rw------- 1 oracle dba 63963136 Feb 8 2007 rlvol30
63963136B = 62464KB = 61MB
-rw------- 1 oracle dba
235929600 Feb 8 2007 rlvol31
235929600B = 230400KB = 225MB
-rw------- 1 oracle dba 80740352 Feb 8 2007 rlvol39
80740352B = 78848KB = 77MB

vgdisplay -v vg12

LV Name /dev/vg12/lvol30
LV Status available/syncd
LV Size (Mbytes) 2048
Current LE 256
Allocated PE 256
Used PV 4

LV Name /dev/vg12/lvol31
LV Status available/syncd
LV Size (Mbytes) 2048
Current LE 256
Allocated PE 256
Used PV 4
LV Name /dev/vg12/lvol39
LV Status available/syncd
LV Size (Mbytes) 32768
Current LE 4096
Allocated PE 4096
Used PV 4

So, the "cooked" files are not larger than the lvols that are to house them.


Here is the same information from the other node:

ll /dev/vg12:

crw------- 1 oracle dba 64 0x0c001e Mar 26 2006 rlvol30
crw------- 1 oracle dba 64 0x0c001f Mar 26 2006 rlvol31

crw------- 1 oracle dba 64 0x0c0027 Mar 26 2006 rlvol39

vgdisplay -v vg12:

LV Name /dev/vg12/lvol30
LV Status available/syncd
LV Size (Mbytes) 2048
Current LE 256
Allocated PE 256
Used PV 4

LV Name /dev/vg12/lvol31
LV Status available/syncd
LV Size (Mbytes) 2048
Current LE 256
Allocated PE 256
Used PV 4
LV Name /dev/vg12/lvol39
LV Status available/syncd
LV Size (Mbytes) 32768
Current LE 4096
Allocated PE 4096
Used PV 4


The DBA now wants to bring up the database immediately after I re-create the character files. I think that he anticipates that the cluster will re-sync itself to have both nodes the same (pointers correct to the same location of the data).

So, I may not have to do the "dd" using remsh. However, if this does not work, I will have a backup tape ready to restore the data, plus I copied the data to an unmodifiable directory - as Clay's warning above stated.

Geoff's advice is to just perform a vgexport on the "correct" node, copy the map file on the "screwed-up" node, vgexport the "bad" vg (vg12), make a new node for vg12, and then vgimport the mapfile from the "correct" node.

If this is right - boy! that will easier than I thought!

Connie

[A. Clay Stephenson]

At this point (and I assume the database is down), I would do a vgdisplay -v /dev/vg12 on both nodes and compare them. If they are identical (except for different disk device paths for the physical volumes -- it would be rather unusual for those to be the same) then next do an ls -l /dev/vg12 on both nodes and compare them. If they are identical then you are ok. If the only differences are those device nodes then you can recreate the rlvolxx device nodes using mknod as discussed earlier. Although it would be perfectly ok to follow Geoff's advice and do a vgexport -p -s ... on the "good" node and a vgimport -s ... on the "bad" node. After doing the vgimport and activating the volume group, I would do a vgcfgbackup so that you can recreate failed disks/LUN's from either node. At this point, you should be ready to try to start the database.

[ConnieK]

Hey Geoff,

You said:


All you do is create the files on one node, then create a map file of the vg, export in preview mode, then import on the alternate node(s).

On node with active vg12:

vgexport -s -v -p -m /tmp/vg12.map /dev/vg12

Then on alternate node(s).

copy the map file to /tmp
vgexport vg12
mkdir /dev/vg12
mknod /dev/vg12/group 64 0x0c0000

vgimport -s -v -m /tmp/vg12.map /dev/vg12


Which adre the steps I intend to take, but when you state "copy the map file to /tmp" - which mapfile are you referring to?

Connie