Changing the nature of a file

[TwoProc]

I've reviewed this section of Connie's steps once again today, and it bugs me:

4. mknod rlvol30 b 64 0x0c0030
5. mknod rlvol30 c 64 0x0c0030

shouldn't this be:

4. mknod lvol30 b 64 0x0c0030
5. mknod rlvol30 c 64 0x0c0030

???

I don't think you can make two special files (or any files) with the same name in the same directory - and, the character (c) devices have the "r" in the front - not the block devices.

Connie, how did you accomplish this portion and end up with a working solution?

Can you list the contents of /dev/vg12/ as it stands now?

[A. Clay Stephenson]

If these data are as described then you technically have what is referred to as "a mess". There is really no way that if this is one package consisting of one Oracle instance that the "other" data can be correct. The data is either activated on one node on the cluster or the other --- unless this is also an Oracle RAC. The problem is that you have (or at least had) those cooked files on one node and the raw files on the other. At least one of those attempts to start the database should be inconsistant.

The very first thing that I would check is if the sizes of those "cooked" files are no larger than the LVOL's which should house them. A vgdisplay -v compared to ls -l's will tell you that. If the "cooked" files are larger than their raw counterparts, then you will need to lvextend those LVOL's before doing a dd.

It is possible to dd across the network using a remsh pipeline; something like this:

dd if=/aaa/bbb/myfile bs=8k | remsh mickey dd bs=8k of=/dev/vg12/rlvol45

You should note that if this is one package then you shouldn't have to move the data at all but you may still need to move those cooked files back to their original LVOL's.

But before you do much of anything, you and the DBA need to sitdown and carefully discuss your plan and both of you need a better understanding of how SG works because otherwise it's very easy to get into a situation from which recovery, at best, would be extremely difficult. Before I did much of anything, I would get a cold backup and the trick will be deciding which of these (the one with the cooked files or the other) is "correct" at this point in time -- if either is.

[ConnieK]

Geoff - True cluster. One location for data - true. Just need to create three character device files on one node of the cluster - true. The other node has what looks like the correct device and block files.

Clay - you are correct - this is an Oracle RAC cluster.

Sizes as below:

ll /dev/vg12:

-rw------- 1 oracle dba 63963136 Feb 8 2007 rlvol30
63963136B = 62464KB = 61MB
-rw------- 1 oracle dba
235929600 Feb 8 2007 rlvol31
235929600B = 230400KB = 225MB
-rw------- 1 oracle dba 80740352 Feb 8 2007 rlvol39
80740352B = 78848KB = 77MB

vgdisplay -v vg12

LV Name /dev/vg12/lvol30
LV Status available/syncd
LV Size (Mbytes) 2048
Current LE 256
Allocated PE 256
Used PV 4

LV Name /dev/vg12/lvol31
LV Status available/syncd
LV Size (Mbytes) 2048
Current LE 256
Allocated PE 256
Used PV 4
LV Name /dev/vg12/lvol39
LV Status available/syncd
LV Size (Mbytes) 32768
Current LE 4096
Allocated PE 4096
Used PV 4

So, the "cooked" files are not larger than the lvols that are to house them.


Here is the same information from the other node:

ll /dev/vg12:

crw------- 1 oracle dba 64 0x0c001e Mar 26 2006 rlvol30
crw------- 1 oracle dba 64 0x0c001f Mar 26 2006 rlvol31

crw------- 1 oracle dba 64 0x0c0027 Mar 26 2006 rlvol39

vgdisplay -v vg12:

LV Name /dev/vg12/lvol30
LV Status available/syncd
LV Size (Mbytes) 2048
Current LE 256
Allocated PE 256
Used PV 4

LV Name /dev/vg12/lvol31
LV Status available/syncd
LV Size (Mbytes) 2048
Current LE 256
Allocated PE 256
Used PV 4
LV Name /dev/vg12/lvol39
LV Status available/syncd
LV Size (Mbytes) 32768
Current LE 4096
Allocated PE 4096
Used PV 4


The DBA now wants to bring up the database immediately after I re-create the character files. I think that he anticipates that the cluster will re-sync itself to have both nodes the same (pointers correct to the same location of the data).

So, I may not have to do the "dd" using remsh. However, if this does not work, I will have a backup tape ready to restore the data, plus I copied the data to an unmodifiable directory - as Clay's warning above stated.

Geoff's advice is to just perform a vgexport on the "correct" node, copy the map file on the "screwed-up" node, vgexport the "bad" vg (vg12), make a new node for vg12, and then vgimport the mapfile from the "correct" node.

If this is right - boy! that will easier than I thought!

Connie

[A. Clay Stephenson]

At this point (and I assume the database is down), I would do a vgdisplay -v /dev/vg12 on both nodes and compare them. If they are identical (except for different disk device paths for the physical volumes -- it would be rather unusual for those to be the same) then next do an ls -l /dev/vg12 on both nodes and compare them. If they are identical then you are ok. If the only differences are those device nodes then you can recreate the rlvolxx device nodes using mknod as discussed earlier. Although it would be perfectly ok to follow Geoff's advice and do a vgexport -p -s ... on the "good" node and a vgimport -s ... on the "bad" node. After doing the vgimport and activating the volume group, I would do a vgcfgbackup so that you can recreate failed disks/LUN's from either node. At this point, you should be ready to try to start the database.

[ConnieK]

Hey Geoff,

You said:


All you do is create the files on one node, then create a map file of the vg, export in preview mode, then import on the alternate node(s).

On node with active vg12:

vgexport -s -v -p -m /tmp/vg12.map /dev/vg12

Then on alternate node(s).

copy the map file to /tmp
vgexport vg12
mkdir /dev/vg12
mknod /dev/vg12/group 64 0x0c0000

vgimport -s -v -m /tmp/vg12.map /dev/vg12


Which adre the steps I intend to take, but when you state "copy the map file to /tmp" - which mapfile are you referring to?

Connie

[A. Clay Stephenson]

The map file is produced by the vgexport command.

vgexport -s -v -p -m /tmp/vg12.map /dev/vg12

In this case two things are happening:
1) -p this is a preview so it creates the map file specified by the -m option but doesn't actually export the VG
2) -s --- also write the VGID into the mapfile.

Normally the mapfile just contains the LVOL names, lvol1, lvol2, lmickey, ldonald and if you use the standard names lvol1,lvol2, ... lvolN , the mapfile is not even needed but the -s adds the VGIS to the mapfile. This is used when you don't know the pathnames to the PV's. In SG Clusters, it's an easy way to match PV's to a common VG.

When you import using -s -m mapfile then all disks are searched looking for that matching VGID. If you know the paths to the disks/LUN's (which are rarely the same on different nodes in the cluster) then the use of the -s vgexport/vgimport is optional and you can actually save time on a box with many disks by not using -s.

[A. Clay Stephenson]

By the way (and this is not intended to question your abilities), it might be a good idea, if you have a spare LUN/disk or two that is accessible from both hosts, to create a play VG from scratch so that you can get very comfortable with building VG's from scratch; adding LVOL's; and then exporting and importing them. This will give you the confidence to tackle the production VG.

[ConnieK]

Well Clay! I guess you and Geoff work together!

I understand now. I should have know this before now. I am getting ready to do this fix in about an hour and I'm still a little nervous because this is a production cluster! Both nodes are active with a cluster lock disk connection. So, here are my steps:

1. cmhaltcl - which brings down the Oracle RAC
2. cmviewcl - (to make sure the cluster is down)
3. vgexport -s -v -p -m /tmp/vg12.map /dev/vg12 - do on "good" node.
4. create the char devices <-- on bad node
mknod rlvol30 c 64 0x0c001e, etc. (There are 3 that are messed up)
5. vgexport -s -v -p -m /tmp/vg12.map /dev/vg12 <- on bad node
6. vgexport vg12 <-- on bad node
7. mkdir vg12 <--- on bad node
8. mknod /dev/vg12/group 64 0x0c0000 <-- on bad node
9. vgimport -s -v -m /tmp/vg12.map /dev/vg12 <--- on bad node
10. cmruncl
11. cmviewcl <--- on both nodes
12. Call Id10t DBA to see if his DB is ok.

What do you think?

Connie

[ConnieK]

Clay - I did not take offense. I question my own abilites! It's just that I have never HAD to do this - fix a char device. I have build vg's and lvols many times in the past, but I've never seen this mess. Just wanted to make absolutely sure I'm doing the right thing. Want to keep my job for a while longer! :-)

[A. Clay Stephenson]

First, you don't want to halt the cluster (cmhaltcl); instead you want to halt the package using cmhaltpkg (I assume that this is all one package). You restart via cmrunpkg. There are man pages for all of these.

Understand that if all that is wrong is that you are missing those /dev/vg12/rlxxxx device nodes on one nodes but vgdisplay -v /dev/vg12 otherwise displays the LVOL's on both nodes then there is no need to do a vgexport -p / vgimport because all that is going to happen is that those missing device nodes will be recreated; you can simply do the mknod's for the missing device nodes and that will be equivalent. Note that vgexport/vgimport writes nothing on the LVM disks themselves. Vgimport writes to /etc/lvmtab and create those device nodes under a pre-existing directory (/dev/vg12) that has a pre-existing group file.

My main concern remians the discrepancy between the cooked files and the raw files. One of those has to be wrong and I don't think it's clear which one is. This is really a DBA task to decide.

I'll try to make this clear to you. Oracle RAC works by allowing all servers in the cluster to read and write to the same disks. It can't use "cooked" files because Server A wouldn't have a clue about server B's buffer cache. (There is an exception called Cluster File System which does allow all hosts to use cooked files just as though they were local vxfs filesystems but that is not germane to your situation.) Now you have 3 files that are cooked files on one node and raw files on the other. Let's pretend that the first few bytes of the cooked file rlvol30 contain the word "Dog". Let's also preten that the raw rlvol30 file contains "Cat" in the same position. Which one is correct? Only one can be because there is supposed to be only one set of physical files in a RAC though they may be accessed by all nodes in the cluster.

If your DBA decides that the cooked files are the accurate version then you need to dd those cooked files to the raw location BUT note that once you do that, if you guessed wrong then your database is toast. If this were me, I would dd the raw data to another set of cooked files before I did anything else. That way you have a Plan B if Plan A was wrong.