Changing the nature of a file

[A. Clay Stephenson]

The map file is produced by the vgexport command.

vgexport -s -v -p -m /tmp/vg12.map /dev/vg12

In this case two things are happening:
1) -p this is a preview so it creates the map file specified by the -m option but doesn't actually export the VG
2) -s --- also write the VGID into the mapfile.

Normally the mapfile just contains the LVOL names, lvol1, lvol2, lmickey, ldonald and if you use the standard names lvol1,lvol2, ... lvolN , the mapfile is not even needed but the -s adds the VGIS to the mapfile. This is used when you don't know the pathnames to the PV's. In SG Clusters, it's an easy way to match PV's to a common VG.

When you import using -s -m mapfile then all disks are searched looking for that matching VGID. If you know the paths to the disks/LUN's (which are rarely the same on different nodes in the cluster) then the use of the -s vgexport/vgimport is optional and you can actually save time on a box with many disks by not using -s.

[A. Clay Stephenson]

By the way (and this is not intended to question your abilities), it might be a good idea, if you have a spare LUN/disk or two that is accessible from both hosts, to create a play VG from scratch so that you can get very comfortable with building VG's from scratch; adding LVOL's; and then exporting and importing them. This will give you the confidence to tackle the production VG.

[ConnieK]

Well Clay! I guess you and Geoff work together!

I understand now. I should have know this before now. I am getting ready to do this fix in about an hour and I'm still a little nervous because this is a production cluster! Both nodes are active with a cluster lock disk connection. So, here are my steps:

1. cmhaltcl - which brings down the Oracle RAC
2. cmviewcl - (to make sure the cluster is down)
3. vgexport -s -v -p -m /tmp/vg12.map /dev/vg12 - do on "good" node.
4. create the char devices <-- on bad node
mknod rlvol30 c 64 0x0c001e, etc. (There are 3 that are messed up)
5. vgexport -s -v -p -m /tmp/vg12.map /dev/vg12 <- on bad node
6. vgexport vg12 <-- on bad node
7. mkdir vg12 <--- on bad node
8. mknod /dev/vg12/group 64 0x0c0000 <-- on bad node
9. vgimport -s -v -m /tmp/vg12.map /dev/vg12 <--- on bad node
10. cmruncl
11. cmviewcl <--- on both nodes
12. Call Id10t DBA to see if his DB is ok.

What do you think?

Connie

[ConnieK]

Clay - I did not take offense. I question my own abilites! It's just that I have never HAD to do this - fix a char device. I have build vg's and lvols many times in the past, but I've never seen this mess. Just wanted to make absolutely sure I'm doing the right thing. Want to keep my job for a while longer! :-)

[A. Clay Stephenson]

First, you don't want to halt the cluster (cmhaltcl); instead you want to halt the package using cmhaltpkg (I assume that this is all one package). You restart via cmrunpkg. There are man pages for all of these.

Understand that if all that is wrong is that you are missing those /dev/vg12/rlxxxx device nodes on one nodes but vgdisplay -v /dev/vg12 otherwise displays the LVOL's on both nodes then there is no need to do a vgexport -p / vgimport because all that is going to happen is that those missing device nodes will be recreated; you can simply do the mknod's for the missing device nodes and that will be equivalent. Note that vgexport/vgimport writes nothing on the LVM disks themselves. Vgimport writes to /etc/lvmtab and create those device nodes under a pre-existing directory (/dev/vg12) that has a pre-existing group file.

My main concern remians the discrepancy between the cooked files and the raw files. One of those has to be wrong and I don't think it's clear which one is. This is really a DBA task to decide.

I'll try to make this clear to you. Oracle RAC works by allowing all servers in the cluster to read and write to the same disks. It can't use "cooked" files because Server A wouldn't have a clue about server B's buffer cache. (There is an exception called Cluster File System which does allow all hosts to use cooked files just as though they were local vxfs filesystems but that is not germane to your situation.) Now you have 3 files that are cooked files on one node and raw files on the other. Let's pretend that the first few bytes of the cooked file rlvol30 contain the word "Dog". Let's also preten that the raw rlvol30 file contains "Cat" in the same position. Which one is correct? Only one can be because there is supposed to be only one set of physical files in a RAC though they may be accessed by all nodes in the cluster.

If your DBA decides that the cooked files are the accurate version then you need to dd those cooked files to the raw location BUT note that once you do that, if you guessed wrong then your database is toast. If this were me, I would dd the raw data to another set of cooked files before I did anything else. That way you have a Plan B if Plan A was wrong.

[ConnieK]

Ok all: Here is exactly what I did to fix this problem:

1. cmhaltpkg
2. cmviewcl <-- to make sure pkg was down on both nodes.
3. ps -ef | grep ora <--- to make sure Oracle was down
4. vgexport -s -v -p -m /tmp/vg12.map /dev/vg12 <-- on the "bad" node
5. cp the messed up rlvols to /tmp
6. rm rlvols 30, 31, & 39 <-- the messed up ones
7. mknod rlvol30, 31, & 39 c 64 0x0c00xx
8. vgexport vg12
9. mkdir /dev/vg12
10. mknod /dev/vg12/group c 64 0x0c0000
11. vgimport -s -v -m /tmp/vg12.map /dev/vg12
12. chmod 755 /dev/vg12
13. chown oracle:dba /dev/vg12/rlvol*
14. vgcfgbackup /dev/vg12
15. cmrunpkg
16. ps -ef | grep ora
17. cmviewcl
18. strings /etc/lvmtab
19. Call DBA to check.

So far everything looks good, except that I have to restore one archive file.

Just letting everyone know that your help has been a blessing. And I WILL get better with practice and try to get as knowledgable as I can.

Thanks again!

Connie