topic Changing the nature of a file in Operating System - HP-UX

Changing the nature of a file

ConnieK — Tue, 07 Aug 2007 10:07:01 GMT

I have "inherited" 2 HPUX 11.23 rp4640 clustered servers running Oracle 10. The DBA needs to add a new database file to an existing VG. He states that his attempt errored out saying that the nature of the file was not a RAW device.

I would assume that the previous SA built the VG's and the LV's and ran "newfs" on the LV's - am I right? If so, how can I change this?

Here is what I see for the specific vg/lvol:

lvdisplay -v /dev/vg12:

LV Name /dev/vg12/lvol30
LV Status available/syncd
LV Size (Mbytes) 2048
Current LE 256
Allocated PE 256
Used PV 4

lvdisplay -v /dev/vg12/lvol30
--- Logical volumes ---
LV Name /dev/vg12/lvol30
VG Name /dev/vg12
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 2048
Current LE 256
Allocated PE 256
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation PVG-strict/distributed
IO Timeout (Seconds) default
--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c5t3d3 64 64
/dev/dsk/c5t3d4 64 64
/dev/dsk/c5t3d5 64 64
/dev/dsk/c5t3d6 64 64

/dev/vg12 > ll

crw------- 1 oracle dba 64 0x0c001d Mar 26 2006 rlvol29
crw------- 1 oracle dba 64 0x0c0003 Mar 26 2006 rlvol3
-rw------- 1 oracle dba 63963136 Feb 8 17:13 rlvol30
-rw------- 1 oracle dba 235929600 Feb 8 17:13 rlvol31
crw------- 1 oracle dba 64 0x0c0020 Mar 26 2006 rlvol32
crw------- 1 oracle dba 64 0x0c0021 Mar 26 2006 rlvol33
crw------- 1 oracle dba 64 0x0c0022 Mar 26 2006 rlvol34
crw------- 1 oracle dba 64 0x0c0023 Mar 26 2006 rlvol35
crw------- 1 oracle dba 64 0x0c0024 Mar 26 2006 rlvol36
crw------- 1 oracle dba 64 0x0c0025 Mar 26 2006 rlvol37
crw------- 1 oracle dba 64 0x0c0026 Mar 26 2006 rlvol38
-rw------- 1 oracle dba 80740352 Feb 8 17:17 rlvol39

As you can (maybe) see above in the LL output of /dev/vg12 indicates that the character device file (crw) was not created.

If the previous SA did a "newfs" after creating the lvols (30, 31 and 39) - how can I reverse these to be usable for the DBA's?

Please help me!

Thanks,

Connie

Re: Changing the nature of a file

Pete Randall — Tue, 07 Aug 2007 10:19:31 GMT

I'll say those aren't a RAW device. Thhey aren't device files at all, but regular files that someone has been writing into. Look at the sizes of rlvol30, 31, and 39. Is your / filesystem filling up by any chance?

You need to rm these files and re-create the proper device files.

Pete

Re: Changing the nature of a file

Geoff Wild — Tue, 07 Aug 2007 10:35:15 GMT

If you are indeed using "raw" lvols, then you do NOT want to newfs them. THat is, they need to stay RAW and not have a filesystem on them.

What is in the control file as far as vg's and LV's(if any) of the Servieguard package?

Rgds...Geoff

Re: Changing the nature of a file

A. Clay Stephenson — Tue, 07 Aug 2007 10:37:28 GMT

Pete has nailed it and because rlvol30-31 are regular (cooked) files in / and because 30-38 are device nodes, there is a very strong probability that someone (that XXXXX DBA) rm'ed the device node and then created new replacements which had the effect of replacing raw device files with cooked files. You need to do a vgdisplay -v and see if the number of LVOL's matches what you expect. You can recover this mess by cp'ing the regular files to a new location. Next rm these regular files and use mknod to recreate the block and character device nodes for each one. Finally use dd to copy the saved regular file back to the corresponding device node. To prevent this happening in the future, you need to remove write permission from oracle on the /dev/vg12
directory for user oracle. The directory should be owned by root but readable and searchable by oracle. The device nodes under this directory should be owned by oracle, group dba and have 600 or 660 mode. This configuration will allow oracle to read and write to any of the device nodes but not remove or create device nodes.

Re: Changing the nature of a file

TwoProc — Tue, 07 Aug 2007 10:37:38 GMT

Connie,
First issue: rlvol30, rlvol31 and rlvol39 are not actually lvols, they are just files named to to look like lvols. My guess is that a dba just added/created a file to a tablespace that didn't exist as an lvol at all.

See that they are "c..." devices? Notice that they don't have major numbers of 64, and a nice hexadecimal minor number? They are just files, not lvols.

I'm not sure how to stick the toothpaste back into the tube on this one.

Step one would be to clone this whole mess over to another server for testing.

Then you begin testing your scenario on the test server. You begin by doing a shutdown on your database. Make sure to backup the WHOLE database before making the next move. Next, backup to another filesystem for handy use in the next steps rlvols 30, 31, and 39. Delete the three rlvols 30,31, and 39 from /dev/vg12/lvol30, 31, and 39. Then actually create those lvols from the real vg, vg12, making sure tha that the lvols are large enough to at least hold the size of the data in the files that you see. DO NOT run mkfs (these are raw, and by that term - they don't have a file system, so you DON'T want to to put one on there. Then "dd" the file contents onto the drive. An example for rlvol30 would be:

dd if=/backuplocation/rlvol30 of=/dev/vg12/rlvol30 bs=64k

Then bring up your up database.
Now run "dbv" on each of the lvols to make sure all is well. Afterward run some programs known to use data from those datafiles, hold breath. Run a whole set of stuff, say a whole night run of production work against your database, lots and lots of processing.

If all is well, you're probably OK. If not restore EVERYTHING from your backup.

Also, if the three files are type "temp", then in you're in luck, then all you have to do is delete those three temp files, and recreate on the lvols as they are supposed to be. Ditto for undo (rollback) files.

Re: Changing the nature of a file

TwoProc — Tue, 07 Aug 2007 11:38:30 GMT

re: my comment above on "temp" and "undo" spaces. Of course, what I mean by "delete the files" I mean go through the standard procedure of taking that stuff offline, etc, etc. and recreating - not just using the Unix "rm" command to delete the file.

Re: Changing the nature of a file

ConnieK — Tue, 07 Aug 2007 13:58:48 GMT

Geoff,

I am not sure how to get into the SG control file to see what's in there. I'm a newbie at clustering and SG. I looked at the cmcluster.conf file and really sam nothing related to VG's & LV's in there as below:

SGCONF=/etc/cmcluster
SGSBIN=/usr/sbin
SGLBIN=/usr/lbin
SGLIB=/usr/lib
SGRUN=/var/adm/cmcluster
SGAUTOSTART=/etc/rc.config.d/cmcluster
SGFFLOC=/opt/cmcluster/cmff
CMSNMPD_LOG_FILE=/var/adm/SGsnmpsuba.log
CMGMSD_LOG_FILE=/var/adm/cmcluster/cmgmsd/cmgmsd.log

John,

I cannot re-create this scenario as I do not have an available test server, but I do appreciate your input - it helped too!

Pete & Clay - your valuable input has helped me understand what I need to do. Here's my plan to fix rlvol30: (the others will be done later)

1. Get a tape back up of rlvol30.
2. cp /dev/vg12/rlvol30 /tmp/space/rlvol30.
By the way, is this figure in KB or Bytes?
-rw------- 1 oracle dba 63963136 Feb 8 17:13 rlvol30
3. rm /dev/vg12/rlvol30
4. mknod rlvol30 b 64 0x0c0030
5. mknod rlvol30 c 64 0x0c0030
5. dd if=/tmp/space/rlvol30 of /dev/vg12/rlvol30 bs=64k

Does this look right?

Re: Changing the nature of a file

Geoff Wild — Tue, 07 Aug 2007 14:15:55 GMT

If you cd /etc/cmcluster and do a ll you should see a directory for each package that is configured.

cmviewcl
will show you which packages are running or not.

cd into that dir, there choud be a packagename.conf and a packagename.cntl

Example, on my test cluster:

root@sha1 [ /etc/cmcluster ]
# ll
total 224
drwxr-xr-x 2 root sys 8192 Oct 7 2005 MISC
drwxr-xr-x 2 root sys 8192 Sep 15 2004 ORACLE
drwxr-xr-x 2 root sys 8192 Nov 22 2005 SAMBA
drwxr-xr-x 2 bin bin 8192 Apr 18 10:23 cfs
drwxr-xr-x 2 bin bin 8192 Feb 13 2006 cfs.old
-rw------- 1 root root 9044 Jul 10 10:20 cmclconfig
-rw------- 1 root root 11344 Feb 13 2006 cmclconfig.old
-rw------- 1 root root 0 Jul 10 10:20 cmclconfig.tmp
-r-------- 1 bin bin 0 Mar 15 2004 cmknowncmds
-rw-r--r-- 1 root sys 25 Mar 29 2004 cmnodelist.1114
---------- 1 root root 0 Sep 28 2004 config.lck
dr-xr-xr-x 2 bin bin 96 Feb 13 2006 cvm
-r--r--r-- 1 root root 7887 Jan 4 2006 sha1.ascii
-rw-r--r-- 1 root sys 4162 Mar 29 2004 sha1.ascii.1114
-rw-r--r-- 1 root sys 7886 Sep 16 2004 sha1.ascii.1116
-r--r--r-- 1 root root 327 Sep 13 2004 sha1.ascii.bak
drwxr-xr-x 2 bin bin 96 Feb 13 2006 sharedtape
drwxr-xr-x 2 root sys 96 Mar 7 2005 testpkg

# cmviewcl

CLUSTER STATUS
sha1 up

NODE STATUS STATE
sha1 up running

PACKAGE STATUS STATE AUTO_RUN NODE
sambapkg up running enabled sha1
testpkg up running enabled sha1

Basically, look in the cntl files and see if you have any LV's defined:

LV[0]="/dev/vg01/lvora"; FS[0]="/app/oracle"

Rgds...Geoff

Re: Changing the nature of a file

TwoProc — Tue, 07 Aug 2007 14:19:19 GMT

Hey Steve, what's with outposting me by 10 seconds??? :-)

Re: Changing the nature of a file

ConnieK — Tue, 07 Aug 2007 14:30:52 GMT

Geoff,

I checked the ServiceGuard cluster control file and it has no reference to VG's or LV's. Just examples - still commented out as below:
# Example: If a package uses two JFS filesystems, pkg01a and pkg01b,
# which are mounted on LVM logical volumes lvol1 and lvol2 for read and
# write operation, you would enter the following:
# LV[0]=/dev/vg01/lvol1; FS[0]=/pkg01a; FS_MOUNT_OPT[0]="-o rw";
# FS_UMOUNT_OPT[0]=""; FS_FSCK_OPT[0]=""; FS_TYPE[0]="vxfs"
#
# LV[1]=/dev/vg01/lvol2; FS[1]=/pkg01b; FS_MOUNT_OPT[1]="-o rw"
# FS_UMOUNT_OPT[1]=""; FS_FSCK_OPT[1]=""; FS_TYPE[1]="vxfs"
#
#LV[0]=""; FS[0]=""; FS_MOUNT_OPT[0]=""; FS_UMOUNT_OPT[0]=""; FS_FSCK_OPT[0]=""
#FS_TYPE[0]=""
#

Do I need to do anything here? And, does it have anything to help me fix the problem I already have - the deleted character device file and how to put it right?

Re: Changing the nature of a file

A. Clay Stephenson — Tue, 07 Aug 2007 14:43:16 GMT

First, before doing anything make sure that you shutdown the database.

The file length are listed in bytes (as opposed to KB, KiB, MB, or MiB) --- they are just 'B'.

Because you are using raw devices for your database, there are no LVOL related commands (unless you have come cooked files for control files or unless you are moving the Oracle binaries with your packages). If no "cooked" file (ie those within a filesystem) are in use by your package then all you will have are commands to activate the volume group. You actually had a ticking time bomb here because if the package shifted to another node, those "cooked" files under /dev/vg12, being on /, would not have moved with the package. At best, you would have the old data on the LVOL's before the rm was done (which would have actually left the data intact) and is thus very out of date --- an instantly corrupt database.

Finally, never, ever allow a DBA to have write permission on the /dev/vgxx directory itself. This will not impair in any way his ability to read and write to files within that directory.

Re: Changing the nature of a file

Geoff Wild — Tue, 07 Aug 2007 14:49:06 GMT

Do I need to do anything here? And, does it have anything to help me fix the problem I already have - the deleted character device file and how to put it right?

Nope - it just proves that you are indeed using Raw logical volumes.

So, to me, it sounds like the DBA did something wrong or maybe there are no free disks in the vg for them to add another raw database file.

Rgds...Geoff

Re: Changing the nature of a file

ConnieK — Tue, 07 Aug 2007 15:01:56 GMT

Well - so okay - the permissions to that directory are not correct.
drwx------ 2 oracle dba 8192 Aug 7 13:49 vg12

I will fix that issue after I fix this one. Regardless of who did this - the DBA actually admitted the mistake, but the previous SA should have not allowed the permissions in the first place.

Is my plan, as above, to fix /dev/vg12/rlvol30 correct?

Except for not including bringing the DB down?

Re: Changing the nature of a file

A. Clay Stephenson — Tue, 07 Aug 2007 15:11:26 GMT

Yes, everything else is fine. My one (minor) quibble is that you are using /tmp for temporary file. /tmp should really only be used for system-related temporary files whereas /var/tmp is intended for user temporary files. However, suppose that you have a system crash while doing this. It's very common to have automatic cleanup scripts run by rc upon boot to clean /tmp and /var/tmp ---- so, if you have sufficient room for these files in a directory that will not be subject to possible automated cleanup, that would be my choice. When doing things like this, I tend to be very careful and not give Mr. Murphy a chance.

Re: Changing the nature of a file

Geoff Wild — Tue, 07 Aug 2007 15:18:53 GMT

I don't think it is - someone correct me if I'm wrong but:

4. mknod rlvol30 b 64 0x0c0030
5. mknod rlvol30 c 64 0x0c0030
6. dd if=/tmp/space/rlvol30 of /dev/vg12/rlvol30 bs=64k

Shouldn't the steps be:

lvcreate -L -n lvol30 /dev/vg12

then the dd command?

Rgds...Geoff

Re: Changing the nature of a file

A. Clay Stephenson — Tue, 07 Aug 2007 15:28:47 GMT

Not necessarily, Geoff. What I think happened was that the device node was rm'ed by that !@XXX%? DBA rather than lvremove'ed. In this case, doing an lvcreate will waste disk. That is why I asked Connie to do a vgdisplay -v because I think the LVOL's are actually there --- just not the device nodes. I also suspect that the /dev/vg12/lxxx block device nodes may still be there because that !@XXX%? DBA didn't know about them.

Re: Changing the nature of a file

ConnieK — Tue, 07 Aug 2007 15:32:14 GMT

Clay,

DANKE!!! Your help is, and always has been invaluable!

Many thnaks to all of you - Geoff, for giving me insight on the cluster control file. Pete, for being right on target and getting the ball rolling. John, for your input that makes sense, but cannot be performed due to lack of a test server.

I hope this will help the next poor SA that gets involved in a similar situation is able to help from this Q&A.

Thanks again!

Connie

Re: Changing the nature of a file

ConnieK — Tue, 07 Aug 2007 15:32:59 GMT

Update & more to this saga. I just HAD to reopen this thread to give an update!!

Firstly, I only have to create the character dev, the block device was there - Clay was correct.

Get this: Now the DBA wants me only to create the char device and then he wants to bring the DB up and test some things. I asked him about the data that I intend to save off. He said that instead of using the data from this server, he wanted me to copy the data from the other server in this cluster. (It appears to be correct)

I told him I wasn't sure that I can actually do a "dd" across the network - never having done it - but I didn't think I would be able to. Am I right?

So, now I have to come up with some other way to get the supposed uncorrupted data from the other server in this cluster to the corrected char device (rlvol) on the first server.

Any suggestions? And "smite the DBA" does not count!

And the saga continues......

I love my job, I love my job, I LOVE my job!

Connie

Re: Changing the nature of a file

A. Clay Stephenson — Tue, 07 Aug 2007 15:38:49 GMT

That's ok. I just wish Geoff could spell Jeff.

Re: Changing the nature of a file

Geoff Wild — Wed, 08 Aug 2007 13:26:39 GMT

Okay - if this is truly a cluster - then there should be only one location for the data. Period.

You don't copy data between nodes in a cluster.

All you do is create the files on one node, then create a map file of the vg, export in preview mode, then import on the alternate node(s).

On node with active vg12:

vgexport -s -v -p -m /tmp/vg12.map /dev/vg12

Then on alternate node(s).

copy the map file to /tmp
vgexport vg12
mkdir /dev/vg12
mknod /dev/vg12/group 64 0x0c0000

vgimport -s -v -m /tmp/vg12.map /dev/vg12

Rgds...Geoff