Whoops. Partially deleted /dev

[George Lenzer]

I was working with LVM last night and was trying to delete /dev/oradata10g with:

rm -rf /dev/oradata10g

My intention was to wipe out the directory and the group file. A very badly placed space character made my life difficult:

rm -rf /dev /oradata10g

It should have only taken a fraction of a second, but it seemed to be taking more than a second and I pressed Ctrl-C sparing the rest of /dev. Still, the damage was done. Hoping that not all was lost I ran the vgimport command and got an error about /dev/config not being present. Then I tried logging into the console. When I typed any user name, I wouldn't get a password prompt and would only get a "login incorrect" message. At this point I knew I was hosed. As luck would have it, this is one system that hasn't had the system disk backup done in a while. Yes... EVERYTHING that happened here was wrong. So is there any possible way to recover /dev without reinstalling?

[Mel Burslan · Accepted solution]

You can try insf -a command to create (install) device special files for whatever you have on your system.

[Torsten.]

Did you know you should use vgexport or vgremove to delete VGs?

However, I would consider to restore from backup - who knows what is missing now ...

[James R. Ferguson]

Hi George:

Ouch. It sounds like you don't have an active 'root' login and that limits what you could try ('insf' as Mel noted). As for attempting to restore, you may be missing some of the '/dev' files for tape, too, and that would require a 'root' login.

The most expedient recovery may be from an Ignite backup tape or Ignite network archive from which you can boot and re-install.

Regards!

...JRF...

[George Lenzer]

I've opened a case with HP as well, but I suspect that at the end of the day, I'll be doing a full OS install. All of the responses have been quite helpful, so thanks to everyone.

To answer Torsten's question, yes, I did use vgexport to remove that VG. But for some reason /dev/oradata10g and it's contents were still left behind and I can't stand to leave things like that around. That happens frequently enough that I usually delete any old volume group directories and group files. The lvols are usually gone, but sometime the dirs and group files stay.

While looking around using the install DVD in export recovery mode, I've found that /dev/vg00 and it's contents are all gone. I suspect that's the final nail in the coffin. The one thing I need to do now is see if I can recover some custom scripts from /.

[George Lenzer]

Well it also looks like /dev/dsk and /dev/rdsk are all gone too. ! I'm quite happy this is not a production system. With that said, here is what I do have access to in expert recovery mode in a chroot environment:

vgscan
mknod
ioscan (all disks are seen)
insf (a limited version it seems since insf -a didn't work)

Of course, as noted earlier, there is no telling what is missing. But, is there any possibility that if I can get this system to boot up most of the way that I can pull some of /dev over from one of our other Itanium boxes?

[Mark S Meadows]

Hi George,

You should be able to recreate /dev/vg00 and the missing standard device files lvol1, lvol2, lvol3, lvol4, etc.....

umask 077
for LV in 1 2 3 4 5 6 7 8
do
mknod /dev/vg00/lvol$LV b 64 0x00000$LV
mknod /dev/vg00/rlvol$LV c 64 0x00000$LV
done

Example listing :

# ls -ld /dev/vg00
drwxr-xr-x 2 root root 8192 Jul 23 08:30 /dev/vg00

/dev/vg00# ll lvol*
brw-r----- 1 root sys 64 0x000001 Jul 23 08:30 lvol1
brw-r----- 1 root sys 64 0x000002 Jul 23 08:30 lvol2
brw-r----- 1 root sys 64 0x000003 Jul 23 08:30 lvol3
brw-r----- 1 root sys 64 0x000004 Jul 23 08:30 lvol4
brw-r----- 1 root sys 64 0x000005 Jul 23 08:30 lvol5
brw-r----- 1 root sys 64 0x000006 Jul 23 08:30 lvol6
brw-r----- 1 root sys 64 0x000007 Jul 23 08:30 lvol7
brw-r----- 1 root sys 64 0x000008 Jul 23 08:30 lvol8

/dev/vg00# ll rlvol*
crw-r----- 1 root sys 64 0x000001 Jul 23 08:30 rlvol1
crw-r----- 1 root sys 64 0x000002 Jul 23 08:30 rlvol2
crw-r----- 1 root sys 64 0x000003 Jul 23 08:30 rlvol3
crw-r----- 1 root sys 64 0x000004 Jul 23 08:30 rlvol4
crw-r----- 1 root sys 64 0x000005 Jul 23 08:30 rlvol5
crw-r----- 1 root sys 64 0x000006 Jul 23 08:30 rlvol6
crw-r----- 1 root sys 64 0x000007 Jul 23 08:30 rlvol7
crw-r----- 1 root sys 64 0x000008 Jul 23 08:30 rlvol8
crw-r----- 1 root sys 64 0x000009 Jul 23 08:30 rlvol9

Obviously you have the other missing /dev device files and directory's to contend with.

Regards,

Mark

[George Lenzer]

Well... I was able to use insf -e to recreate most of the standard dev nodes. Then I used vgscan and mknod to get vg00 back. I rebooted, cleared up the crash dump fallout from previous failed boots, and was able to log in as root.

I absolutely know there is no telling what is missing or wrong until I do a full comparison between the /dev file systems on this box and one of our other Itaniums. I need to get networking going. Config seems fine according to set_parms, but the system can't ping itself or the gateway. syslog doesn't seem to say anything but during boot, configuration of IPv4 interfaces fails.

Once again, I have to say I was really happy this was not a production box.

[Mel Burslan]

If you haven't heard this one yet:

There are two types of system administrators

(1) who lost critical data

(2) who will lose critical data


I can not emphasize value of periodic and frequent ignite imaging process (we try to do it weekly with make_net_recovery utility in my data center) as well as a good daily backup with a preferably monthly restore exercise even if you do not need anything to be restored. A backup is only good if you can restore from it :)

[George Lenzer]

Mel, thanks for the reminder. Fortunately we do keep good backups on the production systems. But the reporting and test systems haven't gotten that much attention. I intend to change that after last night.