someone created a /var/adm filesystem

[someone_4]

hello,
I have a developer was filling up /var .. so they decided to create a /var/adm file system. So now everything in /var/adm is gone. They did it though sam.

How can I get it back ?
And what will break.

Thanks

Richard

[someone_4]

I also dont have backup for this server.

Richard

[A. Clay Stephenson]

I would have a very serious talk with this clown and an even more serious talk with the clown that gave the clown root permission. Probably your most serious issue is that /var/adm/sw is gone which means that your installed software database is gone. As to how you get it back? Can you say, "restore from backup"?

P.S. You also lost your syslogs, shutdownlogs, sulogs, btmp, etc. This was "state of the art" dumb.

[S.K. Chan]

If you do not have it in backup, then it's just not going to be possible to manually recreate what's in /var/adm. Most of the log files yes but not what's in "sw", at least not that I know of. It's possible to fix individual index fileset but not the whole directory structure.

[someone_4]

Hello Mr. Clay ,

Thanks for your blunt and always answer. As far as who gave them root access .. the question really is why root access was never taken away. As much as I have tried and tried and preached and preached about it .. i am talking to deaf ears. ONE DAY something will happened and everything will start to listen.

Anyways..
I didnt know what to do here so I just deleted the lvol they created. And to my surpise everything was there.
All the files .. everything was there.

Now the question is why did this work?

Richard

[A. Clay Stephenson · Accepted solution]

Hold on Richard, Things might not be so bad. If a /var/adm mountpoint was created then you are ok. I would immediately shutdown and boot up single user. Next mount /var, /usr, and (maybe) /tmp. Everything under the old /var/adm should now be visible. You should now mount the new /var/adm under a new mountpoint, e.g. /newadm. I would delete everything in /newadm and then copy everything in /var/adm to /newadm. Use cpio -pudvm so that permissions and ownerships are preserved. After you examine the copied files, you can delete the /var/adm/ files. Umount /newadm, /tmp,/usr, and /var and reboot. You should be good to go.

Now: 1) Never give root to someone 2) Always have a backup.

[Christopher Caldwell]

You might actually get really lucky. Try this ...

If you have a filesystem mount point that contains files and directories - and subsequently you mount another file system on top of it, I believe you'll find that the underlying stuff is still there. umount the file system (e.g. umount /var/adm) and see what happens.

-C

[Christopher Caldwell]

Sorry Richard - I see you found what I said to be true. (I didn't read enough of your post). Think of it this way: the mount essentially added a "directory entry" that pointed to the new filesystem. The underlying structures weren't removed; the new mount point just "overrode" the your view of the directory. When you unmounted the file system, you're left with a empty directory entry and the old contents.

This techniques gives a whole new meaning to hidden file ;-).

[someone_4]

That was a close one...

Thanks,
I awarded 10 points for an actull fix to the issue.

Chris. I was meaning to award you another 8 points for the explaination. But I got click crazy here. IOU - 3 points

You know .. no one here was really worried about /var/adm not being there. I dont think they really understand what goes on in there. I qoute someone: " its only log files"


Richard

[Bill Hassell]

As you have seen, /var/adm is critical to the future operation of the machine. But I hope management now sees the folly of allowing root access to untrained users. If this did not affect upper management yet, you might leave it broken and start sending memos out about the reinstall/rebuild schedule (perhaps 2 weeks?)...oh wait, no backup? That might be attributed to you so dump that idea.

I would simply change the root password and tell no one what it is. Unfortunately, sysadmin wannabe's would think you are being obtuse...so run, don't walk to the tape supplies and start a full backup. At the same time, download the latest version of Ignite/UX and run make_tape_recovery after the full backup is done.

Repeat for every machine that you might get blamed for..

[someone_4]

Hi Mr. Hassell,

I did think about leaving it broken. But when would it be noticed? Maybe when someone tries to install or remove something?

As far as me being attributed for no back ups. I have made big stides in that. I have been able to purchase 10 tape drives and do back ups on "critical servers". And this server did not fall into that catergory. So it would noe have fallen on my lap.. it just falls on my lap when people mess up. And all I can do is try to fix what is broken.

lol .. you know I did try your suggestion about root passwords. Once I did do that. And I even have some servers that no one but me knows the root password. When I first started working here everyone and their grandma logged in as root. Slowley but surley that habbit is getting broken. There were people who complained and complained about root login. But it seems to be one server at a time. Most of the really crutial everyone does not have the root password. And root login with telnet is disabled. We are slowly moving towards no root login.

Richard

[avsrini]

Hi Richard,
Well now you might be knowing the need for having a full backup handy.

One more thing, If you mount any filesystem on a mount point having files, it is not deleted. It just becomes unaccessable. If you just unmount the fs, you can see the files back. So no need to panic for this, unless the original fs is removed.

Well as already suggested, in your case of /var/adm, you have to boot into single user mode to copy the files from /var/adm to new fs to avoid the file in use problems.

Regards
Srini.