URGENT: /usr/conf directories empty.....cannot recompile kernel!

[Daniel Skates]

HI Guys,

Here's a curly one. We have recently taken delivery of an RP7420, installed with HP-UX 11.11

I tried to install MirrorDisk yesterday, and got a message:

OTE: Saving the current system file at "/stand/system" to
"/stand/system.prev"
NOTE: The template file has been extracted from "/stand/vmunix"
It has been placed in "/stand/system" where it will be used
to build a new kernel.
* Installing fileset "LVM.LVM-MIRROR-RUN,r=B.11.11" (1 of 1).
ERROR: Cannot find "/usr/conf/lib/libhp-ux.a" on the system.
This file is necessary to build a kernel.
ERROR: The command "/usr/sbin/mk_kernel", which is used to rebuild
the kernel, has failed. Because kernel-related filesets were
installed, this command must be executed by "swinstall",
without failures, before the load can continue. Check the
above output for details about the failure.
NOTE: The Install Phase has suspended. Check the above output for
reasons.

Now, i checked /usr/conf, and all directories under here are completely empty.(all with the same time/date stamp, and the directory structure is intact).
Last time I generated a kernel, all was well.

Just to rule out easy solutions
1. I have no backup
2. I have no IGNITE
3. Before I get caned for 1 and 2, this machine was only delivered recently, ans still is in config stage. I guess it's a lesson well learned for next time.

Top marks for a non-reinstall solution!

[Sridhar Bhaskarla]

Hi Daniel,

It's not a 100% solution. It's like something is better than nothing.

You can copy /usr/conf directory structure from another RP7420 which is at the *same* OS and patch level of this system.

I do not believe /usr/conf will disappear automatically without human intervention. Someone might have fat fingered a command that might have deleted the directory. If not, check for any backed up directories like 'll -d /usr/conf*', any unmounted filesystems etc.,

-Sri

[Daniel Skates]

I figured I could do that.....now if only I had a server to do it from.....;-)

Thanks for the advice though.

[Daniel Skates]

This machine is only about 10 days old....only myself been playing on it...I'm pretty sure I didn't screw anything up.

Here's the ll output from the directory:
root@parko:/usr/conf > ll
total 192
dr-xr-xr-x 3 bin bin 96 Nov 16 2000 .unsupp
dr-xr-xr-x 2 bin bin 96 Sep 29 12:02 cdfs
dr-xr-xr-x 3 bin bin 8192 Sep 29 12:02 gen
dr-xr-xr-x 2 bin bin 8192 Sep 29 12:02 graf
lrwxr-xr-x 1 root root 5 Sep 15 16:10 h -> ./sys
dr-xr-xr-x 2 bin bin 96 Sep 29 12:02 io
drwxr-xr-x 3 root sys 96 Sep 15 16:25 km.d
dr-xr-xr-x 2 bin bin 96 Sep 29 12:02 ktest
dr-xr-xr-x 2 bin bin 8192 Oct 6 08:48 lib
lrwxr-xr-x 1 root root 4 Sep 15 16:10 machine -> ./pa
dr-xr-xr-x 2 bin bin 8192 Oct 6 08:50 master.d
dr-xr-xr-x 2 bin bin 8192 Sep 29 12:02 net
dr-xr-xr-x 2 bin bin 8192 Sep 29 12:02 netinet
lr-xr-xr-t 1 root sys 19 Sep 15 16:12 nettlconf -> /usr/sbin
/nettlconf
dr-xr-xr-x 2 bin bin 96 Sep 29 12:02 nfs
dr-xr-xr-x 6 bin bin 8192 Sep 29 12:02 pa
dr-xr-xr-x 2 bin bin 8192 Sep 29 12:02 sio
dr-xr-xr-x 2 bin bin 96 Sep 29 12:02 sioserv
dr-xr-xr-x 2 bin bin 8192 Sep 29 12:02 space.h.d
lrwxr-xr-x 1 root root 11 Sep 15 16:10 spmath -> ./pa/spmath
dr-xr-xr-x 2 bin bin 96 Sep 29 12:02 streams
dr-xr-xr-x 2 bin bin 96 Sep 29 12:02 strtio
dr-xr-xr-x 2 bin bin 8192 Sep 29 12:02 sys
dr-xr-xr-x 2 bin bin 8192 Sep 29 12:02 ufs
dr-xr-xr-x 2 bin bin 8192 Sep 29 12:02 wsio

And of the underlying directories.
root@parko:/usr/conf > ll *
lrwxr-xr-x 1 root root 5 Sep 15 16:10 h -> ./sys
lrwxr-xr-x 1 root root 4 Sep 15 16:10 machine -> ./pa
lr-xr-xr-t 1 root sys 19 Sep 15 16:12 nettlconf -> /usr/sbin
/nettlconf
lrwxr-xr-x 1 root root 11 Sep 15 16:10 spmath -> ./pa/spmath

cdfs:
total 0

gen:
total 0
dr-xr-xr-x 2 bin bin 96 Nov 16 2000 templates

graf:
total 0

io:
total 0

km.d:
total 0
drwxr-xr-x 2 root sys 96 Sep 15 16:25 krm

ktest:
total 0

lib:
total 0

master.d:
total 16
-r--r--r-- 1 bin bin 4297 Nov 10 2000 lvm

net:
total 0

netinet:
total 0

nfs:
total 0

pa:
total 32
dr-xr-xr-x 2 bin bin 8192 Sep 29 12:02 spmath
dr-xr-xr-x 2 bin bin 96 Nov 16 2000 sync
dr-xr-xr-x 2 bin bin 8192 Nov 16 2000 sys
dr-xr-xr-x 2 bin bin 96 Nov 16 2000 vm

sio:
total 0

sioserv:
total 0

space.h.d:
total 0

streams:
total 0

strtio:
total 0

sys:
total 0

ufs:
total 0

wsio:
total 0
root@parko:/usr/conf >


I can't help but think it has something to do with the kernel rebuild process.
kmtune returns nothing.
SAM shows nothing when you try and look at the kernel, and all drivers appear OUT when you look at them.
The system however appears happy enough now, but I suspect this may not be the case when I reboot.
Any help here will be greatly appreciated! And as mentioned before, many points to be assigned for a life saver!

[Brad Sheldon]

Hi Daniel looks like you are having fun there. Get Hp to cut you an ignite tape on the same class of machine save you stuffing around. Cheers Brad.

[Daniel Skates]

That's pretty much the plan.....the issue is not so much having to re-install, as is why this has occurred, and make sure it doesn't happen again!

[Cheryl Griffin]

If you are tracking shell history ($HOME/.sh_history) you can determine what commands were run however it doesn't include a date and timestamp of when the commands were run.

Determining what happened after the fact will be difficult unless you have auditing enabled.

[Bharat Katkar]

Hi,
Another way of restoring the /usr/conf may be to boot the system into single user mode.
Mount /usr filesystem and then mount your Core OS CD onto /mnt directory.
See what is missing out by comparing the /usr/conf direcory and the one on the core OS CD.
Try copying the files from CD to the /usr/conf directory.

Hope that should work.
Regards,

[Daniel Skates]

Thanks for the help guys....however we are proceeding with the re-install option, as I need to be absolutely certain that thiungs are right with this machince, as it goes live in a couple of weeks. Shell history doesn't reflect a culprit, but i suspect it's not an rm, as the directory structure, and links are still present.

I guess this has taught me to never take anything for granted, and always assume something will go wrong, even if you don't touch stuff!
POints awarded for effort!

[Daniel Skates]

Re-install of the OS is the best and cleanest way to be sure I am not missing anything.