Hard crash of filesystem on HP-UX 10.20 workstation

[Michael D'Aulerio]

I have a 9000/744 workstation on which I run HP-UX 10.20. Recently the power was turned off and the system crashed. Not sure if it was a power failure or if someone just powered the system down. When I turned power back on (actually someone else turned it on) the system tried to fsck all the partitions but on at least one of partitions, it said to run fsck manually. I think when the guy did the manual fsck he lost some critical files. Now whenever we bring the system up, almost every command cores. I tried cat, grep, more, etc. None of the commands work and they return "Invalid command. Core dumped." I don't think all of the commands are corrupted. I think one of the shared libraries is either missing or bad. Is there any way to check what files have been deleted and recover them? Just throwing the question out there. Thanks.

[Donny Jekels]

get to single user mode/

BCH prompt type >bo

ISL>interaction type y

at the ISL prompt type

ISL>hpux -is

this will boot the box in single user mode.

at the root mount, you need to mount

#mount /usr

cat /etc/and mount the rest of the moint poitns and fix your system from there

shutdown -r -y 0 when youre done.


peace
donnt - ps good luck

[Donny Jekels]

if the mount fail

run
fsck /dev/vg00/lvolX

[Michael D'Aulerio]

Donny,

Thanks for the quick reply. Actually, I have booted up single user and tried to fix it. Forgot to mention that. I get the same errors when I run the cat, grep, more commands in single user mode. I fsck'ed all the filesystems and mounted them. Any other ideas?

It never ceases to amaze me at how quickly I get replies. Thanks again.

Mike

[A. Clay Stephenson]

A good test would be /sbin/cat. Because it is statically linked (unlike it's /usr/bin/ counterpart), it should work. If I were going to guess, I would say that /usr/lib/libc.1 is corrupt (and maybe /usr/lib/libdld.1.

Your best bet would probably be to restore /usr/lib from backup. Note: frecover is in /sbin so you are in good shape if you backup with fbackup. However, tar and cpio do not have statically linked version in /sbin BUT you do have pax which can handle either format.

[curt larson_1]

have you checked the lost+found directories

[Michael D'Aulerio]

Clay,

You're right about the /sbin commands. The commands in that directory work. That's the only reason I was able to come up in single user mode to begin with. I wish the cksum command still worked. I could run in on /usr/lib/libc.1 and /usr/lib/libdld.1. We didn't make a backup so we can't frecover. We do have a tar tape. It was created using pax. I might be able to use pax to restore the whole /usr directory. Thanks again.

[Michael D'Aulerio]

Clay,
I'm almost certain it's a shared library issue. I ran the gzip command from the /usr partition and it was successful. gzip is also statically linked. I'm going to use pax tomorrow and restore all of /usr. Thanks again.

Curt,
I checked the lost+found directories. There were some files in /var/lost+found but not in /usr/lost+found. I recovered the files in /var/lost+found but it still didn't help. By the way, how do you recover the files from lost+found. I did it by comparing the size and inode of the files with comparable files in the /var partition of a good machine.

[Bill Hassell]

lost+found recovery

When the filesystem becomes corrupted, the crucial link between the data area on the disk (represented by inodes) and the filename is broken. Thus, there is no name for the file. So fsck does the best it can and creates a filename with #12345 where 12345 is the actual inode number. Not much help though although it's better than #aaaaaaa or similar.

Start with the file command as in:

file /usr/lost+found/*

This will show you a best guess as to the contents of the files. Commands (executables) will be shown as such, directories identified, etc. Not the original name but the category. This is a guessing game. It helps to have another working workstation around to compare results.

For ASCII/awk/text files, just look at a few lines and you may be able to guess what the original filename was. To rename, you must escape the # character as in:

mv \#12345 mount

You may be able to identify some commands with strings as in:

strings \#12345

When you find directories, you can look inside the directory and you may see all files intact.

This can take hours of comparing file sizes, strings and file types with a known good system. (backups using Ignite/UX take a lot less time)