HP-UX 11.11 system unbootable, bootlif corrupted

[Dante Aubert]

Hi all,

I have an existing B.11.11 system that has become unbootable. I attempted to increase the size of /stand to meet the upgrade requirements for B.11.23 and that was obviously a very bad thing. I've used the 11.11 install and recovery disc to access the recovery menu, but when I choose the option to recover the bootlif/os partitions it fails, saying that / is full and cannot be written to and that mkboot cannot execute. Any shell that I can get to has such a stripped down set of available commands that it's about as good as having no shell at all. I've gotten an idea of how I should have approached this in the first place by looking at the posts on this forum, but right now I'm trying to focus on any way to undo what I did. Help would be greatly appreciated.

Thanks!

[Patrick Wallek]

Do you have a make_tape_recovery tape or a make_net_recovery Ignite image? If so, now is the time to use it.

You can also use that as an opportunity to increase your /stand filesystem.

[Torsten.]

How did you increase /stand?

Normally this is not possible, because it has to be contiguous. A backup and restore with ignite is a good way to go.
Restore your backup and adjust the values.

[Enzo Genuardi]

Hi Sakorm,

Could you say to us the commands that you type, before rebooting the server?

Then you need to decide your way:
-if you have a good backup or better an Ignite recovery then it will less time consuming to startover with, then interact with menu to increase your desired /stand file system.
-Or are you hable to install a second disk into your server do an install of HPUX 11.11 then try to mount your original disk?

You should tell us a little more!
Cheers,
Enzo.

[Albert_31]

Hi,

Ignite restore is the best when time is a constraint.. else you can try this.. i have never tried to reduce a lv in recovery mode..you can give it a try if u want to..

a) boot with recovery cd
b) recovery menu
c) recover a unbootable system -> check there is an option that would be to activate the VG and mount it..

it will also prompt you to run the following commands..

# loadfile chroot
# cd /ROOT
# chroot /ROOT /sbin/sh

Once here try to mount the /usr
and undo what you have done..

If you are not able to activate the volume group..as shown above..then you can try the manual procedure.

#exit to shell from the recovery menu..

# loadfile lvchange vgimport vgdisplay vgchange
# loadfile lvlnboot lvrmboot ioscan
# mkdir /dev/vg00
# mknod /dev/vg00/group c 64 0
# vgimport vg00 /dev/dsk/device file
Creating "/etc/lvmtab".
Warning: A backup of this volume group may not exist on this machine.
Please remember to take a backup using the vgcfgbackup command after activating
the volume group.
# vgchange â a y vg00
Activated volume group
Volume group "vg00" has been successfully changed.
# try to undo what you have done..

Let me know if it is successful or not..

regards

albert

[Dante Aubert]

Hi all,

Thanks for your responses. I'm a relatively new HP Admin and this has basically been a case study in what NOT to do when upgrading this OS :/

To answer your questions about how I got in this predicament: I cheated. When I saw that my filesystem was smaller than recommended for the upgrade to 11.23 I tried to extend the necessary filesystems using the [lvextend - unmount - extendfs - mount] sequence. This was something I'd done successfully before on filesystems other than / and /stand. In my research on how to upgrade, I don't think I came across the red flags that let me know that what I was about to do was B-A-D; I booted the machine in single user mode, set /stand to be non-contiguous, and extended the filesystem. Luckily, / wouldn't let me do it. When I rebooted, the system hung because /stand/vmunix couldn't be executed (probably because of my bad LIF magic). To add to the fun, I don't have a recovery image (tape or otherwise) and am left to try to salvage the situation and learn valuable lessons in the process.

I've booted to the OS install and recovery disc and tried to recover the system, but it keeps telling me that the filesystem (/) is full and can't be written to. When I started this process, the filesystem was at 59% capacity; now it's at 98%. Somewhere along the way, stuff was dumped into / as the system was trying to boot itself. The problem is, I don't know where the stuff was dumped. I've scoured the filesystem looking for recent files, dump files, or anything that looks like it might be the culprit and I've found nothing.

Here are the recovery steps I've taken using the 11.11 install and recovery disc (June 2004 release), and how they've failed:

Boot from install disc
Run a recovery shell
No networking
Recover an unbootable system
Device verification menu
Mount the root disk, then exit to shell only
*This runs a filesystem check (fsck -y) which fails, saying that there was a bad super block, that the mount of /dev/dsk fails, and then kicks me back to the recover menu
Recover bootlif/os partitions
*This fails saying that the / filesystem is full and that mkboot could not be run successfully
Replace kernel on root filesystem
*Haven't tried this yet
Do both B & C (mount root disk, recover bootlif/os)
*Haven't tried this yet

I have another disc on the system; I may just try to load the new OS on that disc and see if I can recover/access the old one.

Albert: I tried following the steps you listed. When I tried to "chroot /ROOT /sbin/sh" it said that /sbin/sh was not found. Did you mean "chroot /ROOT /sbin"? Unfortunately, because the filesystem is full, I can't really load files, so my main concern right now is trying to free up space on /.

A long-winded post, yes, but hopefully it answers the questions you guys have and sparks some ideas about solutions.

Best regards

[Torsten.]

You did destroy it:

"I booted the machine in single user mode, set /stand to be non-contiguous, and extended the filesystem"

That's it. Lesson learned.