Big Problem! - K420 will not fully boot after a simple reboot

[Gavin Westermann]

Can't imagine what happened here. Last night i remotely rebooted a K420 running 11.11 and lost contact with it. I went to it and at a console it was stuck half way through the boot process. rebooted to single user mode, ioscan sees both scsi drives, on raising the run level to 6 I discover that vg02 all of a sudden cannot be seen or is missing? naturally the boot process hangs as /opt is in that group as well as other things. Any tips on how to recover from something like this?

Gavin

[Helen French]

I would do the following:
1) Boot system in single user mode or LVM maintenance mode.
2) Check all suspected disks for any I/O errors.
3) Check file systems with fsck.
4) Check old rc log files and see what is/was failing.
5) Try to activate vg02 manually and mount file systems.
6) Check system log files.

[A. Clay Stephenson]

Run level 6 is very unusual. If this was not a typo, I would first look for some very strange init.d commands that are triggered above run-level 3.

Look at /etc/inittab and see what the initdefault entry is set to; in almost all cases it is set to 3.

An ioscan -fn should indicate whether the disks themselves have disappeared or if your have LVM errors. If a disk shows up as NO_HW then you are having physical disk problems.

[Jeff Schussele]

Hi Gavin,

Sun systems use run levels 5 & 6 for specialized shutdown scripting. HP doesn't. So I'm not sure just *why* you'd attempt run level 6.
Please explain.

Rgds,
Jeff

[Printaporn_1]

Hi Gavin,
Check root's shell does it change from
/sbin/sh ?

[Rajesh G. Ghone]

Hi Gavin,

it looks like disk belongs to VG02 is having some problem, according to me what you can do is go to single user mode try to activate VG02 manually & if it gets activated then run fsck on all the file system blongs to that VG if this things work then i think the system should come up smoothly.

Regards,
Rajesh G.

[Steven E. Protter]

I'm betting that vg02 can't get a quorum and isn't coming up.

There should be something about it in the rc.log or /var/adm/syslog/syslog.log

If the volume group is attached by fiber, check the fiber card

fcmsutil /dev/td0

the device depends on your configuration.

As far as run level goes, I typically deal with production issues with new software that doesn't want to start up right by adding a run level.

Though I prefer to do this on development machines I did once run into a situation where I used Ignite to push out a production server and some oracle product worked right on the golden image server but not production. I dealt with it by changing /etc/inittab and moving the Start and Kill scripts to run level 4 and 3 respectively.

I don't run systems very long that way and recommend returning to a standard configuration as soon as possible.

SEP

[Gavin Westermann]

lets forget about the run level and that is not really an issue, I just picked something off the top of my head to go further in the boot process. it still hangs at that same point. VG02 is definitely not happening, someone mentioned starting it manually? How might I do that, I will have to talk someone through it as I have no physical access to the box at the moment.

[Michael Steele_2]

Gavin:

Display the disk contents of vg02.

strings /etc/lvmtab

-or-

vgcfgrestore -f /etc/lvmconf/vg02.conf -l

Test the disks for failure:

ioscan -fnkC disk | more
(* Check for CLAIMED status *)


dd if=/dev/dsk/cXtYdZ of=/dev/null count=1000000
control c

If prompt does not return then bad disk.

diskinfo /dev/dsk/cXtYdZ
Note type and size if bad, needed for replacement.

If you need vgo2 up then activate without quorum.

vgchange -a y -q n /dev/vg02

[George Liu_2]

Given you can still login in single user mode and vg02 is for /opt only, things are really not very bad. Here is what you need to do,

Login into single user mode;
ioscan to check the disk for vg02 can be recognized. If not, locate problem is from hardware or kernel. Have you performed patch recently?

Run vg commands to display, active, and so on to recover.