LVM failure; System Panic

[rianui]

Hi gurus,

I have an rx2600 system running HP-UX B11.23. After a power blackout the system failed to boot. When I accessed the console I found this error:

Loading.: HP-UX Primary Boot: 0/1/1/0.0.0
Load of HP-UX Primary Boot: 0/1/1/0.0.0 failed: Not Found
Paused - press any key to continue

I run the hpux command from fs0> and while booting I saw the following error messages:

Logical volume 64, 0x3 configured as ROOT
LVM : Failure in attaching PV (0/1/1/0.1.0) to the root volume group.
Cross device link. The disk is not a LVM disk.
LVM : Activation of root volume group failed
Quorum not present, or some physical volume(s) are missing

I then loaded the OS in maintenance mode from this point fs0:\EFI\HPUX>hpux -lm

vgdisplay gives me the following output:

# vgdisplay -v /dev/vg00
vgdisplay: Volume group not activated.
vgdisplay: Cannot display volume group "/dev/vg00".

Based on several comments I can see on the community, I have tried to run vgchange but still get the following error.

# vgchange -a y /dev/vg00
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c2t1d0s2":
Cross-device link
vgchange: Warning: couldn't query physical volume "/dev/dsk/c2t1d0s2":
The specified path does not correspond to physical volume attached to
this volume group
vgchange: Warning: couldn't query all of the physical volumes.
vgchange: Couldn't activate volume group "/dev/vg00":
Quorum not present, or some physical volume(s) are missing.

Kindly give me some pointers to get it running again.

Regards,

Rianui

[Torsten.]

Do you have mirrored disks?

You should try to boot from the alternate disk.

Most likely the boot headers got damaged, so without an alternate disks you may consider to use the recovery console or restore your ignite backup.

[rianui]

Hi Torsten,

I must thank you for such a fast response :)

This is where it gets tricky. The client does not know what I mean by ignite backup and has never seen this before. In short, it does not exist...plus all his 'backup' is on the same machine.

I have requested for a new drive and here is my plan:

- Install a new OS on the new drive and try to 'repair' the LVM. I'm still trying to figure out how I'll do that...

Regards,

Rianui

[Torsten.]

You can also boot from the appropriate install DVD, launch the recovery shell and try to repair from there.

Disks are not mirrored?

[rianui]

Hi Torsten,

I'd say the root disks are mirrored. lvdisplay however is not giving the requested info.

# lvdisplay /dev/vg00/lvol3

lvdisplay: Couldn't query logical volume "/dev/vg00/lvol3":
Volume group not activated.

# ioscan -fnCdisk

Class     I  H/W Path       Driver     S/W State   H/W Type     Description
============================================================================
disk      0  0/0/2/0.0.0.0  sdisk      CLAIMED     DEVICE       TEAC    DV-28E-C
                           /dev/dsk/c0t0d0   /dev/rdsk/c0t0d0
disk      1  0/1/1/0.0.0    sdisk      CLAIMED     DEVICE       HP 73.4GMAS3735NC
                           /dev/dsk/c2t0d0     /dev/rdsk/c2t0d0
                           /dev/dsk/c2t0d0s1   /dev/rdsk/c2t0d0s1
                           /dev/dsk/c2t0d0s2   /dev/rdsk/c2t0d0s2
                           /dev/dsk/c2t0d0s3   /dev/rdsk/c2t0d0s3
                           /dev/dsk/c2t0d0s4   /dev/rdsk/c2t0d0s4
                           /dev/dsk/c2t0d0s5   /dev/rdsk/c2t0d0s5
                           /dev/dsk/c2t0d0s6   /dev/rdsk/c2t0d0s6
                           /dev/dsk/c2t0d0s7   /dev/rdsk/c2t0d0s7
disk      2  0/1/1/0.1.0    sdisk      CLAIMED     DEVICE       HP 73.4GMAS3735NC
                           /dev/dsk/c2t1d0     /dev/rdsk/c2t1d0
                           /dev/dsk/c2t1d0s1   /dev/rdsk/c2t1d0s1
                           /dev/dsk/c2t1d0s2   /dev/rdsk/c2t1d0s2
                           /dev/dsk/c2t1d0s3   /dev/rdsk/c2t1d0s3
                           /dev/dsk/c2t1d0s4   /dev/rdsk/c2t1d0s4
                           /dev/dsk/c2t1d0s5   /dev/rdsk/c2t1d0s5
                           /dev/dsk/c2t1d0s6   /dev/rdsk/c2t1d0s6
                           /dev/dsk/c2t1d0s7   /dev/rdsk/c2t1d0s7

[Torsten.]

Can you boot from the other disk???

What happens if you boot in quorum mode?

You are in maintenance mode now?


Try

# vgchange -a y -q n vg00

[rianui]

Hi Torsten,

I tried that before. See the output below.

# vgchange -a y -q n vg00
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c2t1d0s2":
Cross-device link
vgchange: Warning: couldn't query physical volume "/dev/dsk/c2t1d0s2":
The specified path does not correspond to physical volume attached to
this volume group
vgchange: Warning: couldn't query all of the physical volumes.
vgchange: Couldn't activate volume group "vg00":
Quorum not present, or some physical volume(s) are missing.

[Torsten.]

But you booted from this disk?
In maintenance mode?


There is something really strange - usually there is s1, s2 and s3 - but you have s1 ...s7 - 7 partitions???



Did you boot from the other disk too??

[rianui]

Hi Torsten,

I am trying to boot from the alternate disk.

My map table looks like this:

 fs0  : Acpi(HWP0002,100)/Pci(1|0)/Scsi(Pun0,Lun0)/HD(Part1,Sig8A3D797A-FB9B-11D8-8002-D6217B60E588)
  fs1  : Acpi(HWP0002,100)/Pci(1|0)/Scsi(Pun0,Lun0)/HD(Part3,Sig8A3D79AC-FB9B-11D8-8004-D6217B60E588)
  fs2  : Acpi(HWP0002,100)/Pci(1|0)/Scsi(Pun1,Lun0)/HD(Part1,SigDCA80A9C-FF80-11D9-8002-D6217B60E588)
  fs3  : Acpi(HWP0002,100)/Pci(1|0)/Scsi(Pun1,Lun0)/HD(Part3,SigDCA80AD8-FF80-11D9-8004-D6217B60E588)
  blk0 : Acpi(HWP0002,0)/Pci(2|0)/Ata(Primary,Master)
  blk1 : Acpi(HWP0002,100)/Pci(1|0)/Scsi(Pun0,Lun0)
  blk2 : Acpi(HWP0002,100)/Pci(1|0)/Scsi(Pun0,Lun0)/HD(Part1,Sig8A3D797A-FB9B-11D8-8002-D6217B60E588)
  blk3 : Acpi(HWP0002,100)/Pci(1|0)/Scsi(Pun0,Lun0)/HD(Part2,Sig8A3D7998-FB9B-11D8-8003-D6217B60E588)
  blk4 : Acpi(HWP0002,100)/Pci(1|0)/Scsi(Pun0,Lun0)/HD(Part3,Sig8A3D79AC-FB9B-11D8-8004-D6217B60E588)
  blk5 : Acpi(HWP0002,100)/Pci(1|0)/Scsi(Pun1,Lun0)

[rianui]

fs0:\EFI\HPUX> hpux -lq

The command gives me the following output:

panic: LVM: Configuration failure
Stack Trace:
  IP                  Function Name
  0xe000000000e32300  lv_boothalt+0x110
  0xe000000000e31d70  lv_bootconf+0x3c0
  0xe000000000e3ad00  lvmconf+0x2e0
  0xe000000000e40930  im_lvmconf+0x70
  0xe0000000011d27d0  DoCalllist+0x3a0
End of Stack Trace

linkstamp:          Wed Sep 01 00:29:29 METDST 2004
_release_version:   @(#) $Revision: vmunix:    B11.23_LR FLAVOR=perf Fri Aug 29 22:35:38 PDT 2003 $

sync'ing disks (0 buffers to flush): (0 buffers to flush):
0 buffers not flushed
0 buffers still dirty
crash_callback: calling function e000000000d4da60

*** A system crash has occurred. (See the above messages for details.)
*** The system is now preparing to dump physical memory to disk, for use
*** in debugging the crash.

ERROR:  Your system crashed before I/O and dump configuration was complete.
        This system does not support a crash dump under these circumstances.
        Contact your HP support representative for assistance.

[rianui]

Hi Torsten,

The alternate drive cannot boot either. Both drives are reporting the same panic error.

Regards,

Rianui

[rianui · Accepted solution]

Hi Torsten,

Root disk 0 gave me trouble coming up but the mirror worked after a while. Here is what I finally did to bring disk 1 up.

- I got to the shell:

Shell> map

- I then selected the mirror drive:
Shell>blk6:

- I changed to the EFI and then to the HPUX folders:
blk6:\> cd EFI
blk6:\EFI>cd HPUX

- Then I started the OS in maintenance mode:
blk6:\EFI\HPUX> hpux -lm

- In maintenance mode I  renamed the lvmtab file:
# mv /etc/lvmtab /etc/lvmtab.old

_ Imported the volume group vg00:

# vgimport vg00 /dev/dsk/c2t1d0s2

- I set LVM volume group availability:

# vgchange -a y /dev/vg00

- I then run fsck and transitioned system to run level 3

# init 3

The system is currently up. I will take the necessary DB and application backups and once the client is ready, I will rebuild the system and do a proper ignite backup. Thanks a lot for your support!

Regards,

Rianui

[Torsten.]

Good news.

It is not recommended to do a init 3 from maintenance mode.

However, if it is up now, you should get more details and try to repair.

How about the the partitions?

e.g.

# idisk /dev/rdsk/c2t1d0

then

# lvlnboot -v

# vgdisplay -v vg00

etc ...



If there are still problems, ignitebackup will likely fail.

[rianui]

Hi Torsten,

See the document attached.

Regards,

Rianui

[Torsten.]

According to the earlier ioscan it looks like both internal disks were bootable, but now only 1 disk is in vg00.

There are 3 partitions (correct), but device files for 7 partitions.

I would rmsf them and re-create. Then I would mirror the disk again.

[rianui]

Hi Torsten,

Could you kindly give me the commands to do this ? :)

[Torsten.]

Mirroring?

Refer to the appendix:

When_Good_Disks_Go_Bad_WP
http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c01911837/c01911837.pdf

[rianui]

I meant the rmsf'ing' :)

Anyway, disk 0 has given me too much trouble and still won't come up. The live system is now running on disk 1. I will do a back-up of the system soon and just rebuild the OS to save time.

Thanks again for your invaluable support.

Regards,

Rianui