Unable to boot: quorum check failed

[NDO]

Hi all

Please I need assistance if you can help. I have a system rx4640, running hp-ux 11.23 that cannot boot, after EFI shell and entering:

Shell> fs0:
fs0:>hpux

it goes up to a stage where it says:

Logical volume 64, 0x5 configured as ROOT
LVM: VG 64 0X000000: Quorum check failed!
LVM:  Activation of root volume group failed
Quorum not present,  or some physical volume(s) are missing

Unfortunately I have I inherited that system with not ignite, and for some reason I also cannot boot from dvd, because at the EFI shell, if I pick "boot from internal dvd" it simply says "not found", for and dvd inserted, so it looks like the dvd drive is also faulty.

Please can someone assist

[NDO]

Hi!

I have managed to bring the system up, by typing the following, on:

HPUX> hpux -lq /stand/vmunix

But at the monment I cannot see the following file systems:

/opt
/var
/data2
/data6

I am a bit scared to reboot the system again, because I could end up again with same situation, do you have any ideas?

I try to fsck /var and /data2 and /data6, it came up with the following:

fsck -F vxfs -o full -y /dev/vg00/lvol10
file system is larger than device
vxfs fsck: cannot initialize aggregate
file system check failure, aborting ...

[NDO]

An update:

assuming that I might an issue with one of the internal disks, I did a diskinfo:

diskinfo -v /dev/rdsk/c2t0d0
SCSI describe of /dev/rdsk/c2t0d0:
             vendor: HP 146 G
         product id: ST3146707LC
               type: direct access
               size: 143374744 Kbytes
   bytes per sector: 512
          rev level: HPC1
    blocks per disk: 286749488
        ISO version: 0
       ECMA version: 0
       ANSI version: 3
    removable media: no
    response format: 2
   (Additional inquiry bytes: (32)4b (33)53 (34)33 (35)4d (36)59 (37)51 (38)43 (39)20 (40)20 (41)20 (42)20 (43)20 (44)20 (45)20 (46)20 (47)20 (48)20 (49)20 (50)20 (51)f (52)0 (53)0 (54)60 (55)a (56)dc (57)b (58)59 (59)3 (60)1 (61)3 (62)22 (63)0 (64)0 (65)0 (66)0 (67)0 (68)0 (69)0 (70)0 (71)0 (72)0 (73)0 (74)0 (75)0 (76)0 (77)0 (78)0 (79)0 (80)0 (81)0 (82)0 (83)0 (84)0 (85)0 (86)0 (87)0 (88)0 (89)0 (90)0 (91)0 (92)43 (93)6f (94)70 (95)79 (96)72 (97)69 (98)67 (99)68 (100)74 (101)20 (102)28 (103)63 (104)29 (105)20 (106)32 (107)30 (108)30 (109)35 (110)20 (111)53 (112)65 (113)61 (114)67 (115)61 (116)74 (117)65 (118)20 (119)41 (120)6c (121)6c (122)20 (123)0 (124)0 (125)0 (126)0 (127)0 (128)0 (129)0 (130)0 (131)0 (132)0 (133)0 (134)0 (135)0 (136)0 (137)0 (138)0 )

diskinfo -v /dev/rdsk/c2t1d0
SCSI describe of /dev/rdsk/c2t1d0:
             vendor: HP 146 G
         product id: ST3146707LC
               type: direct access
               size: 143374744 Kbytes
   bytes per sector: 512
          rev level: HPC1
    blocks per disk: 286749488
        ISO version: 0
       ECMA version: 0
       ANSI version: 3
    removable media: no
    response format: 2
   (Additional inquiry bytes: (32)4b (33)53 (34)33 (35)4d (36)59 (37)48 (38)30 (39)20 (40)20 (41)20 (42)20 (43)20 (44)20 (45)20 (46)20 (47)20 (48)20 (49)20 (50)20 (51)f (52)0 (53)0 (54)60 (55)a (56)dc (57)b (58)59 (59)3 (60)1 (61)3 (62)22 (63)0 (64)0 (65)0 (66)0 (67)0 (68)0 (69)0 (70)0 (71)0 (72)0 (73)0 (74)0 (75)0 (76)0 (77)0 (78)0 (79)0 (80)0 (81)0 (82)0 (83)0 (84)0 (85)0 (86)0 (87)0 (88)0 (89)0 (90)0 (91)0 (92)43 (93)6f (94)70 (95)79 (96)72 (97)69 (98)67 (99)68 (100)74 (101)20 (102)28 (103)63 (104)29 (105)20 (106)32 (107)30 (108)30 (109)35 (110)20 (111)53 (112)65 (113)61 (114)67 (115)61 (116)74 (117)65 (118)20 (119)41 (120)6c (121)6c (122)20 (123)0 (124)0 (125)0 (126)0 (127)0 (128)0 (129)0 (130)0 (131)0 (132)0 (133)0 (134)0 (135)0 (136)0 (137)0 (138)0 )

from that it looks that the internal disk are ok, so what options do I have? another reboot? Please help

[Robert_Jewell]

Well it appears you do have a failing disk.  While diskinfo may report it as OK, it may not be responding to actual I/O requests.  The messages shown with the quorom info should indicate which disk was not able to be seen.

Also, it seems that while you have more than one disk device in your VG00 volume group, the logical volumes themselves were not mirrored, or at least not all of them.  I suspect the second disk was added to give VG00 more space.  To help determine this you can run the command "vgdisplay -v /dev/vg00".  This will the status of each logical volume to give you an idea of what data may need to be restored.

Since the disk device is not attaching to the VG, you can simply replace it online and then configure it back into the volume group.  Once that is done, you can start restoring the data needed.

Here is a document that is useful when dealing with disk failures on HP-UX:

http://h20566.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c01911837&lang=en-us&cc=us

-Bob

[Bill Hassell]

fsck -F vxfs -o full -y /dev/vg00/lvol10
file system is larger than device
vxfs fsck: cannot initialize aggregate
file system check failure, aborting ...

 You may have inerited a system with other sysadmin mistakes. This is the message you would see if some reduced the size of a logical volume (lvol10) BEFORE reducing the filesystem. Repair may be quite a task if there aren't good records such as a sysinfo or cfg2html listing.

[NDO]

Hi

I found out from "strings" command:

/dev/vg00
/dev/dsk/c2t1d0s2
/dev/dsk/c34t1d3
/dev/vg01
/dev/dsk/c4t0d1
/dev/dsk/c4t0d2
/dev/dsk/c4t0d3
/dev/dsk/c34t0d4
/dev/dsk/c34t1d2

and EVA (storage) disk was part of vg00, but when doing :

vgdisplay -v /dev/vg00 | more
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c34t1d3":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c34t1d3":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c34t1d3":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c34t1d3":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c34t1d3":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c34t1d3":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c34t1d3":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c34t1d3":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c34t1d3":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c34t1d3":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c34t1d3":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c34t1d3":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c34t1d3":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c34t1d3":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c34t1d3":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
--- Volume groups ---
VG Name                     /dev/vg00
VG Write Access             read/write
VG Status                   available
Max LV                      255
Cur LV                      13
Open LV                     13
Max PV                      16
Cur PV                      2
Act PV                      1
Max PE per PV               4356
VGDA                        2
PE Size (Mbytes)            32
Total PE                    4346
Alloc PE                    4346
Free PE                     0
Total PVG                   0
Total Spare PVs             0
Total Spare PVs in use      0

   --- Logical volumes ---
   LV Name                     /dev/vg00/lvol1
   LV Status                   available/syncd
   LV Size (Mbytes)            512
   Current LE                  16
   Allocated PE                16
   Used PV                     1

   LV Name                     /dev/vg00/lvol2
   LV Status                   available/syncd
   LV Size (Mbytes)            4096
   Current LE                  128
   Allocated PE                128
   Used PV                     1

   LV Name                     /dev/vg00/lvol3
   LV Status                   available/syncd
   LV Size (Mbytes)            4096
   Current LE                  128
   Allocated PE                128
   Used PV                     1

   LV Name                     /dev/vg00/lvol4
   LV Status                   available/syncd
   LV Size (Mbytes)            4096
   Current LE                  128
   Allocated PE                128
   Used PV                     1

   LV Name                     /dev/vg00/lvol5
   LV Status                   available/syncd
   LV Size (Mbytes)            1024
   Current LE                  32
   Allocated PE                32
   Used PV                     1

   LV Name                     /dev/vg00/lvol6
   LV Status                   available/syncd
   LV Size (Mbytes)            1024
   Current LE                  32
   Allocated PE                32
   Used PV                     1

   LV Name                     /dev/vg00/lvol7
   LV Status                   available/syncd
   LV Size (Mbytes)            10240
   Current LE                  320
   Allocated PE                320
   Used PV                     1

   LV Name                     /dev/vg00/lvol8
   LV Status                   available/syncd
   LV Size (Mbytes)            1024
   Current LE                  32
   Allocated PE                32
   Used PV                     1

   LV Name                     /dev/vg00/lvol9
   LV Status                   available/syncd
   LV Size (Mbytes)            5952
   Current LE                  186
   Allocated PE                186
   Used PV                     1

   LV Name                     /dev/vg00/lvol10
   LV Status                   available/syncd
   LV Size (Mbytes)            10240
   Current LE                  320
   Allocated PE                320
   Used PV                     1

   LV Name                     /dev/vg00/lvol11
   LV Status                   available/syncd
   LV Size (Mbytes)            4096
   Current LE                  128
   Allocated PE                128
   Used PV                     1

   LV Name                     /dev/vg00/lvol12
   LV Status                   available/syncd
   LV Size (Mbytes)            55040
   Current LE                  1720
   Allocated PE                1720
   Used PV                     1

   LV Name                     /dev/vg00/lvol13
   LV Status                   available/syncd
   LV Size (Mbytes)            45024
   Current LE                  1407
   Allocated PE                1407
   Used PV                     1


   --- Physical volumes ---
   PV Name                     /dev/dsk/c2t1d0s2
   PV Status                   available
   Total PE                    4346
   Free PE                     0
   Autoswitch                  On
   Proactive Polling           On

and I beleive that c34t1d3 disk does not contain O.S. information. If I lvreduce those logical volumes belonged to that disk would I be able to solve this problem?

[NDO]

Hi

I found out from vgdisplay -v /dev/vg00, that logical volumes /dev/vg00/lvol10 which is /var and /dev/vg00/lvol7 which is /opt are using missing disk c34t1d3 as you can see from a backup of configuration of the system:

LV Name                     /dev/vg00/lvol7
   LV Status                   available/syncd           
   LV Size (Mbytes)            10240           
   Current LE                  320       
   Allocated PE                320         
   Used PV                     2

LV Name                     /dev/vg00/lvol10
   LV Status                   available/syncd           
   LV Size (Mbytes)            10240           
   Current LE                  320       
   Allocated PE                320         
   Used PV                     2 

but now if I run the same command, for those file systems "Used PV" its equal to 1, so it means that some info of those file systems are on the missing disk

[Robert_Jewell]

Run lvdisplay -v on the lvols you mention; then run lvdisplay -v -k to show the PV key values.  That will show where the extents are mapped to and if they were by chance mirrored.  Doubtful, from the looks of things though.

In the end though, if the missing disk is truly gone and was not mirrored, then there may not be much to do but run vgreduce -f vg00

You may need to end up restoring data to those filesystems..

-Bob

[NDO]

Hi

Thanks for the reply, I am afraid nothing is mirrored, but the system is UP with missing file systems such as :

/var
/opt
/data2
/data6

the first two belong to vg00, and the bottom two belongs to /vg01, but other file systems from the same volume groups are mounted:

#bdf
Filesystem          kbytes    used   avail %used Mounted on
/dev/vg00/lvol5    1048576  298680  749896   28% /
/dev/vg00/lvol1     524288  141992  379344   27% /stand
/dev/vg00/lvol9    6094848 2710800 3357656   45% /usr
/dev/vg00/lvol8    1048576  726176  321224   69% /tmp
/dev/vg00/lvol6    1048576  951464   97112   91% /home
/dev/vg00/lvol13   46104576 20200615 24341535   45% /fs2
/dev/vg00/lvol12   56360960 37020940 18140158   67% /fs1
/dev/vg01/lvol14   204800000 199489115 5310885   97% /data5
/dev/vg01/lvol4    307200000 278013584 27769729   91% /data4
/dev/vg01/lvol3    170672128 168798398 1863664   99% /data3
/dev/vg01/lvol1    170672128 152433462 17322236   90% /data1

what are the implications if I run vgreduce -f vg00?

[Robert_Jewell]

First off, you might as well try a standard vgreduce:

# vgreduce /dev/vg00 /dev/dsk/c34t1d3

That may fail with a message about physical extents being attached.  If so, run the lvdisplay as I mentioned to confirm.

Running vgreduce -f will force the removal of missing PV from the VG.  You will need to rebuild the /etc/lvmtab file using vgscan afterwards.  Check the man page for vgreduce and vgscan for more info.

At any rate, I would consider a backup plan in case things get worse.

-Bob