read of super-block on /dev/vg00/* failed

[Seidel_1]

Hi,
after failure of root disk(vg00) which was mirrored and replacement of the faulty disk,
I get the following during reboot of the server:

Logical volume 64, 0x3 configured as ROOT
LVM : Failure in attaching PV (0/1/1/0.0.0) to the root volume group.
Cross device link. The disk is not a LVM disk.
Logical volume 64, 0x2 configured as SWAP
WARNING: Logical volume for Dump expected but not found.
Swap device table: (start & size given in 512-byte blocks)
entry 0 - major is 64, minor is 0x2; ignored - can't read device
WARNING: no swap device configured, so dump cannot be defaulted to primary swap.
WARNING: No dump devices are configured. Dump is disabled.
Starting the STREAMS daemons-phase 1
Checking root file system.
file system is clean - log replay is not required
Root check done.
Create STCP device files
Starting the STREAMS daemons-phase 2
$Revision: vmunix: vw: -proj selectors: CUPI80_BL2000_1108 -c 'Vw for CUPI80_BL2000_1108 build' -- cupi80_bl2000_1108 'CUPI80_BL2000_1108' We
d Nov 8 19:24:56 PST 2000 $
Memory Information:
physical page size = 4096 bytes, logical page size = 4096 bytes
Physical: 6289408 Kbytes, lockable: 4927208 Kbytes, available: 5665024 Kbytes

/sbin/ioinitrc:
: Cannot read from the device/dev/vg00/lvol1: unrecognized file system
Unable to mount /stand - please check entries in /etc/fstab
Skipping KRS database initialization - /stand can't be mounted

/sbin/bcheckrc:
Checking for LVM volume groups and Activating (if any exist)
vxfs fsck: sanity check: root file system OK (mounted read/write)
Checking hfs file systems
/sbin/fsclean: Could not read superblock of /dev/vg00/lvol1: I/O error
fsclean RETURNED AN ERROR, CONTINUING, BUT CHECK /etc/fstab FILE
Checking vxfs file systems
/dev/vg00/lvldap :
vxfs fsck: read of super-block on /dev/vg00/lvldap failed: I/O error
vxfs fsck: read of super-block on /dev/vg00/lvldap failed: I/O error
file system check failure, aborting ...
/dev/vg00/lvnmsopt :
vxfs fsck: read of super-block on /dev/vg00/lvnmsopt failed: I/O error
vxfs fsck: read of super-block on /dev/vg00/lvnmsopt failed: I/O error
file system check failure, aborting ...


System is not coming up any idea what I could do now, before I start a IGNITE restore ?

Thanks

Andree

[Mario_66]

Hi,

you have not mentioned on which system this happened?

What steps did you take during replacement?

Have you tried to boot system from old disk only?

M.

[Mario_66]

Hi,

I assumed that you did all necessary steps before. E.g. pvcreate -B, mkboot, lvlnboot, lif ...

Before you start ignite, try to boot in maintanance mode and execute command:

#lvlnboot -R

Maybe it can help.

Regards,
Mario.

[Devender Khatana]

Hi,

The disk through which the system is trying to boot is either faulty or has got coppution due to failure in other disk. What happens if you boot of the other disk?

HTH,
Devender

[Seidel_1]

Hi,

I did the following, just now(hpux -lm):

# lvlnboot -R
lvlnboot: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
lvlnboot: Warning: couldn't query physical volume "/dev/dsk/c2t1d0":
The specified path does not correspond to physical volume attached to
this volume group
lvlnboot: Warning: couldn't query all of the physical volumes.
lvlnboot: Warning: couldn't query physical volume "/dev/dsk/c4t0d0":
The specified path does not correspond to physical volume attached to
this volume group
lvlnboot: Warning: couldn't query physical volume "/dev/dsk/c6t0d0":
The specified path does not correspond to physical volume attached to
this volume group
.
.
The specified path does not correspond to physical volume attached to
this volume group
vgcfgbackup: Warning: couldn't query physical volume "/dev/dsk/c6t1d0":
The specified path does not correspond to physical volume attached to
this volume group
vgcfgbackup: Warning: couldn't query all of the physical volumes.
vgcfgbackup: Volume group not activated.
# pvcreate -B /dev/rdsk/c2t0d0
pvcreate: The physical volume "/dev/dsk/c2t0d0" is already recorded in the "/etc/lvmtab" file.
# pvcreate -B /dev/rdsk/c2t1d0
Couldn't stat physical volume "/dev/rdsk/c2t1d0":
Physical volume can be created only on whole disk.
# pvcreate -B /dev/rdsk/c2t1d0
pvcreate: The physical volume already belongs to a volume group


I cannot bot from any of the disks, some more ideas ?

Thanka

Andree

[Devender Khatana]

Hi,

Having boot in LVM maintenancemode can you post

#vgdisplay -v /dev/vg00
&
#lvdisplay -v /dev/vg00/lvol*
&
#mountall
&
#bdf

HTH,
Devender

[Seidel_1]

OK, I tried to boot from the disk which was not broken:

ISL> hpux -lm

Boot
: disk(0/1/1/0.1.0.0.0.0.0;0)/stand/vmunix
12406784 + 2093056 + 15248016 start 0x202d68




alloc_pdc_pages: Relocating PDC from 0xfffffff0f0c00000 to 0x3f900000.
gate64: sysvec_vaddr = 0xc0002000 for 2 pages
NOTICE: nfs3_link(): File system was registered at index 3.
NOTICE: cachefs_link(): File system was registered at index 7.
[Osxinit]: Allocated ots_timeout_lock [4074c380]
td: claimed Tachyon XL2 Fibre Channel Mass Storage card at 0/4/1/0
igelan0: INITIALIZING HP PCI 1000Base-T Core at hardware path 0/1/2/0

System Console is on the Built-In Serial Interface
Logical volume 64, 0x3 configured as ROOT
WARNING: Logical volume for Dump expected but not found.
Swap device table: (start & size given in 512-byte blocks)
entry 0 - auto-configured on root device; ignored - no room
WARNING: no swap device configured, so dump cannot be defaulted to primary swap.
WARNING: No dump devices are configured. Dump is disabled.
Starting the STREAMS daemons-phase 1
Checking root file system.
log replay in progress
replay complete - marking super-block as CLEAN
Root check done.
Create STCP device files
Starting the STREAMS daemons-phase 2
$Revision: vmunix: vw: -proj selectors: CUPI80_BL2000_1108 -c 'Vw for CUPI80_BL2000_1108 build' -- cupi80_bl2000_1108 'CUP
I80_BL2000_1108' Wed Nov 8 19:24:56 PST 2000 $
Memory Information:
physical page size = 4096 bytes, logical page size = 4096 bytes
Physical: 6289408 Kbytes, lockable: 4927268 Kbytes, available: 5665024 Kbytes

/sbin/ioinitrc:
Can't open /dev/vg00/lvol1, errno = 6
/dev/vg00/lvol1: CAN'T CHECK FILE SYSTEM.
/dev/vg00/lvol1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
/dev/vg00/lvol1: No such device or address
Unable to mount /stand - please check entries in /etc/fstab
Skipping KRS database initialization - /stand can't be mounted


INIT: Overriding default level with level 's'

INIT: SINGLE USER MODE

INIT: Running /sbin/sh

# vgdisplay -v /dev/vg00
vgdisplay: Volume group not activated.
vgdisplay: Cannot display volume group "/dev/vg00".
# vgdisplay -v /dev/vg00/lvol1
vgdisplay: Volume group "/dev/vg00/lvol1" does not exist in the "/etc/lvmtab" file.
vgdisplay: Cannot display volume group "/dev/vg00/lvol1".
# vgdisplay -v /dev/vg00/lvol2
vgdisplay: Volume group "/dev/vg00/lvol2" does not exist in the "/etc/lvmtab" file.
vgdisplay: Cannot display volume group "/dev/vg00/lvol2".
# vgdisplay -v /dev/vg00/lvol3
vgdisplay: Volume group "/dev/vg00/lvol3" does not exist in the "/etc/lvmtab" file.
vgdisplay: Cannot display volume group "/dev/vg00/lvol3".
# vgdisplay -v /dev/vg00/lvol4
vgdisplay: Volume group "/dev/vg00/lvol4" does not exist in the "/etc/lvmtab" file.
vgdisplay: Cannot display volume group "/dev/vg00/lvol4".
# vgdisplay -v /dev/vg00/lvol5
vgdisplay: Volume group "/dev/vg00/lvol5" does not exist in the "/etc/lvmtab" file.
vgdisplay: Cannot display volume group "/dev/vg00/lvol5".
# vgdisplay -v /dev/vg00/lvol6
vgdisplay: Volume group "/dev/vg00/lvol6" does not exist in the "/etc/lvmtab" file.
vgdisplay: Cannot display volume group "/dev/vg00/lvol6".
# mountall
mountall: /dev/vg00/lvol6 has to be fsck'd
mountall: diagnostics from fsck
vxfs fsck: Cannot open /dev/vg00/lvol6: No such device or address
mountall: cannot fsck /dev/vg00/lvol6
mountall: diagnostics from fsck
file system check failure, aborting ...
No such device or address
mountall: /dev/vg00/lvol6 failed in fsck
mountall: /dev/vg00/lvol6 cannot be mounted
mountall: cannot mount /dev/vg00/lvldap
mountall: /dev/vg00/lvol7 has to be fsck'd
mountall: diagnostics from fsck
vxfs fsck: Cannot open /dev/vg00/lvol7: No such device or address
mountall: cannot fsck /dev/vg00/lvol7
mountall: diagnostics from fsck
file system check failure, aborting ...
No such device or address
mountall: /dev/vg00/lvol7 failed in fsck
mountall: /dev/vg00/lvol7 cannot be mounted
mountall: /dev/vg00/lvol5 has to be fsck'd
mountall: diagnostics from fsck
vxfs fsck: Cannot open /dev/vg00/lvol5: No such device or address
mountall: cannot fsck /dev/vg00/lvol5
mountall: diagnostics from fsck
file system check failure, aborting ...
No such device or address
mountall: /dev/vg00/lvol5 failed in fsck
mountall: /dev/vg00/lvol5 cannot be mounted
mountall: /dev/vg00/lvol1 is seriously damaged
mountall: diagnostics from fsck
Can't open /dev/vg00/lvol1, errno = 6
mountall: cannot mount /dev/vg00/lvol1
mountall: /dev/vg00/lvol4 has to be fsck'd
mountall: diagnostics from fsck
vxfs fsck: Cannot open /dev/vg00/lvol4: No such device or address
mountall: cannot fsck /dev/vg00/lvol4
mountall: diagnostics from fsck
file system check failure, aborting ...
No such device or address
mountall: /dev/vg00/lvol4 failed in fsck
mountall: /dev/vg00/lvol4 cannot be mounted
mountall: /dev/vg00/lvnmsopt has to be fsck'd
mountall: diagnostics from fsck
vxfs fsck: Cannot open /dev/vg00/lvnmsopt: No such device or address
mountall: cannot fsck /dev/vg00/lvnmsopt
mountall: diagnostics from fsck
file system check failure, aborting ...
t: No such device or address
mountall: /dev/vg00/lvnmsopt failed in fsck
mountall: /dev/vg00/lvnmsopt cannot be mounted
mountall: cannot mount 172.25.122.35:/SUTELAB/ups/maint
mountall: diagnostics from mount
crt0: ERROR couldn't open /usr/lib/dld.sl errno:000000002
mountall: cannot mount 172.25.122.35:/SUTELAB/ups/etc
mountall: diagnostics from mount
crt0: ERROR couldn't open /usr/lib/dld.sl errno:000000002
# bfd
sh: bfd^[[3~^[[3~: not found
# bdf
sh: bdf: not found
# /usr/bin/bdf
sh: /usr/bin/bdf: not found


Looks not good correct ?

BR

Andree

[Devender Khatana]

Hi,

Forget to mention that before all these also use

#vgchange -a y -q n /dev/vg00

And then try these commands. Also try running fsck manually and see what happens.

But get ready to use the recent gnite backup. Try a few things before finally going for it.
How critical is the system/downtime?

HTH,
Devender

[Seidel_1]

# vgchange -a y -q n /dev/vg00
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c2t0d0":
Cross-device link
Activated volume group
vgchange: Couldn't re-synchronize stale partitions of the logical volume:
I/O error
vgchange: Couldn't re-synchronize stale partitions of the logical volume:
I/O error
vgchange: Couldn't re-synchronize stale partitions of the logical volume:
I/O error
vgchange: Couldn't re-synchronize stale partitions of the logical volume:
I/O error
vgchange: Couldn't re-synchronize stale partitions of the logical volume:
I/O error
vgchange: Couldn't re-synchronize stale partitions of the logical volume:
I/O error
vgchange: Couldn't re-synchronize stale partitions of the logical volume:
I/O error
vgchange: Couldn't re-synchronize stale partitions of the logical volume:
I/O error
vgchange: Couldn't re-synchronize stale partitions of the logical volume:
I/O error
Volume group "/dev/vg00" has been successfully changed.
# vgdisplay -v /dev/vg00
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
--- Volume groups ---
VG Name /dev/vg00
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 9
Open LV 9
Max PV 16
Cur PV 2
Act PV 1
Max PE per PV 5000
VGDA 2
PE Size (Mbytes) 8
Total PE 4340
Alloc PE 4314
Free PE 26
Total PVG 1
Total Spare PVs 0
Total Spare PVs in use 0

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
--- Logical volumes ---
LV Name /dev/vg00/lvol1
LV Status available/stale
LV Size (Mbytes) 504
Current LE 63
Allocated PE 126
Used PV 1

.
.
.
.
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name /dev/vg00/lvol6
LV Status available/stale
LV Size (Mbytes) 10000
Current LE 1250
Allocated PE 2500
Used PV 1

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name /dev/vg00/lvol7
LV Status available/stale
LV Size (Mbytes) 2000
Current LE 250
Allocated PE 500
Used PV 1


--- Physical volumes ---
PV Name /dev/dsk/c2t1d0
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c2t0d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
PV Status available
Total PE 4340
Free PE 26
Autoswitch On


--- Physical volume groups ---
PVG Name scsi2_1
PV Name /dev/dsk/c2t1d0
# vgdisplay -v /dev/vg00/lvol1
vgdisplay: Volume group "/dev/vg00/lvol1" does not exist in the "/etc/lvmtab" file.
vgdisplay: Cannot display volume group "/dev/vg00/lvol1".
# vgdisplay -v /dev/vg00/lvol2
vgdisplay: Volume group "/dev/vg00/lvol2" does not exist in the "/etc/lvmtab" file.
vgdisplay: Cannot display volume group "/dev/vg00/lvol2".
# vgdisplay -v /dev/vg00/lvol3
vgdisplay: Volume group "/dev/vg00/lvol3" does not exist in the "/etc/lvmtab" file.
vgdisplay: Cannot display volume group "/dev/vg00/lvol3".
# vgdisplay -v /dev/vg00/lvol4
vgdisplay: Volume group "/dev/vg00/lvol4" does not exist in the "/etc/lvmtab" file.
vgdisplay: Cannot display volume group "/dev/vg00/lvol4".

# mountall
mountall: /dev/vg00/lvol6 has to be fsck'd
mountall: diagnostics from fsck
vxfs fsck: read of super-block on /dev/vg00/lvol6 failed: I/O error
mountall: cannot fsck /dev/vg00/lvol6
mountall: diagnostics from fsck
file system check failure, aborting ...
g00/lvol6 failed: I/O error
mountall: /dev/vg00/lvol6 failed in fsck
mountall: /dev/vg00/lvol6 cannot be mounted
mountall: cannot mount /dev/vg00/lvldap
mountall: /dev/vg00/lvol7 has to be fsck'd
mountall: diagnostics from fsck
vxfs fsck: read of super-block on /dev/vg00/lvol7 failed: I/O error
mountall: cannot fsck /dev/vg00/lvol7
mountall: diagnostics from fsck
file system check failure, aborting ...
g00/lvol7 failed: I/O error
mountall: /dev/vg00/lvol7 failed in fsck
mountall: /dev/vg00/lvol7 cannot be mounted
mountall: /dev/vg00/lvol5 has to be fsck'd
mountall: diagnostics from fsck
vxfs fsck: read of super-block on /dev/vg00/lvol5 failed: I/O error
mountall: cannot fsck /dev/vg00/lvol5
mountall: diagnostics from fsck
file system check failure, aborting ...
g00/lvol5 failed: I/O error
mountall: /dev/vg00/lvol5 failed in fsck
mountall: /dev/vg00/lvol5 cannot be mounted
mountall: cannot fsck /dev/vg00/lvol1
mountall: diagnostics from fsck
: Cannot read from the devicemountall: cannot mount /dev/vg00/lvol1
mountall: /dev/vg00/lvol4 has to be fsck'd
mountall: diagnostics from fsck
vxfs fsck: read of super-block on /dev/vg00/lvol4 failed: I/O error
mountall: cannot fsck /dev/vg00/lvol4
mountall: diagnostics from fsck
file system check failure, aborting ...
g00/lvol4 failed: I/O error
mountall: /dev/vg00/lvol4 failed in fsck
mountall: /dev/vg00/lvol4 cannot be mounted
mountall: /dev/vg00/lvnmsopt has to be fsck'd
mountall: diagnostics from fsck
vxfs fsck: read of super-block on /dev/vg00/lvnmsopt failed: I/O error
mountall: cannot fsck /dev/vg00/lvnmsopt
mountall: diagnostics from fsck
file system check failure, aborting ...
g00/lvnmsopt failed: I/O error
mountall: /dev/vg00/lvnmsopt failed in fsck
mountall: /dev/vg00/lvnmsopt cannot be mounted
mountall: cannot mount 172.25.122.35:/SUTELAB/ups/maint
mountall: diagnostics from mount
crt0: ERROR couldn't open /usr/lib/dld.sl errno:000000002
mountall: cannot mount 172.25.122.35:/SUTELAB/ups/etc
mountall: diagnostics from mount
crt0: ERROR couldn't open /usr/lib/dld.sl errno:000000002
#

# fsck
** /dev/vg00/rlvol1: Cannot read from the devicelog replay in progress
replay complete - marking super-block as CLEAN
vxfs fsck: read of super-block on /dev/vg00/lvol4 failed: I/O error
file system check failure, aborting ...
vxfs fsck: read of super-block on /dev/vg00/lvol5 failed: I/O error
file system check failure, aborting ...
vxfs fsck: read of super-block on /dev/vg00/lvol6 failed: I/O error
file system check failure, aborting ...
vxfs fsck: read of super-block on /dev/vg00/lvol7 failed: I/O error
file system check failure, aborting ...
vxfs fsck: read of super-block on /dev/vg00/lvnmsopt failed: I/O error
file system check failure, aborting ...
vxfs fsck: read of super-block on /dev/vg00/lvldap failed: I/O error
file system check failure, aborting ...


Is it time for Ignite ?

[Devender Khatana]

Hi,

Is it time for ignite?

Yes you can start restoring ignite. But first replace the disk. Also first tell with which disk the system is currently booted? and is other disk removed or still inserted?

If only one disk is currently inserted then try LVM maintenance boot from the other disk and retry these steps.

Has you removed the wrong disk?
Also what is the criticality of the system?

What is the output of

#strings /etc/lvmtab

If vg00 is not listed here then run vgscan to recreate lvmtab.

HTH,
Devender