INIT: Command is respawning too rapidly.

[Grayh]

When I do a #mount -a it gives me the below error

ount - Cannot open /dev/vg01/ts: No such device or addressVs in use 0

Model

--- Logica

crt0: ERROR couldn't open /usr/lib/dld.sl errno:000000002 /dev/vg00/lvol1
Good memory require
mount - Cannot open /dev/vg01/sap64vol4: No such device or addressrimary boot path: 8/4.5 (dec)
LV Size (Mbytes)

mount - Cannot open /dev/vg01/sap64vol1: No such device or address 21
Console path:
Allocated PE

mount - Cannot open /dev/vg01/develop: No such device or address 1

LV Name

crt0: ERROR couldn't open /usr/lib/dld.sl errno:000000002
crt0: ERROR couldn't open /usr/lib/dld.sl errno:000000002
mount - Cannot open /dev/vg01/oracle_9i: No such device or address

mount - Cannot open /dev/vg01/ts: No such device or address

crt0: ERROR couldn't open /usr/lib/dld.sl errno:000000002
mount - Cannot open /dev/vg01/sap64vol4: No such device or address

mount - Cannot open /dev/vg01/sap64vol1: No such device or address

mount - Cannot open /dev/vg01/develop: No such device or address

mount - Cannot open /dev/vg01/oracle: No such device or address

mount - Cannot open /dev/vg01/qc: No such device or address

mount - Cannot open /dev/vg01/lvolusr: No such device or address

mount - Cannot open /dev/vg01/lvolopt: No such device or address

mount: /dev/vg00/lvol5 is already mounted on /home
mount: /dev/vg00/lvol1 is already mounted on /stand

[Fabio Ettore]

Hi,

I'll try to clear.
What you get on vg01 is absolutely normal and expected, in single user mode you don't have volume groups other than vg00 activated. But this is not important now, what it's crucial is to get the vg00 (whit its filesystems) working well.

I suggest checking only vg00 filesystems now, post the following output:

mount
cat /etc/fstab

Best regards,
Fabio

[Grayh]

# cat /etc/fstab
/dev/vg00/lvol3 / vxfs delaylog 0 1
/dev/vg00/lvol1 /stand hfs defaults 0 1
/dev/vg00/lvol5 /home vxfs delaylog 0 2
/dev/vg01/lvolopt /opt vxfs delaylog 0 2
/dev/vg01/lvolusr /usr vxfs delaylog 0 2
/dev/vg01/qc /qc vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2
/dev/vg01/oracle /oracle vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2
/dev/vg01/develop /develop vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2
#/dev/vg01/sapvol1 /usr/sap/trans vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2
#/dev/vg01/sapvol5 /oracle/stage/stage_804 vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2
/dev/vg01/swap2 ... swap pri=0 0 0
/dev/vg01/sap64vol1 /sapmnt/D46 vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2
/dev/vg01/sap64vol2 /usr/sap/D46 vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2
/dev/vg01/sap64vol4 /D46 vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2
/dev/vg01/sap64vol7 /D46/origlogA vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2
/dev/vg01/sap64vol8 /D46/origlogB vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2
/dev/vg01/sap64vol9 /D46/mirrorlogA vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2
#/dev/vg01/sap64vol11 /D46/saparch vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2
#/dev/vg01/sap64vol10 /oracle/D46/mirrorlogB vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2
/dev/vg01/sap64vol12 /D46/sapreorg vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2
#/dev/vg01/sap64vol13 /oracle/D46/sapdata1 vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2
/dev/vg01/sap64vol14 /D46/sapdata2 vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2
#/dev/vg01/sap64vol15 /oracle/D46/sapdata3 vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2
/dev/vg01/sap64vol16 /D46/sapdata4 vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2
#/dev/vg01/sap64vol17 /oracle/D46/sapdata5 vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2
/dev/vg01/sap64vol18 /D46/sapdata6 vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2
#/dev/vg01/sap64vol3 /oracle/stage/stage_805 vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2
#/dev/vg01/sap64vol5 /oracle/805_64 vxfs rw,suid,nolargefiles,delaylog,datainlog 0 2
sun8a:/develop/Rational/config /licenses nfs rw,suid 0 0
/dev/vg01/ts /ts

[Grayh]

A vgdisplay gives me the following....


VGDA 2 LV Size (Mbytes)
PE Size (Mbytes) 43
Current LE
Total PE 2169198 h
Allocated PE
Alloc PE 10549
Used PV
Free PE 1115dan sh21080.5

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t8d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
VG Name /dev/vg01
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 27
Open LV 27

Does that mean anything.... Couldn't query physical disk.....

[Grayh]

# mount -a
crt0: ERROR couldn't open /usr/lib/dld.sl errno:000000002
crt0: ERROR couldn't open /usr/lib/dld.sl errno:000000002
crt0: ERROR couldn't open /usr/lib/dld.sl errno:000000002
vxfs mount: /dev/vg01/oracle_9i is corrupted. needs checking
crt0: ERROR couldn't open /usr/lib/dld.sl errno:000000002
mount - read of super-block on /dev/vg01/sap64vol18 failed: I/O error

vxfs mount: /dev/vg01/qc is corrupted. needs checking
mount - read of super-block on /dev/vg01/lvolusr failed: I/O error

mount - read of super-block on /dev/vg01/lvolopt failed: I/O error

mount: /dev/vg00/lvol5 is already mounted on /home
mount: /dev/vg00/lvol1 is already mounted on /stand
mount: /develop/bldacct.bak: No such file or directory

[Grayh]

# strings /etc/lvmtab
sh: strings: not found
# vi /etc/lvmtab
sh: vi: not found
# cat /etc/lvmtab
N/dev/vg00w%b =:/dev/dsk/c0t5d0/dev/vg01w%b =v/dsk/c0t11d0/dev/dsk/c0t8d0#

[Fabio Ettore]

The configuration of this system is not so good. You have fundamental filesystems (/opt and /usr) under vg01, so a vg different from vg00, I admit I've never seen that before:

# cat /etc/fstab
/dev/vg00/lvol3 / vxfs delaylog 0 1
/dev/vg00/lvol1 /stand hfs defaults 0 1
/dev/vg00/lvol5 /home vxfs delaylog 0 2
/dev/vg01/lvolopt /opt vxfs delaylog 0 2
/dev/vg01/lvolusr /usr vxfs delaylog 0 2
....

So active vg01 and then mount /opt and /usr:

vgchange -a y vg01
mount /opt
mount /usr

If you get that they need checking, issue:

/sbin/fs/vxfs/fsck -F vxfs -y /dev/vg01/rlvolopt
/sbin/fs/vxfs/fsck -F vxfs -y /dev/vg01/rlvolusr

When you finish them please post just

mount
(without option!)

So we'll see what is currently mounted.
If you can mount /usr then

strings /etc/lvmtab

should work.

Fabio

[Grayh]

Hi Cannot munout /usr or /opt after a vgchange

# vgchange -a y vg01
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c0t8d0":
Device busy
Volume group "vg01" has been successfully changed.


# mount /opt
mount - read of super-block on /dev/vg01/lvolopt failed: I/O error

# mount /usr
mount - read of super-block on /dev/vg01/lvolusr failed: I/O error

[Fabio Ettore]

Ok, I think we are close to the reason of your problem: the disk of vg01 c0t8d0 has serious problems. The chain of events for your problem should be as follows:

- disk c0t8d0 has I/O problems;
- these I/O problems prevent vg01 to be activated;
- vg01 cannot be activated and so fundamental filesystems like /usr and /opt cannot be mounted;
- if they cannot be mounted on boot then most services won't start! and a consequence is the message "INIT: Command is respawning too rapidly. ".

CONCLUSION: open a call by HP to get disk c0t8d0 replaced as soon as possible (however I don't know how to recover data there, I would think you need to reinstall the system)

[Grayh]

More Info:

# lvlnboot -v
Boot Definitions for Volume Group /dev/vg00:
Physical Volumes belonging in Root Volume Group:
/dev/dsk/c0t5d0 (8/4.5.0) -- Boot Disk
Boot: lvol1 on: /dev/dsk/c0t5d0
Root: lvol3 on: /dev/dsk/c0t5d0
Swap: lvol2 on: /dev/dsk/c0t5d0
Dump: lvol2 on: /dev/dsk/c0t5d0, 0

lvlnboot: Couldn't query physical volume "/dev/dsk/c0t8d0":
The specified path does not correspond to physical volume attached to
this volume group

# pvdisplay -v /dev/dsk/c0t8d0
pvdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t8d0":
The specified path does not correspond to physical volume attached to
this volume group
pvdisplay: Warning: couldn't query all of the physical volumes.
pvdisplay: Couldn't retrieve the names of the physical volumes
belonging to volume group "/dev/vg01".
pvdisplay: Cannot display physical volume "/dev/dsk/c0t8d0".

# bdf
sh: bdf: not found

# ioscan -fnC disk
Class I H/W Path Driver S/W State H/W Type Description
=============================================================
disk 0 8/4.5.0 sdisk CLAIMED DEVICE SEAGATE ST39173WC
/dev/dsk/c0t5d0 /dev/rdsk/c0t5d0
disk 1 8/4.8.0 sdisk CLAIMED DEVICE SEAGATE ST39173WC
/dev/dsk/c0t8d0 /dev/rdsk/c0t8d0
disk 2 8/4.9.0 sdisk CLAIMED DEVICE SEAGATE ST318275LC
/dev/dsk/c0t9d0 /dev/rdsk/c0t9d0
disk 3 8/4.10.0 sdisk CLAIMED DEVICE SEAGATE ST318275LC
/dev/dsk/c0t10d0 /dev/rdsk/c0t10d0
disk 4 8/4.11.0 sdisk CLAIMED DEVICE SEAGATE ST318275LC
/dev/dsk/c0t11d0 /dev/rdsk/c0t11d0

# vgscan -v
vgscan: Warning: couldn't query physical volume "/dev/dsk/c0t8d0":
The specified path does not correspond to physical volume attached to
this volume group
vgscan: Warning: couldn't query all of the physical volumes.
vgscan: The physical volume "/dev/dsk/c0t5d0" is already recorded in the "/etc/lvmtab" file.
vgscan: The physical volume "/dev/dsk/c0t8d0" is already recorded in the "/etc/lvmtab" file.
vgscan: The physical volume "/dev/dsk/c0t9d0" is already recorded in the "/etc/lvmtab" file.
vgscan: The physical volume "/dev/dsk/c0t10d0" is already recorded in the "/etc/lvmtab" file.
vgscan: The physical volume "/dev/dsk/c0t11d0" is already recorded in the "/etc/lvmtab" file.

Couldn't stat physical volume "/dev/dsk/c0t8d0":
Invalid argument

/dev/vg00
/dev/dsk/c0t5d0

vgscan: has no correspoding valid raw device file under /dev/rdsk.
Verification of unique

[Grayh]

many of the commands like ll, swlist, diskinfo were not working in single user mode...

Is this something strange........

[Grayh]

Hi fabio....

Is there anything else that could be done...

I do not know y most of the commands are not working here

[Torsten.]

If you are still in "pure" single user mode, do the following to "enable" the commands:

# vgchange -a y vg00
# mount -a

# export PATH=$PATH:/usr/sbin

"diskinfo" will work now - please use it with the disk in question.

[Grayh]

I did that but when Issuing the mount -a command I get the same error and the commands wont work.

[Torsten.]

You already posted sooo many information - what is "the same error"?

Consider to pull the disk in question and replace it.

[Grayh]

How can I know if the disk is mirriored c0t8d0...

And if so... can this be removed

[Torsten.]

# vgchange -q n -a y vg01
# vgdisplay -v vg01

[Torsten.]

BTW, one of the main problems is your configuration!

your fstab:

/dev/vg01/lvolopt /opt vxfs delaylog 0 2
/dev/vg01/lvolusr /usr vxfs delaylog 0 2

You should never do this! You are about to learn why now.

[Grayh]

Hi Torson,

# vgchange -q n -a y vg01
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c0t8d0":
Device busy
Volume group "vg01" has been successfully changed.

[Torsten.]

This thread would be at least half in size if you would follow all the advises ...

come on, do now

# vgdisplay -v vg01

as advised.

[Grayh]

# vgdisplay -v vg01
vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t8d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
--- Volume groups ---
VG Name /dev/vg01
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 27
Open LV 27
Max PV 16
Cur PV 4
Act PV 3
Max PE per PV 4342
VGDA 6
PE Size (Mbytes) 4
Total PE 13020
Alloc PE 13020
Free PE 0
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t8d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
--- Logical volumes ---
LV Name /dev/vg01/qc
LV Status available/syncd
LV Size (Mbytes) 6892
Current LE 1723
Allocated PE 1723
Used PV 1

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t8d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name /dev/vg01/oracle
LV Status available/syncd
LV Size (Mbytes) 4000
Current LE 1000
Allocated PE 1000
Used PV 1

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t8d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name /dev/vg01/develop
LV Status available/syncd
LV Size (Mbytes) 8000
Current LE 2000
Allocated PE 2000
Used PV 1

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t8d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name /dev/vg01/lvolusr
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 256
Allocated PE 256
Used PV 0

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t8d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name /dev/vg01/sapvol5
LV Status available/syncd
LV Size (Mbytes) 384
Current LE 96
Allocated PE 96
Used PV 1

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t8d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name /dev/vg01/lvolopt
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 256
Allocated PE 256
Used PV 0

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t8d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name /dev/vg01/swap2
LV Status available/syncd
LV Size (Mbytes) 4096
Current LE 1024
Allocated PE 1024
Used PV 1

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t8d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name /dev/vg01/sap64vol2
LV Status available/syncd
LV Size (Mbytes) 400
Current LE 100
Allocated PE 100
Used PV 1

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t8d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name /dev/vg01/sap64vol7
LV Status available/syncd
LV Size (Mbytes) 56
Current LE 14
Allocated PE 14
Used PV 1

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t8d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name /dev/vg01/sap64vol8
LV Status available/syncd
LV Size (Mbytes) 48
Current LE 12
Allocated PE 12
Used PV 1

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t8d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name /dev/vg01/sap64vol9
LV Status available/syncd
LV Size (Mbytes) 56
Current LE 14
Allocated PE 14
Used PV 1

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t8d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name /dev/vg01/sap64vol10
LV Status available/syncd
LV Size (Mbytes) 48
Current LE 12
Allocated PE 12
Used PV 1

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t8d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name /dev/vg01/sap64vol5
LV Status available/syncd
LV Size (Mbytes) 40
Current LE 10
Allocated PE 10
Used PV 0

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t8d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name /dev/vg01/sapvol1
LV Status available/syncd
LV Size (Mbytes) 500
Current LE 125
Allocated PE 125
Used PV 2

vgdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t8d0":
The specified path does not correspond to physical volume attached to
this volume group
vgdisplay: Warning: couldn't query all of the physical volumes.
LV Name /dev/vg01/ts
LV Status available/syncd
LV Size (Mbytes) 5000
Current LE 1250
Allocated PE 1250
Used PV 1

[Torsten.]

Looks like your LVOLs are NOT mirrored.

Use

# lvdisplay -v /dev/vg01/qc

to confirm this.
This will also show you what LVOL is using which disk.

All of the LVOLs that uses c0t8d0 are corrupted, since this disk is bad.

Check all other (working) disks with

pvdisplay -v ...

to see what is on it.



Consider to swap the disk and do a complete data and OS restore.


IMPORTANT:

this time keep all vg00-related LVOLS within vg00 !!!

[Grayh]

Hi Torsten,

I have setup a call to replace the disk... but when the issue the below command i get strange ???? in the o/p

# lvdisplay -v /dev/vg01/qc
lvdisplay: Warning: couldn't query physical volume "/dev/dsk/c0t8d0":
The specified path does not correspond to physical volume attached to
this volume group
lvdisplay: Warning: couldn't query all of the physical volumes.
--- Logical volumes ---
LV Name /dev/vg01/qc
VG Name /dev/vg01
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 6892
Current LE 1723
Allocated PE 1723
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation strict
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c0t9d0 1250 1250

--- Logical extents ---
LE PV1 PE1 Status 1
00000 /dev/dsk/c0t9d0 00000 current
00001 /dev/dsk/c0t9d0 00001 current
00002 /dev/dsk/c0t9d0 00002 current
00003 /dev/dsk/c0t9d0 00003 current
00004 /dev/dsk/c0t9d0 00004 current
00005 /dev/dsk/c0t9d0 00005 current
00006 /dev/dsk/c0t9d0 00006 current
00007 /dev/dsk/c0t9d0 00007 current
.
.
.
01248 /dev/dsk/c0t9d0 01248 current
01249 /dev/dsk/c0t9d0 01249 current
01250 ??? 01697 current
01251 ??? 01698 current
01252 ??? 01699 current
01253 ??? 01700 current
01254 ??? 01701 current
01255 ??? 01702 current
.
.
.
# ls /dev/dsk/c0t9d0
/dev/dsk/c0t9d0
# ls -lart /dev/dsk/c0t9d0
brw-r----- 1 bin sys 31 0x009000 Apr 18 2000 /dev/dsk/c0t9d0
# ls -lart /dev/dsl k/c0t0 t5d0
brw-r----- 1 bin sys 31 0x005000 Apr 18 2000 /dev/dsk/c0t5d0
# ls -lart /dev/dsk/c0t8d0
brw-r----- 1 bin sys 31 0x008000 Apr 18 2000 /dev/dsk/c0t8d0
# ls -lart /dev/dsk/c0t10d0
brw-r----- 1 bin sys 31 0x00a000 Apr 18 2000 /dev/dsk/c0t10d0
# ls -lart /dev/dsk/c0t11d0
brw-r----- 1 bin sys 31 0x00b000 Apr 18 2000 /dev/dsk/c0t11d0

[Grayh]

looks like there is some issue with the assignment of major number and the minor number here... Or Is this Ok to use

[Torsten.]

There is nothing strange - have a look:

Allocated PE 1723

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c0t9d0 1250 1250


473 PEs are on the bad disk -> this corrupted now.




Conclusion from all information you posted:

You have several problems:

1) someone configured
/usr
and
/opt
to be in vg01 instead of vg00!

Never do this again!

2) disks are not mirrored

3) a disk in vg01 is dead, so all LVOLs with PEs on this disk are unusable too.


Suggestion for solution:

1) replace the disk
2) restore the backup



Good luck!