LVM Problem during boot (vxfs fsck sanity check failed)

[ballans]

Hello,
my system is an HP-UX 11iv2 on an rp3440 and a disk array attached on this (MSA-30).
On the server, there are 2 disks in a mirror volume named /dev/vg00
On the raid array, there are 6 disks in a RAID10 level named /dev/vg00

This afternoon, there were no supply during two hours and my server and the msa-30 were shutdown (the power supply was also off).
When the supply is came back, the server has started but for the msa30 there are no light indicators on the disks.
In order to make a test, I remove a disk (physically) and I replace it on the same slot to see if the indicator becomes green.
Finally, I shutdown the msa-30 (unplugged power cables!) and restart it and all the indicators are OK.

But now my hp-ux system doesn't want to mount the vg01 volume...(disks on the raid array)

During the boot sequence, I have this error message :

Checking hfs file systems
/sbin/fsclean: /dev/vg00/lvol1 (mounted) ok
HFS file systems are OK, not running fsck
Checking vxfs file systems
vxfs fsck: /dev/vg00/lvol3:sanity check: root file system OK (mounted read/write
)
vxfs fsck: /dev/vg00/lvol4:sanity check: /dev/vg00/lvol4 OK
vxfs fsck: /dev/vg00/lvol5:sanity check: /dev/vg00/lvol5 OK
vxfs fsck: /dev/vg00/lvol6:sanity check: /dev/vg00/lvol6 OK
vxfs fsck: /dev/vg00/lvol7:sanity check: /dev/vg00/lvol7 OK
vxfs fsck: /dev/vg00/lvol8:sanity check: /dev/vg00/lvol8 OK
vxfs fsck: /dev/vg00/lvol9:sanity check: /dev/vg00/lvol9 OK
vxfs fsck: /dev/vg00/lvol10:sanity check: /dev/vg00/lvol10 OK
vxfs fsck: /dev/vg01/lvol4:sanity check failed: cannot open /dev/vg01/lvol4: No
such device or address
vxfs fsck: /dev/vg01/lvol1:sanity check failed: cannot open /dev/vg01/lvol1: No
such device or address
vxfs fsck: /dev/vg01/lvol2:sanity check failed: cannot open /dev/vg01/lvol2: No
such device or address
vxfs fsck: /dev/vg01/lvol3:sanity check failed: cannot open /dev/vg01/lvol3: No
such device or address

I've made a lot of test to try to solve the problem but with no success... :

1) vgchange -a -y /dev/vg01 and vgchange -a -y -q -n /dev/vg01

vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk
/c4t0d1":
Unknown error
vgchange: Warning: couldn't query physical volume "/dev/dsk/c4t0d1":
The specified path does not correspond to physical volume attached to
this volume group
vgchange: Warning: couldn't query all of the physical volumes.
vgchange: Couldn't activate volume group "/dev/vg01":
Quorum not present, or some physical volume(s) are missing.


2) vgreduce /dev/vg01 /dev/dsk/c4t0d1 and vgextend /dev/vg01 /dev/dsk/c4t0d1

vgextend: The volume group "/dev/vg01" is not active. Only an active
volume group can be extended.

3) I try these commands booting normally and booting in maintenance mode with the quorum override option, hpux -lq

Have you got an idea ?
Please, your help will greatly appreciated, this server have a critical function in my Lab (mail server).

Herve

[Cesare Salvioni · Accepted solution]

hi,
lookslike you lost phisical connectivity between the host and the DA, the vga cannot be activated if the host doesn't see the LUN on the disk array.
No vg command (vgreduce, vgchange etc.) will work if the host does not see the disks.

This could happen because:
1. you changed the phisical connection between host and disk array
2. the disk array or the cards are not working
3. you startup the host BEFORE the disk array

Hoping the 3 is your case try this:
ioscan -kCdisk
and after
ioscan -Cdisk
note the second command is missing the k
in this way you force the system to recheck the hardware, if the two commands give different results, and in the second you see something like CLAIMED where in the first there were NO-HW or something like, the problem is fixed, try again vgchange -a y vg01.
If the two command give the same result post the output of
ioscan -Cdisk
and i can help you more

hope it helps.

[ballans]

Hi Cesare,
thank you a lot for your reply.
Unfortunately, the both command give the same result...
This is the output :

# ioscan -C disk
H/W Path Class Description
========================================================
0/0/2/0.0.0.0 disk TEAC DV-28E-N
0/4/1/0/4/0.0.0 disk HP LOGICAL VOLUME
0/4/1/0/4/0.0.1 disk HP LOGICAL VOLUME

It seems that the system good see the second logical volume (the disks on the raid array system).

I check the cable and verify all the connectivity. It seems to be OK.

I do not understand why the ioscan command doesn't show me all the physical drives but only the logical volume ?

Herve

[Cesare Salvioni]

hi
sorry i forgive one option of ioscan , try with
ioscan -kfCdisk
to see the FULL report for the hardware
The reason for what you see is that from the point of view of the OS what is visible is only the LUN you created on the disk array, not the phisical disks: that's right
Try again the vgchange -a y vg01 now and tell me the output

[ballans]

Hi,

Here are the following output :

# ioscan -kfC disk
Class I H/W Path Driver S/W State H/W Type Description
============================================================================
disk 0 0/0/2/0.0.0.0 sdisk CLAIMED DEVICE TEAC DV-28E-N
disk 1 0/4/1/0/4/0.0.0 sdisk CLAIMED DEVICE HP LOGICAL VOLUME
disk 2 0/4/1/0/4/0.0.1 sdisk CLAIMED DEVICE HP LOGICAL VOLUME

# vgchange -a y /dev/vg01
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c4t0d1":
Unknown error
vgchange: Warning: couldn't query physical volume "/dev/dsk/c4t0d1":
The specified path does not correspond to physical volume attached to
this volume group
vgchange: Warning: couldn't query all of the physical volumes.
vgchange: Couldn't activate volume group "/dev/vg01":
Quorum not present, or some physical volume(s) are missing.

I have the same result with the -q n option

It really seems to be a hardware problem but ioscan see the logical volume. I don't understand ?...

Thank for your help.
Herve

[Cesare Salvioni]

yep, looks like you have a problem on the disk array even if the OS is seeing the devices (CLAIMED).
Try ioscan -fnCdisk, you will see after each line of the two LUN two more info, the block special file and the raw special file identifying the two luns.
This two special should be:
/dev/dsk/c4t0d1
/dev/rdsk/c4t0d1
/dev/dsk/c4t0d0
/dev/rdsk/c4t0d0

if these are correct run:
diskinfo /dev/rdsk/c4t0d1
diskinfo /dev/rdsk/c4t0d0
if the special are not these, let me know
also, is there any led flashing on the MSA?
Last question, you said you detached a disk on the msa, did you put it back before turning it on or after?

[Cesare Salvioni]

One more question, from your output i can see only three disks, did you post all the output?
For what i understood from your first post there should be:
1 dvd (the first one)
2 internal disks (the ones in vg00)
1 or more LUN from the msa

if you posted all the output and i well understood something is missing.
please post:
strings /etc/lvmtab
vgdisplay -v vg00
ioscan -kfn

[ballans]

>This two special should be:
>/dev/dsk/c4t0d1
>/dev/rdsk/c4t0d1
>/dev/dsk/c4t0d0
>/dev/rdsk/c4t0d0

Correct !

>if these are correct run:
>diskinfo /dev/rdsk/c4t0d1
>diskinfo /dev/rdsk/c4t0d0

All is ok here

>if the special are not these, let me know
>also, is there any led flashing on the MSA?

No you right. Only the green led that shows that the disk is connected. But nothing flashing ??

>Last question, you said you detached a disk >on the msa, did you put it back before >turning it on or after?

I detached a disk after (ie when the msa is turned on) because I would like to test if the led disk is flashing when I reconnect it on the msa...
Herve

[ballans]

# strings /etc/lvmtab
/dev/vg00
/dev/dsk/c4t0d0
/dev/vg01
/dev/dsk/c4t0d1

# vgdisplay -v vg00
--- Volume groups ---
VG Name /dev/vg00
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 10
Open LV 10
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 4350
VGDA 2
PE Size (Mbytes) 8
Total PE 4340
Alloc PE 2806
Free PE 1534
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg00/lvol1
LV Status available/syncd
LV Size (Mbytes) 304
Current LE 38
Allocated PE 38
Used PV 1

LV Name /dev/vg00/lvol2
LV Status available/syncd
LV Size (Mbytes) 4096
Current LE 512
Allocated PE 512
Used PV 1

LV Name /dev/vg00/lvol3
LV Status available/syncd
LV Size (Mbytes) 2048
Current LE 256
Allocated PE 256
Used PV 1

LV Name /dev/vg00/lvol4
LV Status available/syncd
LV Size (Mbytes) 512
Current LE 64
Allocated PE 64
Used PV 1

LV Name /dev/vg00/lvol5
LV Status available/syncd
LV Size (Mbytes) 128
Current LE 16
Allocated PE 16
Used PV 1

LV Name /dev/vg00/lvol6
LV Status available/syncd
LV Size (Mbytes) 4096
Current LE 512
Allocated PE 512
Used PV 1

LV Name /dev/vg00/lvol7
LV Status available/syncd
LV Size (Mbytes) 512
Current LE 64
Allocated PE 64
Used PV 1

LV Name /dev/vg00/lvol8
LV Status available/syncd
LV Size (Mbytes) 512
Current LE 64
Allocated PE 64
Used PV 1

LV Name /dev/vg00/lvol9
LV Status available/syncd
LV Size (Mbytes) 4096
Current LE 512
Allocated PE 512
Used PV 1

LV Name /dev/vg00/lvol10
LV Status available/syncd
LV Size (Mbytes) 6144
Current LE 768
Allocated PE 768
Used PV 1


--- Physical volumes ---
PV Name /dev/dsk/c4t0d0
PV Status available
Total PE 4340
Free PE 1534
Autoswitch On


# vgdisplay -v vg01
vgdisplay: Volume group not activated.
vgdisplay: Cannot display volume group "vg01".

# ioscan -kfn
Class I H/W Path Driver S/W State H/W Type Description
==============================================================================
root 0 root CLAIMED BUS_NEXUS
ioa 0 0 sba CLAIMED BUS_NEXUS System Bus Adapter (880)
ba 0 0/0 lba CLAIMED BUS_NEXUS Local PCI-X Bus Adapter (783)
usb 0 0/0/1/0 hcd CLAIMED INTERFACE NEC USB Interface
usbhub 0 0/0/1/0.1 hub CLAIMED DEVICE USB Root Hub
usb 1 0/0/1/1 hcd CLAIMED INTERFACE NEC USB Interface
usbhub 1 0/0/1/1.1 hub CLAIMED DEVICE USB Root Hub
/dev/hub
usb 2 0/0/1/2 ehci CLAIMED INTERFACE NEC USB Interface
sideba 0 0/0/2/0 side_multi CLAIMED INTERFACE CMD IDE controller
ext_bus 0 0/0/2/0.0 side CLAIMED INTERFACE IDE Primary Channel
target 0 0/0/2/0.0.0 tgt CLAIMED DEVICE
disk 0 0/0/2/0.0.0.0 sdisk CLAIMED DEVICE TEAC DV-28E-N
/dev/dsk/c0t0d0 /dev/rdsk/c0t0d0
target 1 0/0/2/0.0.7 tgt CLAIMED DEVICE
ctl 0 0/0/2/0.0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c0t7d0
ext_bus 1 0/0/2/0.1 side CLAIMED INTERFACE IDE Secondary Channel
target 2 0/0/2/0.1.7 tgt CLAIMED DEVICE
ctl 1 0/0/2/0.1.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c1t7d0
ba 1 0/1 lba CLAIMED BUS_NEXUS Local PCI-X Bus Adapter (783)
ext_bus 2 0/1/1/0 c8xx CLAIMED INTERFACE SCSI C1010 Ultra160 Wide LVD
target 7 0/1/1/0.7 tgt CLAIMED DEVICE
ctl 4 0/1/1/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c2t7d0
ext_bus 3 0/1/1/1 c8xx CLAIMED INTERFACE SCSI C1010 Ultra160 Wide LVD
target 5 0/1/1/1.3 tgt CLAIMED DEVICE
tape 0 0/1/1/1.3.0 stape CLAIMED DEVICE HP C5683A
/dev/rmt/0m /dev/rmt/c3t3d0BEST /dev/rmt/c3t3d0DDS
/dev/rmt/0mb /dev/rmt/c3t3d0BESTb /dev/rmt/c3t3d0DDSb
/dev/rmt/0mn /dev/rmt/c3t3d0BESTn /dev/rmt/c3t3d0DDSn
/dev/rmt/0mnb /dev/rmt/c3t3d0BESTnb /dev/rmt/c3t3d0DDSnb
target 6 0/1/1/1.7 tgt CLAIMED DEVICE
ctl 3 0/1/1/1.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c3t7d0
lan 0 0/1/2/0 igelan CLAIMED INTERFACE HP PCI 1000Base-T Core
ba 2 0/2 lba CLAIMED BUS_NEXUS Local PCI-X Bus Adapter (783)
ba 3 0/3 lba CLAIMED BUS_NEXUS Local PCI-X Bus Adapter (783)
ba 4 0/4 lba CLAIMED BUS_NEXUS Local PCI-X Bus Adapter (783)
ba 5 0/4/1/0 PCItoPCI CLAIMED BUS_NEXUS PCItoPCI Bridge
ext_bus 4 0/4/1/0/4/0 ciss CLAIMED INTERFACE PCI-X SmartArray 6402 RAID Controller
/dev/ciss4
target 3 0/4/1/0/4/0.0 tgt CLAIMED DEVICE
disk 1 0/4/1/0/4/0.0.0 sdisk CLAIMED DEVICE HP LOGICAL VOLUME
/dev/dsk/c4t0d0 /dev/rdsk/c4t0d0
disk 2 0/4/1/0/4/0.0.1 sdisk CLAIMED DEVICE HP LOGICAL VOLUME
/dev/dsk/c4t0d1 /dev/rdsk/c4t0d1
target 4 0/4/1/0/4/0.7 tgt CLAIMED DEVICE
ctl 2 0/4/1/0/4/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c4t7d0
ba 6 0/6 lba CLAIMED BUS_NEXUS Local PCI-X Bus Adapter (783)
ba 7 0/7 lba CLAIMED BUS_NEXUS Local PCI-X Bus Adapter (783)
tty 0 0/7/1/0 asio0 CLAIMED INTERFACE PCI SimpleComm (103c1290)
/dev/diag/mux0 /dev/mux0 /dev/tty0p0
tty 1 0/7/1/1 asio0 CLAIMED INTERFACE PCI Serial (103c1048)
/dev/GSPdiag1 /dev/mux1 /dev/tty1p2
/dev/diag/mux1 /dev/tty1p0 /dev/tty1p4
unknown -1 0/7/2/0 UNCLAIMED UNKNOWN PCI Display (10025159)
memory 0 8 memory CLAIMED MEMORY Memory
ipmi 0 16 ipmi CLAIMED INTERFACE IPMI Controller
/dev/ipmi
processor 0 128 processor CLAIMED PROCESSOR Processor
processor 1 129 processor CLAIMED PROCESSOR Processor

[Cesare Salvioni]

so looks like you have defined two lun on the msa, one (c4t0d0) is in the vg00 and contain the OS and this is working fine, the other (c4t0d1) is in the vg01 and is not working.

Before you run the command
diskinfo /dev/rdsk/c4t0d1
this should return the size of the LUN and should work ONLY if the LUN is reachable, did it work?
If yes the LUN looks like fine
try pvdisplay /dev/rdsk/c4t0d1 and post the result

[Cesare Salvioni]

sorry have to go,
anyway i would call HOP assistance, cause looks like there is something problem with your msa, don't know if hardware or software but something is wrong with ur conf.

see u tomorrow

[Cesare Salvioni]

sorry have to go,
anyway i would call HP assistance, cause looks like there is something problem with your msa, don't know if hardware or software but something is wrong with ur conf.

see u tomorrow

[ballans]

>so looks like you have defined two lun on the >msa, one (c4t0d0) is in the vg00 and contain >the OS and this is working fine, the other >(c4t0d1) is in the vg01 and is not working.

Non in fact there is only one LUN on the msa (c4t0d1). The other one is composed by two mirrored internals disks on the rp3440 server.

# diskinfo /dev/rdsk/c4t0d1
SCSI describe of /dev/rdsk/c4t0d1:
vendor: HP
product id: LOGICAL VOLUME
type: direct access
size: 430110072 Kbytes
bytes per sector: 512


# pvdisplay /dev/rdsk/c4t0d1
pvdisplay: Physical volume "/dev/rdsk/c4t0d1" is not a block special file.
Usage: pvdisplay
[-v]
[-b BlockList]
PhysicalVolumePath...

# pvdisplay /dev/dsk/c4t0d1
pvdisplay: Warning: couldn't query physical volume "/dev/dsk/c4t0d1":
The specified path does not correspond to physical volume attached to
this volume group
pvdisplay: Warning: couldn't query all of the physical volumes.
pvdisplay: Couldn't retrieve the names of the physical volumes
belonging to volume group "/dev/vg01".
pvdisplay: Cannot display physical volume "/dev/dsk/c4t0d1".

Thanks again for your help...
Herve

[Torsten.]

Your MSA30 is connected to a smartarray controller.

product id: LOGICAL VOLUME

You can also see this in your ioscan:

ext_bus 4 0/4/1/0/4/0 ciss CLAIMED INTERFACE PCI-X SmartArray 6402 RAID Controller
/dev/ciss4
target 3 0/4/1/0/4/0.0 tgt CLAIMED DEVICE
disk 1 0/4/1/0/4/0.0.0 sdisk CLAIMED DEVICE HP LOGICAL VOLUME
/dev/dsk/c4t0d0 /dev/rdsk/c4t0d0
disk 2 0/4/1/0/4/0.0.1 sdisk CLAIMED DEVICE HP LOGICAL VOLUME
/dev/dsk/c4t0d1 /dev/rdsk/c4t0d1


Now check the controller status

sautil /dev/ciss4

What does it gives?

[ballans]

Hello Torsten,
thanks for your reply.
I attached the output of the follwing command :

# sautil /dev/ciss4

Herve

[Torsten.]

Bad news, the logical drive 1 (this is your c4t0d1) has status failed:

1 1+0 420029 MB FAILED


---- LOGICAL DRIVE 1 ---------------------------------------------------------

Logical Drive Device File........... c4t0d1
Fault Tolerance Mode................ RAID 1+0 (Disk Mirroring)
Logical Drive Size.................. 420029 MB
Logical Drive Status................ FAILED
# of Participating Physical Disks... 6

Participating Physical Disk(s)...... Ch:ID
2: 0
2: 1
2: 2
2: 3
2: 4
2: 5


Last Failure Reason.............................. MISSING IN SAVE RIS



I guess this could be a controller problem.
Check the cabling.

I would recommend to ask your HP support to get this solved.

[ballans]

I asked HP support this morning.
For the moment, they analyse the log of system (script send by them) in order to see if this is an hardware or software problem.

My question is : if the logical volume is failed, does it means that data on the logical volume are lost ? (I don't have an external backup!). Or there is a way to reconstruct the logical volume without lost anything ?

Herve

[Torsten.]

What I noticed in your log, is this:

Connector Location............................... external
Channel Number................................... 2
SCSI ID.......................................... 3
Device Type...................................... DISK
Disk Capacity.................................... 146.8 GB
Device Status.................................... OK
Device Vendor ID................................. COMPAQ
Device Product ID................................ BF1468A4CC
Device Serial Number............................. 3KN0SF6900007619S7X7
Device Firmware Version.......................... HPB5
SCSI Transfer Rate (actual)...................... Ultra-320
Reserved Area (cfg/status info).................. 557.1 KB (0.6 MB)
Block Size (bytes/sector)........................ 512 bytes
M&P Data Stamped?................................ yes
Last Failure Reason.............................. TIMEOUT


Maybe (!) this drive is blocking the bus.
Do you have a good backup?
If, pull this disk and reboot.

This is slot 4, because bays are numbered 1,2,3 ... SCSI ID's are 0,1,2,...

[Torsten.]

No backup?
No you know why backups are so important!!!!

If only 1 drive is failed and blocking the bus, you have a good chance to get your data back.


Remember to do backups in the future!!!!

(Even a RAID may fail - as you know now)

[ballans]

OK promise for backups on the future ;-)

You said that about you, the drive wich causes problem is on the fourth slot (ID3 TIMEOUT)

The disk that I removed from bay and reconnected was on the slot 6...

Now I waiting for the HP support to see if they confirm your diagnostic.

Thanks again
Herve

[Torsten.]

Please keep us informed, this may help others in the future.

[ballans]

[SOLVED]

Hello,
excuse me for the break, but as soon as the problem was solved, I'm going to holidays for few days...

The problem was solved the 26th of july (3 days after the beginning) by an HP technician (he has working on site during 5 hours!!).

as I understood, the source of the problem was the state of last failure disk (MISSING IN SAVE RIS).
In order to make disks in a normal state, he used the following command :

# sautil -d /dev/ciss4 accept_media_xchg "lun number"

For information, about him, this command have to be used with caution, because it could be delete datas of raid system.
In my case, all my datas are presents and OK and nothing deleted.

Thank you again all for your help...
...and now, I am scheduling weekly external backup on another raid box !!

Herve

[ballans]

Precision : the problem (maybe) appeared because the msa-30 was shutdown before the rp server...