LVM error on HP UX for vxfs

[nitin_31]

We have two partitions on HP UX (rx2620) from HITACHI SAN box.
But when I used dmesg it showed the LVM errore as " LVM: VG 64 0x010000: PVLink 31 0x040000 Failed! The PV is still accessible "
Please guide me.
The details dmesg snapshot is attached
Regards,
Nitin

[Mustafa Gulercan]

hi;
could you post
ls -l /dev/*/group | grep '0x040000'

The might be a disk failed.

[Victor Fridyev]

Hi,

Please run the command
#>dmesg -
and answer with an output of the following commands:
#>ioscan -fnCdisk
#>dmesg -

If dmesg - shown nothing suspicious, run
#>dd if=/dev/dsk/XXXX of=/dev/null count=1000

for the disk where you have the problem.

HTH

[Darrel Louis]

Hi,

Check which disk is causing the error:

ls -l /dev/*/group | grep '0x040000'
- diskinfo of the disk
- pvdisplay of the disk
- vgdisplay of the VG to check if you have stale extends

Darrel

[Florian Heigl (new acc)]

The messages are quite verbose, and don't hold much good news.

towards the reason I would guess that Your zoning or LUN presentation must have gotten messed up at some point, thus the 'wrong disk' messages.

no matter if there still are STALE PE or not, some of Your filesystems were marked as corrupt by the OS.

I would recover, if You can take some risk or have means of checking data integrity use

fsck -Fvxfs -o full,nolog on the filesystems. I wouldn't trust my data any more at the point You're at.

[sathish kannan]

Hi Nitin,
The messages tells you that one of the PVLINK to device "0x040000" is down. Check your device 0x040000 and the respective volumegroup. probably the device might have been unavailable to your host.

Regards
Sathish

[nitin_31]

Hi Everybody,

Thanks for your valuable support & time.

When I am using dmesg - it doesn't show any new error message so I am not able to conclude abt the error time stamp. but still I am sending u the all out puts from the cmnds u suggested for your reference.

# ioscan -funC disk
Class I H/W Path Driver S/W State H/W Type Description
============================================================================
disk 0 0/0/2/0.0.0.0 sdisk CLAIMED DEVICE TEAC DV-28E-N /dev/dsk/c0t0d0 /dev/rdsk/c0t0d0
disk 1 0/1/1/0.0.0 sdisk CLAIMED DEVICE HP 73.4GST373454LC /dev/dsk/c2t0d0 /dev/dsk/c2t0d0s2 /dev/rdsk/c2t0d0 /dev/rdsk/c2t0d0s2
/dev/dsk/c2t0d0s1 /dev/dsk/c2t0d0s3 /dev/rdsk/c2t0d0s1 /dev/rdsk/c2t0d0s3
disk 2 0/1/1/0.1.0 sdisk CLAIMED DEVICE HP 73.4GST373454LC /dev/dsk/c2t1d0 /dev/dsk/c2t1d0s2 /dev/rdsk/c2t1d0 /dev/rdsk/c2t1d0s2
/dev/dsk/c2t1d0s1 /dev/dsk/c2t1d0s3 /dev/rdsk/c2t1d0s1 /dev/rdsk/c2t1d0s3
disk 3 0/2/1/0.1.0.0.0.0.0 sdisk CLAIMED DEVICE HITACHI DF600F /dev/dsk/c4t0d0 /dev/rdsk/c4t0d0
disk 5 0/2/1/0.1.0.0.0.0.1 sdisk CLAIMED DEVICE HITACHI DF600F /dev/dsk/c4t0d1 /dev/rdsk/c4t0d1
disk 7 0/2/1/0.1.0.0.0.0.2 sdisk CLAIMED DEVICE HITACHI DF600F /dev/dsk/c4t0d2 /dev/rdsk/c4t0d2
disk 18 0/2/1/0.1.0.0.0.0.3 sdisk CLAIMED DEVICE HITACHI DF600F /dev/dsk/c4t0d3 /dev/rdsk/c4t0d3
disk 12 0/5/1/0.1.0.0.0.0.0 sdisk CLAIMED DEVICE HITACHI DF600F /dev/dsk/c9t0d0 /dev/rdsk/c9t0d0
disk 14 0/5/1/0.1.0.0.0.0.1 sdisk CLAIMED DEVICE HITACHI DF600F /dev/dsk/c9t0d1 /dev/rdsk/c9t0d1
disk 16 0/5/1/0.1.0.0.0.0.2 sdisk CLAIMED DEVICE HITACHI DF600F /dev/dsk/c9t0d2 /dev/rdsk/c9t0d2
disk 19 0/5/1/0.1.0.0.0.0.3 sdisk CLAIMED DEVICE HITACHI DF600F /dev/dsk/c9t0d3 /dev/rdsk/c9t0d3

# bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 2097152 219752 1862792 11% /
/dev/vg00/lvol1 524288 186064 335632 36% /stand
/dev/vg00/lvol8 9224192 1895488 7273104 21% /var
/dev/vg00/lvol7 6144000 2546952 3569000 42% /usr
/dev/vg00/lvol6 2097152 1406536 686328 67% /tmp
/dev/vg00/lvol5 6144000 2178056 3934984 36% /opt
/dev/vg00/lvol4 2097152 22472 2058520 1% /home
/dev/vg01/oracle 10469376 3632257 6411951 36% /oracle
/dev/vg02/prod1 41910272 26195211 14732931 64% /prod1
/dev/vg02/prod2 41926656 4125980 35438138 10% /prod2
/dev/vg03/prod03 41926656 13510720 26640042 34% /prod3
/dev/vg03/prod04 41943040 20675307 19942875 51% /prod4
/dev/vg04/backup1 20480000 2141998 17191932 11% /backup1
/dev/vg04/backup2 20480000 3289884 16115763 17% /backup2

# strings lvmtab
/dev/vg00
MC_E
/dev/dsk/c2t0d0s2
/dev/dsk/c2t1d0s2
/dev/vg01
/dev/dsk/c4t0d0
/dev/dsk/c9t0d0
/dev/vg02
/dev/dsk/c9t0d1
/dev/dsk/c4t0d1
/dev/vg03
/dev/dsk/c4t0d2
/dev/dsk/c9t0d2
/dev/vg04
/dev/dsk/c9t0d3

# dmesg

Jun 24 09:45
Found adjacent data tr. Growing size. 0x34c9000 -> 0x74c9000.
Pinned PDK malloc pool: base: 0xe000000100b37000 size=119588K
Loaded ACPI revision 2.0 tables.

MFS is defined: base= 0xe000000100b37000 size= 1184 KB


Unpinned PDK malloc pool: base: 0xe000000108000000 size=180224K
NOTICE: cachefs_link(): File system was registered at index 4.
NOTICE: nfs3_link(): File system was registered at index 7.
0 sba
0/0 lba
0/0/1/0 UsbOhci
0/0/1/1 UsbOhci
0/0/1/2 UsbEhci
0/0/2/0 side_multi
0/0/2/0.0 side
0/0/2/0.0.0 tgt
0/0/2/0.0.0.0 sdisk
0/0/2/0.0.7 tgt
0/0/2/0.0.7.0 sctl
0/0/2/0.1 side
0/0/2/0.1.7 tgt
0/0/2/0.1.7.0 sctl
0/1 lba
Initializing the Ultra320 SCSI Controller at 0/1/1/0. Controller firmware version is 01.03.35.65
0/1/1/0 mpt
SCSI Ultra320 0/1/1/0 instance 2: The driver is now online
0/1/1/0.0 tgt
0/1/1/0.0.0 sdisk
0/1/1/0.1 tgt
0/1/1/0.1.0 sdisk
Initializing the Ultra320 SCSI Controller at 0/1/1/1. Controller firmware version is 01.03.35.65
0/1/1/1 mpt
0/1/2/0 iether
0/1/2/1 iether
0/2 lba
0/2/1/0 td
SCSI Ultra320 0/1/1/1 instance 3: The driver is now online
0/1/1/0.7 tgt
0/1/1/0.7.0 sctl
td: claimed Tachyon XL2 Fibre Channel Mass Storage card at 0/2/1/0
0/3 lba
0/3/1/0 igelan
0/4 lba
0/4/1/0 iether
0/4/1/1 iether
0/5 lba
0/5/1/0 td
td: claimed Tachyon XL2 Fibre Channel Mass Storage card at 0/5/1/0
0/6 lba
0/6/1/0 asio0
0/6/1/1 asio0
0/6/2/0 gvid_core
120 processor
121 processor
250 pdh
250/0 ipmi
250/1 asio0
250/2 asio0
250/3 acpi_node
0/2/1/0.1 fcp
0/5/1/0.1 fcp
0/5/1/0.1.0.0.0 fcparray
0/5/1/0.1.0.255.0 fcpdev
0/2/1/0.1.0.0.0 fcparray
0/2/1/0.1.0.255.0 fcpdev
0/2/1/0.1.4.255.0 fcpdev
0/5/1/0.1.0.255.0.0 tgt
0/5/1/0.1.0.0.0.0 tgt
0/5/1/0.1.0.255.0.0.0 sctl
0/5/1/0.1.0.0.0.0.0 sdisk
0/2/1/0.1.0.0.0.0 tgt
0/2/1/0.1.0.255.0.0 tgt
0/2/1/0.1.0.0.0.0.0 sdisk
0/2/1/0.1.0.255.0.0.0 sctl
0/2/1/0.1.4.255.0.0 tgt
0/5/1/0.1.0.0.0.0.1 sdisk
0/2/1/0.1.0.0.0.0.1 sdisk
0/5/1/0.1.0.0.0.0.2 sdisk
0/2/1/0.1.0.0.0.0.2 sdisk
0/1/1/1.7 tgt
0/1/1/1.7.0 sctl
Boot device's HP-UX HW path is: 0/1/1/0.0.0
iether0: INITIALIZING HP PCI-X 1000Base-T Dual-port Built-in at hardware path 0/1/2/0
iether1: INITIALIZING HP PCI-X 1000Base-T Dual-port Built-in at hardware path 0/1/2/1
igelan2: INITIALIZING HP A6825-60101 PCI 1000Base-T Adapter at hardware path 0/3/1/0
iether3: INITIALIZING HP A7012-60001 PCI/PCI-X 1000Base-T Dual-port Adapter at hardware path 0/4/1/0
iether4: INITIALIZING HP A7012-60001 PCI/PCI-X 1000Base-T Dual-port Adapter at hardware path 0/4/1/1

System Console is on the Built-In Serial Interface
Logical volume 64, 0x3 configured as ROOT
Logical volume 64, 0x2 configured as SWAP
Logical volume 64, 0x2 configured as DUMP
Swap device table: (start & size given in 512-byte blocks)
entry 0 - major is 64, minor is 0x2; start = 0, size = 33816576
Dump device table: (start & size given in 1-Kbyte blocks)
entry 0000000000000000 - major is 31, minor is 0x20000; start = 1039220, size = 16908284
Starting the STREAMS daemons-phase 1
Create STCP device files
Starting the STREAMS daemons-phase 2
$Revision: vmunix: B11.23_LR FLAVOR=perf Fri Aug 29 22:35:38 PDT 2003 $
Memory Information:
physical page size = 4096 bytes, logical page size = 4096 bytes
Physical: 8376616 Kbytes, lockable: 6102064 Kbytes, available: 7099332 Kbytes


SCSI: Read error -- dev: b 31 0x090100, errno: 126, resid: 1024,
blkno: 8, sectno: 16, offset: 8192, bcount: 1024.
LVM: Performed a switch for Lun ID = 0 (pv = 0xe0000001681f4000), from raw device 0x1f090100 (with priority: 0, and current flags: 0x40) to raw device 0x1f040100 (with priority: 1, and current flags: 0x0).
LVM: VG 64 0x020000: PVLink 31 0x090100 Failed! The PV is still accessible.
LVM: Performed a switch for Lun ID = 0 (pv = 0xe0000001681f4000), from raw device 0x1f040100 (with priority: 1, and current flags: 0x0) to raw device 0x1f090100 (with priority: 0, and current flags: 0x80).
LVM: VG 64 0x020000: PVLink 31 0x090100 Recovered.

SCSI: Read error -- dev: b 31 0x040000, errno: 126, resid: 1024,
blkno: 8, sectno: 16, offset: 8192, bcount: 1024.
LVM: Performed a switch for Lun ID = 0 (pv = 0xe000000162a6c000), from raw device 0x1f040000 (with priority: 0, and current flags: 0x40) to raw device 0x1f090000 (with priority: 1, and current flags: 0x0).
LVM: VG 64 0x010000: PVLink 31 0x040000 Failed! The PV is still accessible.
LVM: Performed a switch for Lun ID = 0 (pv = 0xe000000168272000), from raw device 0x1f040200 (with priority: 0, and current flags: 0x40) to raw device 0x1f090200 (with priority: 1, and current flags: 0x0).
LVM: VG 64 0x030000: PVLink 31 0x040200 Failed! The PV is still accessible.
LVM: Performed a switch for Lun ID = 0 (pv = 0xe000000162a6c000), from raw device 0x1f090000 (with priority: 1, and current flags: 0x0) to raw device 0x1f040000 (with priority: 0, and current flags: 0x80).
LVM: VG 64 0x010000: PVLink 31 0x040000 Recovered.
LVM: Performed a switch for Lun ID = 0 (pv = 0xe000000168272000), from raw device 0x1f090200 (with priority: 1, and current flags: 0x0) to raw device 0x1f040200 (with priority: 0, and current flags: 0x80).
LVM: VG 64 0x030000: PVLink 31 0x040200 Recovered.
0/2/1/0.1.4.255.0.0.1 stape
0/2/1/0.1.0.0.0.0.3 sdisk
0/5/1/0.1.0.0.0.0.3 sdisk
LVM: VG 64 0x040000: Flushing the deferred attach list.

SCSI: Read error -- dev: b 31 0x090100, errno: 126, resid: 1024,
blkno: 8, sectno: 16, offset: 8192, bcount: 1024.
LVM: Performed a switch for Lun ID = 0 (pv = 0xe0000001681f4000), from raw device 0x1f090100 (with priority: 0, and current flags: 0x40) to raw device 0x1f040100 (with priority: 1, and current flags: 0x0).
LVM: VG 64 0x020000: PVLink 31 0x090100 Failed! The PV is still accessible.
LVM: VG 64 0x040000: Lost quorum.
This may block configuration changes and I/Os. In order to reestablish quorum at least 1 of the following PVs (represented by current link) must become available:
<31 0x090300>
LVM: VG 64 0x040000: PVLink 31 0x090300 Failed! The PV is not accessible.
LVM: VG 64 0x040000: PVLink 31 0x090300 Wrong Disk!
msgcnt 1 vxfs: mesg 025: vx_wsuper - /dev/vg04/backup1 file system super-block update failed
msgcnt 2 vxfs: mesg 031: vx_disable - /dev/vg04/backup1 file system disabled
msgcnt 3 vxfs: mesg 025: vx_wsuper - /dev/vg04/backup2 file system super-block update failed
msgcnt 4 vxfs: mesg 031: vx_disable - /dev/vg04/backup2 file system disabled
LVM: Performed a switch for Lun ID = 0 (pv = 0xe0000001681f4000), from raw device 0x1f040100 (with priority: 1, and current flags: 0x0) to raw device 0x1f090100 (with priority: 0, and current flags: 0x80).
LVM: VG 64 0x020000: PVLink 31 0x090100 Recovered.
LVM: VG 64 0x040000: PVLink 31 0x090300 Wrong Disk!
LVM: VG 64 0x040000: PVLink 31 0x090300 Wrong Disk!
LVM: VG 64 0x040000: PVLink 31 0x090300 Wrong Disk!
LVM: VG 64 0x040000: PVLink 31 0x090300 Wrong Disk!
LVM: VG 64 0x040000: PVLink 31 0x090300 Wrong Disk!
LVM: VG 64 0x040000: PVLink 31 0x090300 Wrong Disk!
LVM: VG 64 0x040000: Data in one or more logical volumes on PV 31 0x090300 was lost when the disk was replaced.
This occured because the disk contained the only copy of the data.
Prior to using these logical volumes, restore the data from backup.
LVM: vg[4] pv[0] Bad MCR, resyncing all LVs on the PV
msgcnt 5 vxfs: mesg 001: vx_nospace - /dev/vg03/prod04 file system full (1 block extent)
msgcnt 7 vxfs: mesg 001: vx_nospace - /dev/vg03/prod04 file system full (1 block extent)
msgcnt 9 vxfs: mesg 001: vx_nospace - /dev/vg03/prod04 file system full (1 block extent)
msgcnt 11 vxfs: mesg 001: vx_nospace - /dev/vg03/prod04 file system full (1 block extent)
msgcnt 13 vxfs: mesg 001: vx_nospace - /dev/vg03/prod04 file system full (1 block extent)
msgcnt 15 vxfs: mesg 001: vx_nospace - /dev/vg03/prod04 file system full (1 block extent)
msgcnt 17 vxfs: mesg 001: vx_nospace - /dev/vg03/prod04 file system full (3 block extent)
msgcnt 19 vxfs: mesg 001: vx_nospace - /dev/vg03/prod04 file system full (1 block extent)
msgcnt 21 vxfs: mesg 001: vx_nospace - /dev/vg03/prod04 file system full (1 block extent)
msgcnt 23 vxfs: mesg 001: vx_nospace - /dev/vg03/prod04 file system full (1 block extent)
msgcnt 25 vxfs: mesg 001: vx_nospace - /dev/vg03/prod04 file system full (1 block extent)
msgcnt 27 vxfs: mesg 001: vx_nospace - /dev/vg03/prod04 file system full (1 block extent)
msgcnt 29 vxfs: mesg 001: vx_nospace - /dev/vg03/prod04 file system full (1 block extent)
msgcnt 31 vxfs: mesg 001: vx_nospace - /dev/vg03/prod04 file system full (3 block extent)
msgcnt 33 vxfs: mesg 001: vx_nospace - /dev/vg03/prod04 file system full (2 block extent)
msgcnt 35 vxfs: mesg 001: vx_nospace - /dev/vg03/prod04 file system full (1 block extent)
msgcnt 37 vxfs: mesg 001: vx_nospace - /dev/vg03/prod04 file system full (1 block extent)
msgcnt 39 vxfs: mesg 001: vx_nospace - /dev/vg03/prod04 file system full (1 block extent)
msgcnt 41 vxfs: mesg 001: vx_nospace - /dev/vg03/prod04 file system full (1 block extent)
msgcnt 43 vxfs: mesg 001: vx_nospace - /dev/vg03/prod04 file system full (1 block extent)
msgcnt 45 vxfs: mesg 001: vx_nospace - /dev/vg03/prod04 file system full (1 block extent)
msgcnt 47 vxfs: mesg 001: vx_nospace - /dev/vg03/prod04 file system full (1 block extent)
msgcnt 49 vxfs: mesg 001: vx_nospace - /dev/vg03/prod04 file system full (1 block extent)


# vgdisplay /dev/vg04
--- Volume groups ---
VG Name /dev/vg04
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 2
Open LV 2
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 10239
VGDA 2
PE Size (Mbytes) 4
Total PE 10238
Alloc PE 10000
Free PE 238
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

#/usr/sbin/vgdisplay -v /dev/vg04 |more
--- Volume groups ---
VG Name /dev/vg04
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 2
Open LV 2
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 10239
VGDA 2
PE Size (Mbytes) 4
Total PE 10238
Alloc PE 10000
Free PE 238
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0
--- Logical volumes ---
LV Name /dev/vg04/backup1
LV Status available/syncd
LV Size (Mbytes) 20000
Current LE 5000
Allocated PE 5000
Used PV 1
LV Name /dev/vg04/backup2
LV Status available/syncd
LV Size (Mbytes) 20000
Current LE 5000
Allocated PE 5000
Used PV 1

--- Physical volumes ---
PV Name /dev/dsk/c9t0d3
PV Status available
Total PE 10238
Free PE 238
Autoswitch On

/home/nitink#
#/usr/sbin/vgdisplay -v /dev/vg03 |more
--- Volume groups ---
VG Name /dev/vg03
VG Write Access read/write
VG Status available, exclusive
Max LV 255
Cur LV 2
Open LV 2
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 5119
VGDA 2
PE Size (Mbytes) 16
Total PE 5119
Alloc PE 5119
Free PE 0
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0
--- Logical volumes ---
LV Name /dev/vg03/prod03
LV Status available/syncd
LV Size (Mbytes) 40944
Current LE 2559
Allocated PE 2559
Used PV 1
LV Name /dev/vg03/prod04
LV Status available/syncd
LV Size (Mbytes) 40960
Current LE 2560
Allocated PE 2560
Used PV 1

--- Physical volumes ---
PV Name /dev/dsk/c4t0d2
PV Name /dev/dsk/c9t0d2 Alternate Link
PV Status available
Total PE 5119
Free PE 0
Autoswitch On

/home/nitink#
#/usr/sbin/vgdisplay -v /dev/vg02 |more
--- Volume groups ---
VG Name /dev/vg02
VG Write Access read/write
VG Status available, exclusive
Max LV 255
Cur LV 2
Open LV 2
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 5119
VGDA 2
PE Size (Mbytes) 16
Total PE 5119
Alloc PE 5117
Free PE 2
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0
--- Logical volumes ---
LV Name /dev/vg02/prod1
LV Status available/syncd
LV Size (Mbytes) 40928
Current LE 2558
Allocated PE 2558
Used PV 1
LV Name /dev/vg02/prod2
LV Status available/syncd
LV Size (Mbytes) 40944
Current LE 2559
Allocated PE 2559
Used PV 1

--- Physical volumes ---
PV Name /dev/dsk/c9t0d1
PV Name /dev/dsk/c4t0d1 Alternate Link
PV Status available
Total PE 5119
Free PE 2
Autoswitch On

usr/sbin/vgdisplay -v /dev/vg01 |more
--- Volume groups ---
VG Name /dev/vg01
VG Write Access read/write
VG Status available, exclusive
Max LV 255
Cur LV 1
Open LV 1
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 1016
VGDA 2
PE Size (Mbytes) 16
Total PE 639
Alloc PE 639
Free PE 0
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0
--- Logical volumes ---
LV Name /dev/vg01/oracle
LV Status available/syncd
LV Size (Mbytes) 10224
Current LE 639
Allocated PE 639
Used PV 1

--- Physical volumes ---
PV Name /dev/dsk/c4t0d0
PV Name /dev/dsk/c9t0d0 Alternate Link
PV Status available
Total PE 639
Free PE 0
Autoswitch On
# ls -l /dev/*/group | grep '0x040000'
crw-rw-rw- 1 root sys 64 0x040000 Jan 29 10:17 /dev/vg04/group
/home/nitink#
# ls -l /dev/vg04/group | grep '0x040000'
crw-rw-rw- 1 root sys 64 0x040000 Jan 29 10:17 /dev/vg04/group
#
# dd if=/dev/dsk/c9t0d3 of=/dev/null count=1000
1000+0 records in
1000+0 records out
#
# diskinfo /dev/rdsk/c9t0d3
SCSI describe of /dev/rdsk/c9t0d3:
vendor: HITACHI
product id: DF600F
type: direct access
size: 41943040 Kbytes
bytes per sector: 512
#
# pvdisplay /dev/dsk/c9t0d3
--- Physical volumes ---
PV Name /dev/dsk/c9t0d3
VG Name /dev/vg04
PV Status available
Allocatable yes
VGDA 2
Cur LV 2
PE Size (Mbytes) 4
Total PE 10238
Free PE 238
Allocated PE 10000
Stale PE 0
IO Timeout (Seconds) default
Autoswitch On

[Darrel Louis]

Hi nitin,

It could be that the messages where caused by a timeout issue, or a issue with the connection to the switch.

Can you check if you have the following patches installed:
PHCO_34421
PHKL_33361
PHKL_33312
PHKL_33827

LVM: VG 64 0x030000: PVLink 31 0x040200 Recovered.
vg03 disk c4t0d2

LVM: VG 64 0x020000: PVLink 31 0x090100 Failed! The PV is still accessible.
vg02 disk c9t0d1

LVM: VG 64 0x040000: PVLink 31 0x090300 Wrong Disk!
vg04 disk c9t0d3

Darrel

[Mridul Shrivastava]

I would suggest you to run Patch Assessment
(please find below the procedure for the same):

1) open the page ==> http://www1.itrc.hp.com/service/patch/assessSystemsPage.do?BC=main|
2) click on "select a system to assess or upload new system information"
3) Download the script "swainv ". Right click on this and save to the system
4) Transfer this file to the HP-UX system. If ftp is used, then select binary mode of transfer.
5) Go to the directory in HP-UX system where the script "swainv " is loaded and give the executable permission by using the command " chmod 700 swainv "
6) Execute the script as " ./swainv "
7) This will generate a file called "inventory.xml" in current directory.
8) Transfer the file to the system which has got access to ITRC site.
9) Use the above link " http://www1.itrc.hp.com/service/patch/assessSystemsPage.do?BC=main " again and upload the "inventory.xml" to ITRC by clicking "submit".
10) click on " display candidate patches". The site will give the recommendations.
11) click on "create assessment report". This will give consolidated report.
12) Check at the line " Patches recommended by InstalledPatchAnalyzer (summary)" for recommendations.

[Florian Heigl (new acc)]

LVM: VG 64 0x020000: PVLink 31 0x090100 Recovered.
LVM: VG 64 0x040000: PVLink 31 0x090300 Wrong Disk!

These messages definitely show that one of the LUNs turned into something different than HP-UX expected.

I would suggest You verify Your SAN config and/or open a call with HP, aiming to gather all the PV IDs on-disk and in-kernel and then figure out where that missing LUN went.