" PV Link failed , The PV is not accessible" message in syslog

[Ranoop Nambiar]

Hello,
I am getting these errors in syslog consciously… Its an RX 6600 server with 8 SAS disks, running on 11.23

Jan 11 07:48:44 u01vmd01 vmunix: SCSI: Read error -- dev: b 31 0x035000, errno: 126, resid: 1024,
Jan 11 07:48:44 u01vmd01 vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 1024.
Jan 11 07:48:44 u01vmd01 vmunix: LVM: VG 64 0x010000: PVLink 31 0x035000 Failed! The PV is not accessible.
Jan 11 07:56:22 u01vmd01 vmunix: LVM: VG 64 0x010000: PVLink 31 0x035000 Recovered.
Jan 11 08:02:19 u01vmd01 vmunix:
Jan 11 08:02:19 u01vmd01 vmunix: SCSI: Read error -- dev: b 31 0x035000, errno: 126, resid: 1024,
Jan 11 08:02:19 u01vmd01 vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 1024.
Jan 11 08:02:19 u01vmd01 vmunix: LVM: VG 64 0x010000: PVLink 31 0x035000 Failed! The PV is not accessible.
Jan 11 08:07:38 u01vmd01 vmunix: LVM: VG 64 0x010000: PVLink 31 0x035000 Recovered.
Jan 11 08:11:03 u01vmd01 vmunix: LVM: Failed to automatically resync PV 1f035000 error: 5

History =

We had replaced a disk /dev/dsk/c3t6d0 (0/4/1/0.0.0.6.0) . The above errors started during the lvresync, and still coming. But all the Lvs got synced well. All the PV‘s showing status as available .But still the errors are there..

This is the device, which gives error.. diskinfo comes out with full size on this!!

root@u01vmd01:/dev/dsk > ls -ltra |grep -i 0x035000
brw-r----- 1 bin sys 31 0x035000 Feb 8 2008 c3t5d0
disk 6 0/4/1/0.0.0.5.0 sdisk CLAIMED DEVICE HP DG072A9BB7
/dev/dsk/c3t5d0 /dev/rdsk/c3t5d0

Any thing else can be done to get ride of these errors…?

Thanks.
ranoop

[likid0]

It looks like that disk has problems.

Try running around the hole disk with dd:

For example, the following shows a successful read of the first 64 megabytes of the disk:
# dd if=/dev/rdsk/c3t5d0 of=/dev/null bs=1024k count=64
64+0 records in
64+0 records out
The following shows an unsuccessful read of the whole disk:
# dd if=/dev/rdsk/c3t5d0 of=/dev/null bs=1024k
dd read error: I/O error
0+0 records in 0+0 records out

Allways a good doc to have:

http://docs.hp.com/en/5991-1236/When_Good_Disks_Go_Bad_WP.pdf

[R.K. #]

Hi Ranoop,

Please perform a FULL "dd" read test on BOTH the disks, to make sure BOTH can be read properly.

PV is not accessible but gets recovered in few minutes everytime. What activity is done on this PV, are LVs on this PV accessed very frequently??

[sujit kumar singh]

hi


>>>Jan 11 07:48:44 u01vmd01 vmunix: SCSI: Read error -- dev: b 31 0x035000, errno: 126, resid: 1024,
Jan 11 07:48:44 u01vmd01 vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 1024.



there is an evident read error on the disk at read time, doubting that you can ask for replacement of the disk.

>>>>root@u01vmd01:/dev/dsk > ls -ltra |grep -i 0x035000
brw-r----- 1 bin sys 31 0x035000 Feb 8 2008 c3t5d0


regards

[Vishu]

Hi,

Your SCSI error shows the Read error on the disk. DD command is the best way to get thr proper read in/out from the disk. so, perform a full dd on that disk.

If the error is frequent as you pasted. I would suggest go for that disk replacement.

[Ranoop Nambiar]

Thanks all. I did dd on both the disks..

root@u01vmd01:/ > dd if=/dev/rdsk/c3t5d0 of=/dev/null bs=1024k count=64
64+0 records in
64+0 records out
root@u01vmd01:/ > dd if=/dev/rdsk/c3t6d0 of=/dev/null bs=1024k count=64
64+0 records in
64+0 records out
root@u01vmd01:/ >


Its doesnâ t return anything abnormal. Other thing I have noticed is the time stamp on the syslog/CSTM logs, it looks like.. They have been triggered in the same time on everyday. I have attached the syslsog here. Pls have a look..

Thanks,
Ranoop

[Vishu]

Hi Ranoop,

it is good if you run full dd without count.

Your syslog says repetitive read/write alerts on this disk. Better go for the disk replacement for this disk.

[Ranoop Nambiar]

Hi, If you notice the syslog.. Its throwing these errors for the other disks as well.:-)

How do we disable CSTM from polling the disks regularly ?

Any way.. Will do a full DD now.. WIll it make any implications on the sys performance?


Thanks
Ranoop

[Torsten.]

When you replaced the disk, did you redirect the device file???

[Ranoop Nambiar]

I did that 3 days back..yes i have done the below one, after the disk was replaced

sasmgr replace_tgt -D /dev/sasd0 -q old_dev=/dev/dsk/c3t6d0 -q new_tgt_hwpath=0/4/1/0.0.0.8.0

[Md. Minhaz Khan]

Hi Ranoop,

Please give me the below command outputs:

1)diskinfo -v /dev/rdsk/c3t5d0
2)pvdisplay -v /dev/dsk/c3t5d0|grep "PV Status"


Thanks
Minahz

[AVV]

Hi Ranoop,

Running dd fully on both disks may take longer time. I assume the disk is bad and go ahead for a replacement since the logs clearly says there is a read error.

#vgsync vgxx

Try and see , how long it takes you to the out.

[Ranoop Nambiar]

dd shows no eerrors.. and the messages are logged in a particaular time.

any views or thoughts ?

nohup dd if=/dev/rdsk/c3t5d0 of=/dev/null bs=1024k &
root@u01vmd01:/ > cat nohup.out
70007+1 records in
70007+1 records out
root@u01vmd01:/ >

oot@u01vmd01:/ > diskinfo -v /dev/rdsk/c3t5d0
SCSI describe of /dev/rdsk/c3t5d0:
vendor: HP
product id: DG072A9BB7
type: direct access
size: 71687369 Kbytes
bytes per sector: 512
rev level: HPD0
blocks per disk: 143374738
ISO version: 0
ECMA version: 0
ANSI version: 5
removable media: no
response format: 2
(Additional inquiry bytes: (32)33 (33)36 (34)35 (35)50 (36)36 (37)43 (38)30 (39)39 (40)54 (41)42 (42)48 (43)0 (44)0 (45)0 (46)0 (47)0 (48)0 (49)0 (50)0 (51)0 (52)0 (53)c (54)0 (55)0 (56)75 (57)3 (58)0 (59)3 (60)24 (61)0 (62)0 (63)0 (64)0 (65)0 (66)0 (67)0 (68)0 (69)0 (70)0 (71)0 (72)0 (73)0 (74)0 (75)0 (76)0 (77)0 (78)0 (79)0 (80)0 (81)0 (82)0 (83)0 (84)0 (85)0 (86)0 (87)0 (88)0 (89)0 (90)0 )
root@u01vmd01:/ >


Cheers,
ranoop

[Amitav]

Hi,
If the errors have occured on a particular time and diskinfo and dd are running perfectly, i guess the disk is OK.

Can u confirm from SAN end, whether any activity was been carried out by them...??