identify the right disk

[Christopher Pereira]

I have a SAN configuration with an HDS 9980 disk array can you help me identify which disk had an issue here?

Feb 22 09:51:51 sahblpd1 vmunix: SCSI: Read error -- dev: b 31 0x166500, errno: 126, resid: 2048,
Feb 22 09:51:51 sahblpd1 vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 2048.
Feb 22 09:51:51 sahblpd1 vmunix: LVM: vg[19]: pvnum=6 (dev_t=0x1f166500) is POWERFAILED
Feb 22 09:51:51 sahblpd1 vmunix: SCSI: Read error -- dev: b 31 0x166500, errno: 126, resid: 8192,
Feb 22 09:51:51 sahblpd1 vmunix: blkno: 51400, sectno: 102800, offset: 52633600, bcount: 8192.
Feb 22 09:51:51 sahblpd1 vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000071b7f000), from raw device 0x1f166500
(with priority: 0, and current flags: 0xc0) to raw device 0x1f0e6500 (with priority: 1, and current flags: 0x0).
Feb 22 09:51:51 sahblpd1 vmunix:
Feb 22 09:51:56 sahblpd1 vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000071b7f000), from raw device 0x1f0e6500
(with priority: 1, and current flags: 0x40) to raw device 0x1f106500 (with priority: 2, and current flags: 0x0).
Feb 22 09:51:56 sahblpd1 vmunix: LVM: Recovered Path (device 0x1f166500) to PV 6 in VG 19.
Feb 22 09:51:56 sahblpd1 vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x0000000071b7f000), from raw device 0x1f106500
(with priority: 2, and current flags: 0x0) to raw device 0x1f166500 (with priority: 0, and current flags: 0x80).
Feb 22 09:51:56 sahblpd1 vmunix: LVM: Recovered Path (device 0x1f0e6500) to PV 6 in VG 19.
Feb 22 09:51:57 sahblpd1 vmunix: LVM: Restored PV 6 to VG 19.

[Sridhar Bhaskarla · Accepted solution]

Hi

in 0xabcd00 - ab togehter is controller
c represents the target and
d is the lun number

So dev: b 31 0x166500 translates to

c16t6d5

In other way you can do a ll /dev/dsk |grep 0x166500

-Sri

[Robert-Jan Goossens]

Hi,

# ls -la /dev/dsk | grep 0x166500

should give you the right disk.

Hope this helps,
Robert-Jan

[Sridhar Bhaskarla]

Oops. correction. It should be in hexa. So

0x166500 shoudl get translated to c22t6d5 if my hex-to-dec math is correct.

-Sri

[Steven E. Protter]

This code helps identify this stuff for you.

#!/bin/sh
#
# disk.status Created 8/29/2003
# Steven E. Protter
#
# Designed to check that all disks in /etc/lvmtab are alive
# send an alert email if that is not the case.
#


#
# set up the temporary file
#

tfile=/tmp/disk.status.$$
dfile=/tmp/disk.report.$$

export tfile

# Create the temporary file...
/usr/bin/touch $tfile

/usr/bin/strings /etc/lvmtab | grep dsk > $tfile
while read -r a
do

disk=$a
echo "About the check disk ${a} "
/usr/sbin/pvdisplay ${disk} >> $dfile
# /usr/sbin/pvdisplay ${disk}
rc=$?
echo "disk $a return code: $rc"
if [ $rc -ne 0 ]
then
disktrigger="y"
diskaffected=$a
export disktigger diskaffected
fi

done < $tfile



# errs=`wc -l /tmp/checkwardware | awk {'print $1'}`
if [ "$disktrigger" = "y" ]
export disktigger diskaffected
fi

done < $tfile



# errs=`wc -l /tmp/checkwardware | awk {'print $1'}`
if [ "$disktrigger" = "y" ]
then
# echo "Errors ${errs}"
# /usr/sbin/dmesg > /tmp/checkwardware
cat $dfile | mailx -s "Hardware disk error detectede $diskaffected" stevenpro
tter@somedomain.org
cat $dfile | mailx -s "Hardware disk error detectede $diskaffected" sprotter@
yourdomain.com
/usr/bin/llp -f stock -d konica1 $tfile
fi


SEP

[Christopher Pereira]

Thanks guys....Both methods work just fine.
sahblpd1->ls -la /dev/dsk | grep 0x166500
brw-r----- 1 bin sys 31 0x166500 Apr 16 2003 c22t6d5
sahblpd1->ll /dev/dsk |grep 0x166500
brw-r----- 1 bin sys 31 0x166500 Apr 16 2003 c22t6d5

c16t6d5 is an alternate path for c22t6d5.

[Stuart Abramson_2]

You should assign points (not to me..)

..Stuart

[Christopher Pereira]

Steve your script does not work too well in clustered environments since several volume groups in the lvmtab are not activated....(I received about 700 emails for non activated disk) there are also some syntax errors that I had to fix....thanks. Chris.

[Steven E. Protter]

Thanks for the heads up on the script.

I'll stop recommending it for clustered environments. If you could give me some of the errors though, I might be able to adapt the script to not bother with deactivated volume groups.

You could turn off the email feature and the script will identify the bad disk along with all those deactivated ones.

I do apologize for the emails. I truly had no idea that deactivated volume groups would blow the script up since all of ours are active.

We are doing Service Guard Clustering later this year so I'll bet I have a better version of the script by fall.

SEP

[Christopher Pereira]

Hi Steve,

The errors were as follows:
pvdisplay: Warning: couldn't query all of the physical volumes.
pvdisplay: Couldn't retrieve the names of the physical volumes
belonging to volume group "/dev/vgapp01".
pvdisplay: Cannot display physical volume "/dev/dsk/c23t3d0".
+ rc=1
+ echo disk /dev/dsk/c23t3d0 return code: 1
disk /dev/dsk/c23t3d0 return code: 1
+ [ 1 -ne 0 ]
+ disktrigger=y
+ diskaffected=/dev/dsk/c23t3d0
+ export disktrigger diskaffected
+ [ y = y ]
+ mailx -s Hardware disk error detectede /dev/dsk/c23t3d0 chris.pereira@eds.com
+ cat /tmp/disk.report.3677
+ read -r a
+ disk=/dev/dsk/c23t3d1
+ echo About the check disk /dev/dsk/c23t3d1
About the check disk /dev/dsk/c23t3d1
+ /usr/sbin/pvdisplay /dev/dsk/c23t3d1
+ 1>> /tmp/disk.report.3677

pvdisplay: Warning: couldn't query physical volume "/dev/dsk/c20t2d6":
The specified path does not correspond to physical volume attached to
this volume group
pvdisplay: Warning: couldn't query physical volume "/dev/dsk/c20t2d7":
The specified path does not correspond to physical volume attached to
this volume group
pvdisplay: Warning: couldn't query physical volume "/dev/dsk/c20t3d0":
The specified path does not correspond to physical volume attached to
this volume group
pvdisplay: Warning: couldn't query physical volume "/dev/dsk/c20t3d1":
The specified path does not correspond to physical volume attached to
this volume group
pvdisplay: Warning: couldn't query physical volume "/dev/dsk/c20t3d2":
The specified path does not correspond to physical volume attached to
this volume group
pvdisplay: Warning: couldn't query physical volume "/dev/dsk/c20t3d3":
The specified path does not correspond to physical volume attached to
this volume group

No probs on the emails...I selected all and deleted them....

Thank you...
Chris.

[Steven E. Protter]

Thanks Chris.

I'll have to set up test conditions to duplicate yours to fix this script..

Hard to believe you have 700 disks in inactive volume groups.

The quick fix would be to cut a temporary file and list the bad disks in there.

When the new version is ready you'll find out at itrc.

Thanks for the points, hoped the script would work out better.

SEP

[Christopher Pereira]

Sure Steve...the reason we have so many disk's is the SAN configuration with zoning had several paths from the server to this disk, thus even though each volume group only contains about 60 disk 5 additional redundant paths make it 300 paths for that one volume group alone...Thanks Chris.