PV unavailable

[vijay alur alur]

Hi All,

i have a VG, having 2 PV's c6t12d0 and c7t12d0. As per my knowledge these 2 disks are actually 1 physical disk and accessed by 2 paths since the t12 and d0 are same for both the PV. Also this VG has 2 PV group each disk making one PVG. The problem is that the disk c7t12d0 is unavailable in the vgdisplay o/p and pvdisplay o/p. when i give pvdisplay -v c7t12d0 it shows only some of the PE's as stale while rest of the PE's are current. I suspect that one of the two FC Link to the disk is failed. Just want to know how can i recover from this situation?

Please refer output of pvdisplay and vgdisplay

Please help.....

Regards,

Vijay

[Duncan Edmonstone]

>> As per my knowledge these 2 disks are actually 1 physical disk and accessed by 2 paths since the t12 and d0 are same for both the PV.

No. They are physically seperate disks/LUNs. If they were alternate links, the vgdisplay command on the volume group would indicate that, and it doesn't.

They are seperate disks, and the logical volume appears to be mirrored between them (you can see this as the LV has 2 x the physical extents to logical extents, and the pvdisplay shows exactly half the physical extents on the failing disk)

Are you sure these disks are out of FC controllers? You should identify the HW path of the failing disk using lssf:

lssf /dev/dsk/c7t12d0

and then try and see if that path is out of one of your FC controllers:

ioscan -funC fc

Then use fcmsutil to check if the FC controller is up and active or not:

fcmsutil /dev/td0

for example... you should also of course, consult the "When Good Disks Go Bad" technical paper:

http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c01911837/c01911837.pdf

[vijay alur alur]

Hi Duncan,

Thanks very much for your reply.

Yes these disk come out of a FC Controller. Below is the output.

# lssf /dev/dsk/c7t12d0
sdisk card instance 7 SCSI target 12 SCSI LUN 0 section 0 at address 0/3/1/0/4/0.8.0.255.0.12.0 /dev/dsk/c7t12d0

# ioscan -fnCfc
Class I H/W Path Driver S/W State H/W Type Description
========================================================================
fc 0 0/2/1/0/4/0 fcd CLAIMED INTERFACE HP 2Gb PCI/PCI-X Fibre Channel FC/GigE Dual Port Combo Adapter
/dev/fcd0
fc 1 0/3/1/0/4/0 fcd CLAIMED INTERFACE HP 2Gb PCI/PCI-X Fibre Channel FC/GigE Dual Port Combo Adapter
/dev/fcd1

# /opt/fcms/bin/fcmsutil /dev/fcd1

Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012c7
PCI Mode = PCI-X 133 MHz
ISP Code version = 3.2.171
ISP Chip version = 3
Topology = PRIVATE_LOOP
Link Speed = 2Gb
Local N_Port_id is = 0x000001
Previous N_Port_id is = 0x000001
Local Loop_id is = 125
N_Port Node World Wide Name = 0x50060b0000324d2f
N_Port Port World Wide Name = 0x50060b0000324d2e
Switch Port World Wide Name = N/A
Switch Node World Wide Name = N/A
Driver state = ONLINE
Hardware Path is = 0/3/1/0/4/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.23.02 /ux/core/isu/FCD/kern/src/common/wsio/fcd_init.c:Aug 31 2004,13:48:17

The FC link is online, i suspect that the link through /dev/fcd1 had a fluctuation and after regaining connectivity this particular disk didn't ( c7t12d0 ) reinitialized and may need a reboot. What do you think?

is there any way i can probe and chk connectivity between /dev/fcd1 and c7t12do?? i guess there is a fcmsutil cmd for the same..

regards,

vijay

[Duncan Edmonstone]

vijay,

So the FC card is online, but is the disk?

Will it respond to a diskinfo command?

diskinfo /dev/rdsk/c7t12d0

Can you read from it?

dd if=/dev/rdsk/c7t12d0 of=/dev/null bs=8k count=1024

Can you read the whole thing (will take some time)

dd if=/dev/rdsk/c7t12d0 of=/dev/null bs=8k

If you get errors off that, are there other disks out this same controller in the same disk cab as c7t12d0 which are still working? We are trying to determine here if you have a failing disk, or failing controller on the disk cab (I'm guessing these a phyiscal disks and not LUNs give you are operating in private loop rather than on a point-to-point fabric)

[vijay alur alur]

hii Duncan,

Thanks again,

i am really enjoying this exchange of knowledge with you!!

diskinfo does not give the desired output.

# diskinfo /dev/rdsk/c7t12d0
diskinfo: can't open /dev/rdsk/c7t12d0: No such device or address

doing a dd on it wont cause any issue?? this disk is part of cluster.... do let me know if it is safe to do that?

Yes, the other disk connected to same controller are working fine.

i want you to have a look at this

disk 28 0/3/1/0/4/0.8.0.255.0.12.0 sdisk NO_HW DEVICE HP 36.4GST336607FC
/dev/dsk/c7t12d0 /dev/rdsk/c7t12d0

disk 27 0/2/1/0/4/0.8.0.255.0.12.0 sdisk CLAIMED DEVICE HP 36.4GST336607FC
/dev/dsk/c6t12d0 /dev/rdsk/c6t12d0

did you notice the hardware path above??its looks like same disk but through different FC path...

what are your thoughts??

Regards,

Vijay

[Torsten.]

You probably have a JBOD like DS2405 connected. To confirm, please send

# ioscan -fn

# echo "map" | cstm

[vijay alur alur]

Hi Torsten,

Thanks for replying.

It is a Disk array with FC Card. Dont know the complete details as the server is located at remote location.

I have attached the file with requested details.

Regards,

Vijay.

[Torsten.]

Could you now please do

 # echo "sel dev 37;info;wait;il" | cstm

 # echo "sel dev 58;info;wait;il" | cstm

to get more

information about the enclosure?

[vijay alur alur]

hi torsten,

just wanted to confirm these command wont cause any disruption to the running services??

Regards,

Vijay

[Torsten.]

No, this runs the information tool of the online diagnostics on the JBOD controllers.

[vijay alur alur]

Ok Trosten,

Please find attached the required output's.

Regards,

Vijay

[Torsten.]

You have 2 DS2405 disk enclosures, each with 15 disks. In the first enclosure there is a problem with disk in slot 2 (NO_HW), in the second with disk in slot 12 (FAILED). Check both and replace if needed.

[vijay alur alur]

Thanks a lot Torsten!!

Regards,

Vijay.

[Torsten.]

Note: if you replace the disks online, they will remain in NO_HW status until you run fcmsutil with replace_dsk option. The syslog will tell you the details after an ioscan.

Example: fcmsutil /dev/fcd0 replace_dsk 0x0a

[vijay alur alur]

Hi Torsten,

i was just going through the cstm log's, in the first disk enclosure logs i.e. with device id 37, i cannot see any fault. where as in the second disk enclosure with device id 58, i can see a fault for disk in slot # 12. But as you said, both the disk enclosure's have faulty disks.

can you please help me with that???

By the way we just did a reboot for both the nodes in the cluster and now the disk is not seen in the ioscan output, but it is visible in the vgdisplay or /etc/lvmtab. it seems the metadata got updated after reboot. 

Also i have one doubt about the device naming, i was in an assumption that the device file c6t12d0 and c7t12d0 are physically same disk but accessed through 2 different controller. my assumption was that t12d0 is same for both the disk that means its same disk but 2 device file for 2 different path's. Can you hlep ,me with this as well?

Thanks and Regards,

Vijay

[Torsten.]

Look at your ioscan:

target      5  0/2/1/0/4/0.8.0.255.0.2     tgt        NO_HW       DEVICE

target     32  0/3/1/0/4/0.8.0.255.0.12    tgt        NO_HW       DEVICE

You will also notice this:

disk       18  0/3/1/0/4/0.8.0.255.0.7.0   sdisk      CLAIMED     DEVICE       HPQ     BD03659532

disk       17  0/2/1/0/4/0.8.0.255.0.7.0   sdisk      CLAIMED     DEVICE       HP 36.4GST336607FC

Same slot, different disks. This means you have 2 enclosures. It is a cluster?  "cmviewcl" will tell. For this reason you have a lvmpvg file! I assume you have 2 nodes, first connected to controller A of the first JBOD, second node to controller B, vice versa for the second JBOD.

After a reboot, a "NO_HW" device will disappear, this is not nice, but normal.

Do a "diskinfo" for both present disks in a same slotnumber and you will see different serial numbers.

[vijay alur alur]

Hi Torsten,

if i have a failed disk and i am replacing it with a new disk. how can i give the same disk name as the old disk device file name to the new disk's device file name?? the OS Version is hpux 11.23.

Please suggest??

Regards,

Vijay.

[Torsten.]

The device special file name is bound to the slot, so it will be the same after replacement.

[vijay alur alur]

OK, thanks Torsten.

then after replacing the disk i do not need to make any chnages in the cluster config file since the disk that got failed is also part of second cluster lock disk along with being a data disk.....

we have setup a plan for the disk replacment and would be scheduling the disk replacement activity.

Your suggestion were very helpful and knowledgable for me.Thanks again!!

Regards,

Vijay

[Torsten.]

Follow this guide to replace the disk:

When_Good_Disks_Go_Bad_WP
http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c01911837/c01911837.pdf



When it comes to "vgchange -a y ..." replace this by "vgchange -a e ..." because of the cluster. Perform the command from the node that owns the VG. You did not mention this is a cluster, but I assumed this already.

[vijay alur alur]

Hi Torsten,

Disk replacement activity completed successfully!!Thanks a lot! just wanted to review what we did with you. We prepared a game plan as follows

1. remove failed disk and replace it with new disk in the same slot.

2. identify the new disk ioscan -fnCdisk

3. vgcfgrestore

4. vgsync

5. check the sync completion with lvdisplay -v /dev/vg01/u90

6. after sync completed, generated the map file and copied it to passive node and imported the vg on passive node after ioscan and detecting the new disk on passive node

The issue we faced was we were not able to perform step #4 with a error "Cannotquery the VG the PV is connected to" later we changed the activation mode from Exclusive mode to normal active mode with -a y  option. After doing this i was able to perform vgsync and go ahead with other steps successfully.

So i wasn't much clear that why i wasn't able to go ahead with vgsync with vg activated in exclusive mode?

may be vg in exclusive mode doesnt allow me to do vgsync which vg -a y does.

Also note that when we took a reboot initially the lvmtab entry for failed disk was gone but the LV still had the mirror failed disk details,so there was an inconsistency in LVM.

Regards,

Vijay.

[Torsten.]

What is the OS version (date of the patch bundle)? It is probably very old, because on current versions you cannot do a vgcfgrestore without doing a "pvchange -a n ..." first. So your issue could be related to some missing patches ...

[vijay alur alur]

Hi,

OS Version

# uname -a
HP-UX Hostname B.11.23 U 9000/800 4254030036 unlimited-user license

Patch Level

# swlist -l bundle |grep -i qpk
  QPKAPPS               B.11.23.1012.086a Applications Patches for HP-UX 11i v2, December 2010
  QPKBASE               B.11.23.1012.086a Base Quality Pack Bundle for HP-UX 11i v2, December 2010
# swlist -l bundle |grep -i hwe
  HWEnable11i           B.11.23.1012.085a Hardware Enablement Patches for HP-UX 11i v2, October 2010

i am not sure, i guess we were not even able to do vgcfgrestore only after vgchange -a y  or not...... i will confirm that to you.

Regards,

Vijay.