topic Re: PV unavailable in Operating System - HP-UX

PV unavailable

vijay alur alur — Tue, 15 Nov 2011 08:29:34 GMT

Hi All,

i have a VG, having 2 PV's c6t12d0 and c7t12d0. As per my knowledge these 2 disks are actually 1 physical disk and accessed by 2 paths since the t12 and d0 are same for both the PV. Also this VG has 2 PV group each disk making one PVG. The problem is that the disk c7t12d0 is unavailable in the vgdisplay o/p and pvdisplay o/p. when i give pvdisplay -v c7t12d0 it shows only some of the PE's as stale while rest of the PE's are current. I suspect that one of the two FC Link to the disk is failed. Just want to know how can i recover from this situation?

Please refer output of pvdisplay and vgdisplay

Please help.....

Regards,

Vijay

Re: PV unavailable

Duncan Edmonstone — Fri, 18 Jun 2021 10:56:36 GMT

>> As per my knowledge these 2 disks are actually 1 physical disk and accessed by 2 paths since the t12 and d0 are same for both the PV.

No. They are physically seperate disks/LUNs. If they were alternate links, the vgdisplay command on the volume group would indicate that, and it doesn't.

They are seperate disks, and the logical volume appears to be mirrored between them (you can see this as the LV has 2 x the physical extents to logical extents, and the pvdisplay shows exactly half the physical extents on the failing disk)

Are you sure these disks are out of FC controllers? You should identify the HW path of the failing disk using lssf:

lssf /dev/dsk/c7t12d0

and then try and see if that path is out of one of your FC controllers:

ioscan -funC fc

Then use fcmsutil to check if the FC controller is up and active or not:

fcmsutil /dev/td0

for example... you should also of course, consult the "When Good Disks Go Bad" technical paper:

http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c01911837/c01911837.pdf

Re: PV unavailable

vijay alur alur — Tue, 15 Nov 2011 09:31:45 GMT

Hi Duncan,

Thanks very much for your reply.

Yes these disk come out of a FC Controller. Below is the output.

# lssf /dev/dsk/c7t12d0
sdisk card instance 7 SCSI target 12 SCSI LUN 0 section 0 at address 0/3/1/0/4/0.8.0.255.0.12.0 /dev/dsk/c7t12d0

# ioscan -fnCfc
Class I H/W Path Driver S/W State H/W Type Description
========================================================================
fc 0 0/2/1/0/4/0 fcd CLAIMED INTERFACE HP 2Gb PCI/PCI-X Fibre Channel FC/GigE Dual Port Combo Adapter
/dev/fcd0
fc 1 0/3/1/0/4/0 fcd CLAIMED INTERFACE HP 2Gb PCI/PCI-X Fibre Channel FC/GigE Dual Port Combo Adapter
/dev/fcd1

# /opt/fcms/bin/fcmsutil /dev/fcd1

Vendor ID is = 0x001077
Device ID is = 0x002312
PCI Sub-system Vendor ID is = 0x00103c
PCI Sub-system ID is = 0x0012c7
PCI Mode = PCI-X 133 MHz
ISP Code version = 3.2.171
ISP Chip version = 3
Topology = PRIVATE_LOOP
Link Speed = 2Gb
Local N_Port_id is = 0x000001
Previous N_Port_id is = 0x000001
Local Loop_id is = 125
N_Port Node World Wide Name = 0x50060b0000324d2f
N_Port Port World Wide Name = 0x50060b0000324d2e
Switch Port World Wide Name = N/A
Switch Node World Wide Name = N/A
Driver state = ONLINE
Hardware Path is = 0/3/1/0/4/0
Maximum Frame Size = 2048
Driver-Firmware Dump Available = NO
Driver-Firmware Dump Timestamp = N/A
Driver Version = @(#) libfcd.a HP Fibre Channel ISP 23xx Driver B.11.23.02 /ux/core/isu/FCD/kern/src/common/wsio/fcd_init.c:Aug 31 2004,13:48:17

The FC link is online, i suspect that the link through /dev/fcd1 had a fluctuation and after regaining connectivity this particular disk didn't ( c7t12d0 ) reinitialized and may need a reboot. What do you think?

is there any way i can probe and chk connectivity between /dev/fcd1 and c7t12do?? i guess there is a fcmsutil cmd for the same..

regards,

vijay

Re: PV unavailable

Duncan Edmonstone — Tue, 15 Nov 2011 09:40:33 GMT

vijay,

So the FC card is online, but is the disk?

Will it respond to a diskinfo command?

diskinfo /dev/rdsk/c7t12d0

Can you read from it?

dd if=/dev/rdsk/c7t12d0 of=/dev/null bs=8k count=1024

Can you read the whole thing (will take some time)

dd if=/dev/rdsk/c7t12d0 of=/dev/null bs=8k

If you get errors off that, are there other disks out this same controller in the same disk cab as c7t12d0 which are still working? We are trying to determine here if you have a failing disk, or failing controller on the disk cab (I'm guessing these a phyiscal disks and not LUNs give you are operating in private loop rather than on a point-to-point fabric)

Re: PV unavailable

vijay alur alur — Tue, 15 Nov 2011 10:12:56 GMT

hii Duncan,

Thanks again,

i am really enjoying this exchange of knowledge with you!!

diskinfo does not give the desired output.

# diskinfo /dev/rdsk/c7t12d0
diskinfo: can't open /dev/rdsk/c7t12d0: No such device or address

doing a dd on it wont cause any issue?? this disk is part of cluster.... do let me know if it is safe to do that?

Yes, the other disk connected to same controller are working fine.

i want you to have a look at this

disk 28 0/3/1/0/4/0.8.0.255.0.12.0 sdisk NO_HW DEVICE HP 36.4GST336607FC
/dev/dsk/c7t12d0 /dev/rdsk/c7t12d0

disk 27 0/2/1/0/4/0.8.0.255.0.12.0 sdisk CLAIMED DEVICE HP 36.4GST336607FC
/dev/dsk/c6t12d0 /dev/rdsk/c6t12d0

did you notice the hardware path above??its looks like same disk but through different FC path...

what are your thoughts??

Regards,

Vijay

Re: PV unavailable

Torsten. — Tue, 15 Nov 2011 11:26:46 GMT

You probably have a JBOD like DS2405 connected. To confirm, please send

# ioscan -fn

# echo "map" | cstm

Re: PV unavailable

vijay alur alur — Tue, 15 Nov 2011 12:02:34 GMT

Hi Torsten,

Thanks for replying.

It is a Disk array with FC Card. Dont know the complete details as the server is located at remote location.

I have attached the file with requested details.

Regards,

Vijay.

Re: PV unavailable

Torsten. — Tue, 15 Nov 2011 12:14:47 GMT

Could you now please do

# echo "sel dev 37;info;wait;il" | cstm

# echo "sel dev 58;info;wait;il" | cstm

to get more

information about the enclosure?

Re: PV unavailable

vijay alur alur — Tue, 15 Nov 2011 12:40:37 GMT

hi torsten,

just wanted to confirm these command wont cause any disruption to the running services??

Regards,

Vijay

Re: PV unavailable

Torsten. — Tue, 15 Nov 2011 12:43:56 GMT

No, this runs the information tool of the online diagnostics on the JBOD controllers.

Re: PV unavailable

vijay alur alur — Tue, 15 Nov 2011 13:05:42 GMT

Ok Trosten,

Please find attached the required output's.

Regards,

Vijay

Re: PV unavailable

Torsten. — Tue, 15 Nov 2011 12:58:00 GMT

You have 2 DS2405 disk enclosures, each with 15 disks. In the first enclosure there is a problem with disk in slot 2 (NO_HW), in the second with disk in slot 12 (FAILED). Check both and replace if needed.

Re: PV unavailable

vijay alur alur — Tue, 15 Nov 2011 13:08:29 GMT

Thanks a lot Torsten!!

Regards,

Vijay.

Re: PV unavailable

Torsten. — Tue, 15 Nov 2011 13:11:34 GMT

Note: if you replace the disks online, they will remain in NO_HW status until you run fcmsutil with replace_dsk option. The syslog will tell you the details after an ioscan.

Example: fcmsutil /dev/fcd0 replace_dsk 0x0a

Re: PV unavailable

vijay alur alur — Tue, 15 Nov 2011 16:30:40 GMT

Hi Torsten,

i was just going through the cstm log's, in the first disk enclosure logs i.e. with device id 37, i cannot see any fault. where as in the second disk enclosure with device id 58, i can see a fault for disk in slot # 12. But as you said, both the disk enclosure's have faulty disks.

can you please help me with that???

By the way we just did a reboot for both the nodes in the cluster and now the disk is not seen in the ioscan output, but it is visible in the vgdisplay or /etc/lvmtab. it seems the metadata got updated after reboot.

Also i have one doubt about the device naming, i was in an assumption that the device file c6t12d0 and c7t12d0 are physically same disk but accessed through 2 different controller. my assumption was that t12d0 is same for both the disk that means its same disk but 2 device file for 2 different path's. Can you hlep ,me with this as well?

Thanks and Regards,

Vijay

Re: PV unavailable

Torsten. — Tue, 15 Nov 2011 20:57:40 GMT

Look at your ioscan:

target      5  0/2/1/0/4/0.8.0.255.0.2     tgt        NO_HW       DEVICE

target     32  0/3/1/0/4/0.8.0.255.0.12    tgt        NO_HW       DEVICE

You will also notice this:

disk       18  0/3/1/0/4/0.8.0.255.0.7.0   sdisk      CLAIMED     DEVICE       HPQ     BD03659532

disk       17  0/2/1/0/4/0.8.0.255.0.7.0   sdisk      CLAIMED     DEVICE       HP 36.4GST336607FC

Same slot, different disks. This means you have 2 enclosures. It is a cluster? "cmviewcl" will tell. For this reason you have a lvmpvg file! I assume you have 2 nodes, first connected to controller A of the first JBOD, second node to controller B, vice versa for the second JBOD.

After a reboot, a "NO_HW" device will disappear, this is not nice, but normal.

Do a "diskinfo" for both present disks in a same slotnumber and you will see different serial numbers.

Re: PV unavailable

vijay alur alur — Wed, 16 Nov 2011 13:56:25 GMT

Hi Torsten,

if i have a failed disk and i am replacing it with a new disk. how can i give the same disk name as the old disk device file name to the new disk's device file name?? the OS Version is hpux 11.23.

Please suggest??

Regards,

Vijay.

Re: PV unavailable

Torsten. — Wed, 16 Nov 2011 18:07:50 GMT

The device special file name is bound to the slot, so it will be the same after replacement.

Re: PV unavailable

vijay alur alur — Wed, 16 Nov 2011 18:29:19 GMT

OK, thanks Torsten.

then after replacing the disk i do not need to make any chnages in the cluster config file since the disk that got failed is also part of second cluster lock disk along with being a data disk.....

we have setup a plan for the disk replacment and would be scheduling the disk replacement activity.

Your suggestion were very helpful and knowledgable for me.Thanks again!!

Regards,

Vijay

Re: PV unavailable

Torsten. — Wed, 16 Nov 2011 18:46:32 GMT

Follow this guide to replace the disk:

When_Good_Disks_Go_Bad_WP
http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c01911837/c01911837.pdf

When it comes to "vgchange -a y ..." replace this by "vgchange -a e ..." because of the cluster. Perform the command from the node that owns the VG. You did not mention this is a cluster, but I assumed this already.