Volume group on SAN won't activate during boot

[vilius_m]

Hello,

I have 2 old rx3600 servers, and using fibre channel disks.
HP-UX 11iv3 patchlevel on OK server is really old like 2010 - that is working with no issues.
HP-UX 11iv3 patchlevel on Problem server is more recent like 2020 - that fails to activate SAN volume group during boottime, however when boot is complete everything is working with no issues - manual vg activation, lvs, fs etc...
Storage is IBM SVC.

Errors during boot on Problem server:

/sbin/bcheckrc:
Checking for LVM volume groups and Activating (if any exist)
Volume group "/dev/vg00" is already active on this system.
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/disk/disk12":
The path of the physical volume refers to a device that does not
exist, or is not configured into the kernel.
vgchange: Couldn't activate volume group "/dev/vgignite":
Quorum not present, or some physical volume(s) are missing.
vgchange: Warning: couldn't query physical volume "/dev/disk/disk12":
The volume group is not activated
vgchange: Warning: couldn't query the liLVM: VG 128 0x000000: Unable to initialize the latest status information.
st of physical volumes.
vgdisplay: Volume group not activated.
vgdisplay: Cannot display volume group "/dev/vgignite".

Fibre channel driver on Problem server:

# ioscan -fnC fc
Class     I  H/W Path  Driver S/W State   H/W Type     Description
================================================================
fc        0  0/3/1/0   fcd   CLAIMED     INTERFACE    HP AB378-60101      4Gb Single Port PCI/PCI-X Fibre Channel Adapter (FC Port 1)
                      /dev/fcd0
fc        1  0/7/1/0   fcd   CLAIMED     INTERFACE    HP AB378-60101      4Gb Single Port PCI/PCI-X Fibre Channel Adapter (FC Port 1)
                      /dev/fcd1

# ioscan -fnNC disk
Class     I  H/W Path  Driver S/W State   H/W Type     Description
===================================================================
......
disk     12  64000/0xfa00/0x6  esdisk   CLAIMED     DEVICE       IBM     2145
                      /dev/disk/disk12   /dev/rdisk/disk12

# fcmsutil /dev/fcd0 vpd
                V I T A L   P R O D U C T   D A T A
                ---------   -------------   -------
 Product Description    : HP PCI-X 2.0 4Gb Fibre Channel HBA
 Part number            : AB378-60101
 Engineering Date Code  : A-4713
 Part Serial number     : ********
 Misc. Information      : PW=7.5W;PCI=66MHZ;PCI-X=266MHZ
 Mfd. Date              : 4836
 Mfd. ID                : AB378-60101
 Check Sum              : 0x29
 EFI version            : 01.09
 ROM Firmware version   : 04.00.90
 BIOS version           : 01.26
 FCODE version          : 01.17
 Asset Tag              : NA

# fcmsutil /dev/fcd0
                           Vendor ID is = 0x1077
                           Device ID is = 0x2422
            PCI Sub-system Vendor ID is = 0x103C
                   PCI Sub-system ID is = 0x12D6
                               PCI Mode = PCI-X 266 MHz
                       ISP Code version = 5.6.5
                       ISP Chip version = 3
                               Topology = PTTOPT_FABRIC
                             Link Speed = 4Gb
                     Local N_Port_id is = ********
                  Previous N_Port_id is = None
            N_Port Node World Wide Name = ********
            N_Port Port World Wide Name = ********
            Switch Port World Wide Name = ********
            Switch Node World Wide Name = ********
              N_Port Symbolic Port Name = ********_fcd0
              N_Port Symbolic Node Name = ********_HP-UX_B.11.31
                           Driver state = ONLINE
                       Hardware Path is = 0/3/1/0
                     Maximum Frame Size = 2048
         Driver-Firmware Dump Available = NO
         Driver-Firmware Dump Timestamp = N/A
                                   TYPE = PFC
                         NPIV Supported = YES
                         Driver Version = @(#) fcd B.11.31.1905 Jan 20 2019

Fibre channel driver on OK server:

# ioscan -fnC fc
Class     I  H/W Path  Driver S/W State   H/W Type     Description
================================================================
fc        0  0/3/1/0   fcd   CLAIMED     INTERFACE    HP AB378-60101      4Gb Single Port PCI/PCI-X Fibre Channel Adapter (FC Port 1)
                      /dev/fcd0
fc        1  0/7/1/0   fcd   CLAIMED     INTERFACE    HP AB378-60101      4Gb Single Port PCI/PCI-X Fibre Channel Adapter (FC Port 1)
                      /dev/fcd1

# ioscan -fnNC disk
Class     I  H/W Path  Driver S/W State   H/W Type     Description
===================================================================
......
disk    124  64000/0xfa00/0x1b  esdisk   CLAIMED     DEVICE       IBM     2145
                      /dev/disk/disk124   /dev/rdisk/disk124

# fcmsutil /dev/fcd0 vpd
                V I T A L   P R O D U C T   D A T A
                ---------   -------------   -------
 Product Description    : HP PCI-X 2.0 4Gb Fibre Channel HBA
 Part number            : AB378-60101
 Engineering Date Code  : A-4713
 Part Serial number     : ********
 Misc. Information      : PW=7.5W;PCI=66MHZ;PCI-X=266MHZ
 Mfd. Date              : 4838
 Mfd. ID                : AB378-60101
 Check Sum              : 0x1e
 EFI version            : 01.09
 ROM Firmware version   : 04.00.90
 BIOS version           : 01.26
 FCODE version          : 01.17
 Asset Tag              : NA

# fcmsutil /dev/fcd0
                           Vendor ID is = 0x1077
                           Device ID is = 0x2422
            PCI Sub-system Vendor ID is = 0x103C
                   PCI Sub-system ID is = 0x12D6
                               PCI Mode = PCI-X 266 MHz
                       ISP Code version = 4.4.4
                       ISP Chip version = 3
                               Topology = PTTOPT_FABRIC
                             Link Speed = 4Gb
                     Local N_Port_id is = ********
                  Previous N_Port_id is = None
            N_Port Node World Wide Name = ********
            N_Port Port World Wide Name = ********
            Switch Port World Wide Name = ********
            Switch Node World Wide Name = ********
              N_Port Symbolic Port Name = ********_fcd0
              N_Port Symbolic Node Name = ********_HP-UX_B.11.31
                           Driver state = ONLINE
                       Hardware Path is = 0/3/1/0
                     Maximum Frame Size = 2048
         Driver-Firmware Dump Available = NO
         Driver-Firmware Dump Timestamp = N/A
                         Driver Version = @(#) fcd B.11.31.0909 Jun  8 2009

My guess is that newer drviver version is the problem here ?
I tried installing older pachlevel HP-UX - but oldest HP-UX provides is 1705, ant it has the same problem.

Any ideas how to solve this ?

Thanks,
Vilius M.

[Maisiefrost]

The new volume group does not get activated on boot, but can be activated manually after the server is up. In the example below, the lrg volume group was added along with the data logical volume associated with it.

[vilius_m]

What example below ?

[vilius_m · Accepted solution]

I was able to solve this - setting switch port from autonegotiate to fixed speed solved the problem.

[Sunitha_Mod]

Hello @vilius_m, 

Awesome! 

We are glad to know you were able to find the solution and we appreciate you for keeping us updated.