Re: HP FlexFabric 10Gb 2-port 536FLB Adapter Drivers and Firmware on rhel 7.2

patrickkangethe · ‎09-06-2016

Hi all,

I have installed RHEL 7.2 on HP bl660c gen 9 server with HP FlexFabric 10Gb 2-port 536FLB Adapter.

Additionally I have installed the kmod-netxtreme2-7.13.04.1-1.rhel7u2.x86_64.rpm driver and updated the server with online firmware upgrade utility CP027566.scexe successfully and performed a reboot.

Apparently the server doesn't do a FLOGI into the brocade 300b switch.

additional steps include;

followed the following link;

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Storage_Administration_Guide/fcoe-config.html - for interfaces eno49 and en050.

[root@CAD-MVCSVR3 ~]# fcoeadm -i
No FCoE interfaces created.

attached is an output of other actions done.

Your help will be appreciated.

Sebastian.Koehler · ‎09-06-2016

Hi Patrick,

what Interconnect modules are used in the c7000 Enclosure and what is the firmware of the modules? What is the FOS on the Brocade B300 SAN Switches? Do you see the FC ports from the Enclosure logging into a F-Port (with or without NPIV) on the switch?

This should be the right driver and readme for the CNA and RHEL 7.2.

HPE QLogic NX2 1/10/20 GbE Multifunction Drivers for Red Hat Enterprise Linux 7 x86_64
http://h20565.www2.hpe.com/hpsc/swd/public/detail?swItemId=MTX_a0a5853c691e471ab4c66c9d5b

Please use the Accept Solution button next to my post and assign a KUDO (thumbs up icon) if this works for you.

Regards,
Sebastian Koehler

patrickkangethe · ‎09-06-2016

Hello The C7000 have 2 X Virtual Connect 10/24 with firmware 4.5 on VC. Even the OA is also 4.5.

I will confirm the FOS for brocade but to answer, one of the BL 660 is running Windows Server 2012 R2 and this is working perfectly and able to login to brocade and hence storage. We are thus somehow sure its something about redhat drivers.

[root@server3 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eno49
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME=eno49
UUID=11450180-7d58-42df-9dd6-267d208c5e21
DEVICE=eno49
ONBOOT=no
DNS1=192.168.2.50
IPADDR=192.168.2.46
PREFIX=24
GATEWAY=192.168.2.1
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
[root@server3 ~]# vim /etc/sysconfig/network-scripts/ifcfg-eno49
[root@server3 ~]# vim /etc/sysconfig/network-scripts/ifcfg-eno50
[root@server3 ~]# cp /etc/fcoe/cfg-eth0 /etc/fcoe/cfg-eno49
cp: cannot stat ‘/etc/fcoe/cfg-eth0’: No such file or directory
[root@server3 ~]# cd /etc/fcoe/
[root@server3 fcoe]# ls
cfg-ethx
[root@server3 fcoe]# cd /
[root@server3 /]# cp /etc/fcoe/cfg-ethx /etc/fcoe/cfg-eno49
[root@server3 /]# cat /etc/fcoe/cfg-eno49
## Type: yes/no
## Default: no
# Enable/Disable FCoE service at the Ethernet port
# Normally set to "yes"
FCOE_ENABLE="yes"

## Type: yes/no
## Default: no
# Indicate if DCB service is required at the Ethernet port
# Normally set to "yes"
DCB_REQUIRED="yes"

## Type: yes/no
## Default: no
# Indicate if VLAN discovery should be handled by fcoemon
# Normally set to "yes"
AUTO_VLAN="yes"

## Type: fabric/vn2vn
## Default: fabric
# Indicate the mode of the FCoE operation, either fabric or vn2vn
# Normally set to "fabric"
MODE="fabric"

## Type: yes/no
## Default: no
# Indicate whether to run a FIP responder for VLAN discovery in vn2vn mode
#FIP_RESP="yes"
[root@server3 /]# systemctl status lldpad
? lldpad.service - Link Layer Discovery Protocol Agent Daemon.
Loaded: loaded (/usr/lib/systemd/system/lldpad.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2016-09-06 05:45:50 EDT; 32min ago
Main PID: 4862 (lldpad)
CGroup: /system.slice/lldpad.service
+-4862 /usr/sbin/lldpad -t

Sep 06 05:45:50 server3 systemd[1]: Started Link Layer Discovery Protocol Agent Daemon..
Sep 06 05:45:50 server3 systemd[1]: Starting Link Layer Discovery Protocol Agent Daemon....
[root@server3 /]# systemctl restart lldpad
[root@server3 /]# systemctl status lldpad
? lldpad.service - Link Layer Discovery Protocol Agent Daemon.
Loaded: loaded (/usr/lib/systemd/system/lldpad.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2016-09-06 06:18:02 EDT; 1s ago
Main PID: 5612 (lldpad)
CGroup: /system.slice/lldpad.service
+-5612 /usr/sbin/lldpad -t

Sep 06 06:18:02 server3 systemd[1]: Started Link Layer Discovery Protocol Agent Daemon..
Sep 06 06:18:02 server3 systemd[1]: Starting Link Layer Discovery Protocol Agent Daemon....
[root@server3 /]# dcbtool sc eno49 dcb on
Command: Set Config
Feature: DCB State
Port: eno49
Status: Successful
[root@server3 /]# dcbtool sc eno49 app:fcoe e:1
Command: Set Config
Feature: Application FCoE
Port: eno49
Status: Successful
[root@server3 /]# dcbtool sc eno50 dcb on
Command: Set Config
Feature: DCB State
Port: eno50
Status: Successful
[root@server3 /]# dcbtool sc eno50 app:fcoe e:1
Command: Set Config
Feature: Application FCoE
Port: eno50
Status: Successful
[root@server3 /]# ifconfig eno49
eno49: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.2.46 netmask 255.255.255.0 broadcast 192.168.2.255
inet6 fe80::eeb1:d7ff:fe9d:c880 prefixlen 64 scopeid 0x20<link>
ether ec:b1:d7:9d:c8:80 txqueuelen 1000 (Ethernet)
RX packets 4603 bytes 444189 (433.7 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 566 bytes 88354 (86.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 16 memory 0xac800000-acffffff

[root@server3 /]# ifconfig eno50
eno50: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether ec:b1:d7:9d:c8:88 txqueuelen 1000 (Ethernet)
RX packets 3730 bytes 357991 (349.6 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 17 memory 0xad800000-adffffff

[root@server3 /]# systemctl restart fcoe
[root@server3 /]# systemctl status fcoe
? fcoe.service - Open-FCoE Inititator.
Loaded: loaded (/usr/lib/systemd/system/fcoe.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2016-09-06 06:21:41 EDT; 5s ago
Process: 5665 ExecStart=/usr/sbin/fcoemon $FCOEMON_OPTS (code=exited, status=0/SUCCESS)
Process: 5663 ExecStartPre=/sbin/modprobe -qa $SUPPORTED_DRIVERS (code=exited, status=0/SUCCESS)
Main PID: 5667 (fcoemon)
CGroup: /system.slice/fcoe.service
+-5667 /usr/sbin/fcoemon --syslog

Sep 06 06:21:45 server3 fcoemon[5667]: eno49, WARNING: APP:0 priority (0x00) doesn't intersect with PFC priority (0x00)
Sep 06 06:21:45 server3 fcoemon[5667]: eno49, WARNING: DCB may be configured incorrectly
Sep 06 06:21:46 server3 fcoemon[5667]: eno49, WARNING: FCoE operational mode is false
Sep 06 06:21:46 server3 fcoemon[5667]: 0x00 - none
Sep 06 06:21:46 server3 fcoemon[5667]: eno49, WARNING: APP:0 operational mode is false
Sep 06 06:21:46 server3 fcoemon[5667]: 0x00 - none
Sep 06 06:21:46 server3 fcoemon[5667]: eno49, WARNING: PFC operational mode is false
Sep 06 06:21:46 server3 fcoemon[5667]: 0x00 - none
Sep 06 06:21:46 server3 fcoemon[5667]: eno49, WARNING: APP:0 priority (0x00) doesn't intersect with PFC priority (0x00)
Sep 06 06:21:46 server3 fcoemon[5667]: eno49, WARNING: DCB may be configured incorrectly
[root@server3 /]# systemctl status fcoe -l
? fcoe.service - Open-FCoE Inititator.
Loaded: loaded (/usr/lib/systemd/system/fcoe.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2016-09-06 06:21:41 EDT; 33s ago
Process: 5665 ExecStart=/usr/sbin/fcoemon $FCOEMON_OPTS (code=exited, status=0/SUCCESS)
Process: 5663 ExecStartPre=/sbin/modprobe -qa $SUPPORTED_DRIVERS (code=exited, status=0/SUCCESS)
Main PID: 5667 (fcoemon)
CGroup: /system.slice/fcoe.service
+-5667 /usr/sbin/fcoemon --syslog

Sep 06 06:21:50 server3 fcoemon[5667]: eno49, WARNING: APP:0 priority (0x00) doesn't intersect with PFC priority (0x00)
Sep 06 06:21:50 server3 fcoemon[5667]: eno49, WARNING: DCB may be configured incorrectly
Sep 06 06:21:52 server3 fcoemon[5667]: eno49, WARNING: FCoE operational mode is false
Sep 06 06:21:52 server3 fcoemon[5667]: 0x00 - none
Sep 06 06:21:52 server3 fcoemon[5667]: eno49, WARNING: APP:0 operational mode is false
Sep 06 06:21:52 server3 fcoemon[5667]: 0x00 - none
Sep 06 06:21:52 server3 fcoemon[5667]: eno49, WARNING: PFC operational mode is false
Sep 06 06:21:52 server3 fcoemon[5667]: 0x00 - none
Sep 06 06:21:52 server3 fcoemon[5667]: eno49, WARNING: APP:0 priority (0x00) doesn't intersect with PFC priority (0x00)
Sep 06 06:21:52 server3 fcoemon[5667]: eno49, WARNING: DCB may be configured incorrectly
[root@server3 /]# fcoeadm -i
No FCoE interfaces created.
[root@server3 /]# systemctl status dcbd
? dcbd.service
Loaded: not-found (Reason: No such file or directory)
Active: inactive (dead)
[root@server3 /]# systemctl start dcbd
Failed to start dcbd.service: Unit dcbd.service failed to load: No such file or directory.
[root@server3 /]# cp /etc/fcoe/cfg-ethx /etc/fcoe/cfg-eno50
[root@server3 /]# cd /etc/fcoe/
[root@server3 fcoe]# ls
cfg-eno49 cfg-eno50 cfg-ethx
[root@server3 fcoe]# cat cfg-eno50
## Type: yes/no
## Default: no
# Enable/Disable FCoE service at the Ethernet port
# Normally set to "yes"
FCOE_ENABLE="yes"

## Type: yes/no
## Default: no
# Indicate if DCB service is required at the Ethernet port
# Normally set to "yes"
DCB_REQUIRED="yes"

## Type: yes/no
## Default: no
# Indicate if VLAN discovery should be handled by fcoemon
# Normally set to "yes"
AUTO_VLAN="yes"

## Type: fabric/vn2vn
## Default: fabric
# Indicate the mode of the FCoE operation, either fabric or vn2vn
# Normally set to "fabric"
MODE="fabric"

## Type: yes/no
## Default: no
# Indicate whether to run a FIP responder for VLAN discovery in vn2vn mode
#FIP_RESP="yes"
[root@server3 fcoe]# dcbtool sc ethX app:fcoe e:1
Command: Set Config
Feature: Application FCoE
Port: ethX
Status: Device not capable
[root@server3 fcoe]# dcbtool sc eno50 app:fcoe e:1
Command: Set Config
Feature: Application FCoE
Port: eno50
Status: Successful
[root@server3 fcoe]# fcoeadm -i
No FCoE interfaces created.
[root@server3 fcoe]# man fcoeadm
[root@server3 fcoe]# fcoeadm -c eno49
fcoeadm: Connection already created on interface eno49
Try 'fcoeadm --help' for more information.
[root@server3 fcoe]# fcoeadm -c eno50
[root@server3 fcoe]# fcoeadm -i
Description: BCM57840 NetXtreme II 10/20-Gigabit Ethernet
Revision: 11
Manufacturer: Broadcom Corporation
Serial Number: ECB1D79DC880
Driver: bnx2x 1.710.51-0
Number of Ports: 1

Symbolic Name: fcoe v0.1 over eno50
OS Device Name: host2
Node Name: 0x0000000000000000
Port Name: 0x0000000000000000
FabricName: 0x0000000000000000
Speed: 10 Gbit
Supported Speed: 1 Gbit, 10 Gbit
MaxFrameSize: 1452
FC-ID (Port ID): 0xFFFFFFFF
State: Offline

[root@server3 mnt]# ls
CP027566.scexe kmod-netxtreme2-7.13.04.1-1.rhel7u2.x86_64.rpm
[root@server3 mnt]# ./CP027566.scexe
Copyright (c) 2015 Hewlett Packard Enterprise Development LP
HPE QLogic NX2 Online Firmware Upgrade Utility for Linux x86_64

Found HP FlexFabric 10Gb 2-port 536FLB Adapter MAC: ECB1D79DC880
Update Boot Code 7.10.39 to 7.13.23 - y/n/q (y):
Update PXE/UEFI 7.10.33/7.10.55 to 7.13.2/7.13.16 - y/n/q (y):
Update FCOE 7.10.32 to 7.13.3 - y/n/q (y):
Update CCM 7.10.31 to 7.13.3 - y/n/q (y):
Update BRCM_ISCSI 7.10.33 to 7.13.1 - y/n/q (y):

The Firmware Upgrade will take several minutes. Please be patient.

Selecting HP FlexFabric 10Gb 2-port 536FLB Adapter MAC: ECB1D79DC880

Upgrading Boot Code 7.10.39 to 7.13.23