Can't get 10Gig networking working with BL860c i2 blade in a c3000 chassis

[Torsten.]

You have 4 NICs on such blade built-in.
In VC mode each interface is splitted into 4 sub-NIC, hence 16 NICS.
If you don't have VC modules, there is no counterpart for the NICs, hence your problem.

How about the ioscan now?

[Duncan Edmonstone]

Are you sure you have a 10GbE pass-thru module (538113-B21) and not another 1GbE pass-thru module? (406740-B21) - if you had the 1GbE module it wouldn't surprise me if you had to hard set both ports (NIC and switchj) to 1Gb to get them to work.

[DonOBrien]

I'm pretty sure I have the 10Gig Module.  I did connect to each module thru the c3000.   Here are my stats on the modules.   hp25 is the name of my c3000. 

Module 1 is used for the Mezz1 ports of my blades.  blade1 and blade2 are i2 and have 10gig nics,  blade3 is a i1 and has 1G nics.   (port 3,7,11,15 CU fp).

hp25> show interconnect list

Bay Interconnect Type     Manufacturer      Power   Health   UID  Management IP
--- ----------------- -------------------- ------- --------- --- ---------------
  1 Ethernet          HP                   On      OK        Off 0.0.0.0
  2 Ethernet          HP                   On      OK        Off 0.0.0.0
  3 Fibre Channel     BROCADE              On      OK        Off 192.168.17.31
  4 Fibre Channel     BROCADE              On      OK        Off 192.168.17.33
Totals: 4 interconnect modules installed, 4 powered on.

Module 1

System# info

System:                  HP 10Gbe Pass-Thru
Version:                 1.0.7.0
Board:                   HP 10GbE Pass-Thru Module
Serial:                  MY31375CUH

S
System# /port

[Port Menu]
    auto     -  Port autonegotiation configuration
    info     -  Port information

Port# info

         Uplink                            Downlink
Port Ena Link Speed 1G-Neg  Type  Approved Link Speed Neg
  1  ena down  --    on    SR SFP+   yes   down  --   on
  2  ena down  --    on    SR SFP+   yes   down  --   on
  3  ena down  --    on    CU SFP    yes   down  --   on
  4  dis down  --    on      N/A     no    down  --   on
  5  ena  up  10Gb   on    SR SFP+   yes    up  10Gb  on
  6  ena  up  10Gb   on    SR SFP+   yes    up  10Gb  on
  7  ena  up   1Gb   on    CU SFP    yes    up   1Gb  on
  8  dis down  --    on    SR SFP+   yes   down  --   on
  9  ena down  --    on    SR SFP+   yes   down  --   on
 10  ena down  --    on    SR SFP+   yes   down  --   on
 11  ena down  --    on    CU SFP    yes   down  --   on
 12  dis down  --    on      N/A     no    down  --   on
 13  ena  up  10Gb   on    SR SFP+   yes    up  10Gb  on
 14  ena  up  10Gb   on    SR SFP+   yes    up  10Gb  on
 15  ena down  --    on    CU SFP    yes   down  --   on
 16  dis down  --    on    SR SFP+   yes   down  --   on

Port#

Module 2

System# info

System:                  HP 10Gbe Pass-Thru
Version:                 1.0.7.0
Board:                   HP 10GbE Pass-Thru Module
Serial:                  MY31355BRV

System# /port

[Port Menu]
    auto     -  Port autonegotiation configuration
    info     -  Port information

Port# info

         Uplink                            Downlink
Port Ena Link Speed 1G-Neg  Type  Approved Link Speed Neg
  1  ena  up   1Gb   on    CU SFP    yes    up   1Gb  on
  2  ena  up   1Gb   on    CU SFP    yes    up   1Gb  on
  3  dis down  --    on      N/A     no    down  --   on
  4  dis down  --    on    CU SFP    yes   down  --   on
  5  ena down  --    on    CU SFP    yes   down  --   on
  6  ena down  --    on    CU SFP    yes   down  --   on
  7  dis down  --    on      N/A     no    down  --   on
  8  dis down  --    on    CU SFP    yes   down  --   on
  9  ena down  --    on    CU SFP    yes   down  --   on
 10  ena down  --    on    CU SFP    yes   down  --   on
 11  dis down  --    on      N/A     no    down  --   on
 12  dis down  --    on    CU SFP    yes   down  --   on
 13  ena down  --    on    CU SFP    yes   down  --   on
 14  ena down  --    on    CU SFP    yes   down  --   on
 15  dis down  --    on      N/A     no    down  --   on
 16  dis down  --    on    CU SFP    yes   down  --   on

Port#

[DonOBrien]

Hi Torsten,

Thank you again for all your help on this.   I tried to clear any VC mode setting, but either I did it incorrectly, or that isn't the issue.   I see what you mean about having 16 nics in the OS, but only four ports available on the passthru. 

After I ran the ILO clear commands (posted earlier) and reset of the server, and a reinstall of HPUX, the 16 nics persist.   I am guessing that the i2 blades may still be in VC mode, while the c3000 is not.    Is there a command that I can run to query the blade/ilo to confirm this?  

Here is my sytsem's current output for the network info.

# ioscan -funC lan
Class     I  H/W Path       Driver S/W State   H/W Type     Description
========================================================================
lan       0  0/0/0/3/0/0/0  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       1  0/0/0/3/0/0/1  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       2  0/0/0/3/0/0/2  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       3  0/0/0/3/0/0/3  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       4  0/0/0/3/0/0/4  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       5  0/0/0/3/0/0/5  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       6  0/0/0/3/0/0/6  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       7  0/0/0/3/0/0/7  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       8  0/0/0/4/0/0/0  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       9  0/0/0/4/0/0/1  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      10  0/0/0/4/0/0/2  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      11  0/0/0/4/0/0/3  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      12  0/0/0/4/0/0/4  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      13  0/0/0/4/0/0/5  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      14  0/0/0/4/0/0/6  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      15  0/0/0/4/0/0/7  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      16  0/0/0/5/0/0/0/2/0/0/0  iether   CLAIMED     INTERFACE    HP 447881-001  PCIe 4-port 1000Mb/s Mezzanine Adapter
lan      17  0/0/0/5/0/0/0/2/0/0/1  iether   CLAIMED     INTERFACE    HP 447881-001  PCIe 4-port 1000Mb/s Mezzanine Adapter
lan      18  0/0/0/5/0/0/0/4/0/0/0  iether   CLAIMED     INTERFACE    HP 447881-001  PCIe 4-port 1000Mb/s Mezzanine Adapter
lan      19  0/0/0/5/0/0/0/4/0/0/1  iether   CLAIMED     INTERFACE    HP 447881-001  PCIe 4-port 1000Mb/s Mezzanine Adapter
# lanscan
Hardware Station        Crd Hdw   Net-Interface  NM  MAC       HP-DLPI DLPI
Path     Address        In# State NamePPA        ID  Type      Support Mjr#
0/0/0/3/0/0/0 0x643150008850 0   UP    lan0 snap0     1   ETHER     Yes     119
0/0/0/3/0/0/1 0x643150008854 1   UP    lan1 snap1     2   ETHER     Yes     119
0/0/0/3/0/0/2 0x643150008851 2   UP    lan2 snap2     3   ETHER     Yes     119
0/0/0/3/0/0/3 0x643150008855 3   UP    lan3 snap3     4   ETHER     Yes     119
0/0/0/3/0/0/4 0x643150008852 4   UP    lan4 snap4     5   ETHER     Yes     119
0/0/0/3/0/0/5 0x643150008856 5   UP    lan5 snap5     6   ETHER     Yes     119
0/0/0/3/0/0/6 0x643150008853 6   UP    lan6 snap6     7   ETHER     Yes     119
0/0/0/3/0/0/7 0x643150008857 7   UP    lan7 snap7     8   ETHER     Yes     119
0/0/0/4/0/0/0 0x643150008858 8   UP    lan8 snap8     9   ETHER     Yes     119
0/0/0/4/0/0/1 0x64315000885C 9   UP    lan9 snap9     10  ETHER     Yes     119
0/0/0/4/0/0/2 0x643150008859 10  UP    lan10 snap10   11  ETHER     Yes     119
0/0/0/4/0/0/3 0x64315000885D 11  UP    lan11 snap11   12  ETHER     Yes     119
0/0/0/4/0/0/4 0x64315000885A 12  UP    lan12 snap12   13  ETHER     Yes     119
0/0/0/4/0/0/5 0x64315000885E 13  UP    lan13 snap13   14  ETHER     Yes     119
0/0/0/4/0/0/6 0x64315000885B 14  UP    lan14 snap14   15  ETHER     Yes     119
0/0/0/4/0/0/7 0x64315000885F 15  UP    lan15 snap15   16  ETHER     Yes     119
0/0/0/5/0/0/0/2/0/0/0 0x0025B3B44464 16  UP    lan16 snap16   17  ETHER     Yes     119
0/0/0/5/0/0/0/2/0/0/1 0x0025B3B44465 17  UP    lan17 snap17   18  ETHER     Yes     119
0/0/0/5/0/0/0/4/0/0/0 0x0025B3B44466 18  UP    lan18 snap18   19  ETHER     Yes     119
0/0/0/5/0/0/0/4/0/0/1 0x0025B3B44467 19  UP    lan19 snap19   20  ETHER     Yes     119
LinkAgg0 0x000000000000 900 DOWN  lan900 snap900 22  ETHER     Yes     119
LinkAgg1 0x000000000000 901 DOWN  lan901 snap901 23  ETHER     Yes     119
LinkAgg2 0x000000000000 902 DOWN  lan902 snap902 24  ETHER     Yes     119
LinkAgg3 0x000000000000 903 DOWN  lan903 snap903 25  ETHER     Yes     119
LinkAgg4 0x000000000000 904 DOWN  lan904 snap904 26  ETHER     Yes     119
# netstat -in
Name      Mtu  Network         Address         Ipkts              Ierrs Opkts              Oerrs Coll
lan1      1500 192.168.4.0     192.168.4.26    6                  0     6                  0     0
lan0      1500 192.168.5.0     192.168.5.26    0                  0     0                  0     0
lo0      32808 127.0.0.0       127.0.0.1       50770              0     50770              0     0
lan16     1500 192.168.16.0    192.168.16.184  574708             0     366149             0     0
#

[Torsten.]

Did you run the ILO reset command while the blade was powered off?
If not, please try.

[Torsten. · Accepted solution]

For clear VC mode command all blades must be powered off. Please try.

[DonOBrien]

Ok.  Just to be clear. 

1. I should power off all three blades in my c3000. 
2. Run the "CILON -iloKeepLan" command to clear the nvram, run on each blade. that is showing this problem (the i1 blade dosen't have 10G and its ports are working normally).   Run this command from the CM menu in the ILO.
3. Reset each of the three servers using the "reset server x" command in the OA.

When the servers come up and I run a "ioscan -funC lan" I should only see four 10G NICs and not 16.  

Does it sound like I have this correct? 

Thank you again Torsten. 

[Torsten.]

First power off the i2 blade only and do step 2 again.

[DonOBrien]

Hi Torsten,  Thank you again for all your help on this.   I have tried to reset the blade, but again, the behavior goes unchanged.   It looks like the blade is still behaving as if it is in VC mode.

I shutdown the blade, cleared the config on the ILO, and then reset the blade.  I powered it up, The UEFI menus did not have the local hard disks available to boot from.   So I booted from the DVD and just did a fresh install (hoped that would also allow for a fresh look at the NICs).    Once the HPUX was installed, I logged and I'm still seeing the 16 NICs for the on-board network adapters.

1. Shutdown the i2 blade.

[hp26ilo] CM:hpiLO-> ps


PS
For System Processor Status see the SS command.

For Power Supply and Fan status see the BladeSystem Onboard Administrator.
System Power state      : Off
System Power usage      : 34 Watts used in Auxiliary Mode
System Power allocation : 0 Watts allocated for Main Power
Ambient temperature     : 19 C
Temperature status      : Normal

[hp26ilo] CM:hpiLO->


Confirm that the i2 Blade in bay 1 off.

hp25> show server list

Bay iLO Name                      iLO IP Address  Status   Power   UID Partner
--- ----------------------------- --------------- -------- ------- --- -------
  1 hp26ilo                       192.168.16.177  OK       Off     Off
  2 hp27ilo                       192.168.16.178  OK       On      Off
  3 hp28ilo                       192.168.16.179  OK       On      Off
  4 [Absent]
  5 [Subsumed]
  6 [Subsumed]
  7 [Subsumed]
  8 [Absent]
Totals: 3 server blades installed, 2 powered on.

hp25>

------------
------------

Reset the ILO

[hp26ilo] CM:hpiLO-> CILON -iloKeepLan


CILON -iloKeepLan


WARNING: This is not an HP supported command and it is intended for internal
use only.  Execution of this command is not recommended and can lead to
unpredictable results.

The following command options are available:
   "CILON -ilo"         - Clears MP NVRAM and reboots MP.
   "CILON -iloKeepLan"  - Clears MP NVRAM, except for MP LAN settings,
                          and reboots MP.
   "CILON -sfw"         - Clears SFW NVRAM and reboots MP.
   "CILON -all default" - Clears both MP and SFW NVRAM and reboots MP.


This command will clear MP NVRAM and reboot MP.
All MP parameters will be reset to their default values.

MP LAN Settings will not be cleared.



Are you REALLY sure you want to continue? (Y/[N]) y
y
   -> MP NVRAM has been cleared.


MP is now being reset...




-> Command successful.

[hp26ilo] CM:hpiLO->
------------
------------

Reset the server after the ILO has been cleared

hp25> reset server 1

WARNING: Resetting the server trips its E-Fuse. This causes all power to be momentarily removed from the server. This command should only be used when physical access to 

the server is unavailable, and the server must be removed and
reinserted.

Any disk operations on direct attached storage devices will be affected. I/O
will be interrupted on any direct attached I/O devices.

Entering anything other than 'YES' will result in the command not executing.

Do you want to continue ? YES

Successfully reset the E-Fuse for device bay 1.


hp25>

=================

Log back into the ILO using OA manager and configure the admin userid.  Then Log info the ILO an power on the blade.

[hp26ilo] CM:hpiLO-> ps


PS
For System Processor Status see the SS command.

For Power Supply and Fan status see the BladeSystem Onboard Administrator.

System Power state      : Off
System Power usage      : 29 Watts used in Auxiliary Mode
System Power allocation : 0 Watts allocated for Main Power
Ambient temperature     : 15 C
Temperature status      : Normal


[hp26ilo] CM:hpiLO-> pc
PC
Current System Power State: Off


Power Control Menu:
     C   - Power Cycle
     ON  - Power On
     OFF - Power Off
     G   - Graceful Shutdown

Enter menu item or [Q] to Quit: on
on

System will be powered on.
   Confirm? (Y/[N]): y
y
   System power on in progress.

   -> System has been powered on.

[hp26ilo] CM:hpiLO-> ma

===============
Blade boots up.


Local hard disks did not appear in the boot menus.  I ended up booting from the still attached USB DVD drive and re-installling HPUX.  The installer did see the two local 

disks.   This will usually insert the boot path into the UEFI menus. 

After the HPUX was installed, I logged in.

# ioscan -funC lan
Class     I  H/W Path       Driver S/W State   H/W Type     Description
========================================================================
lan       0  0/0/0/3/0/0/0  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       1  0/0/0/3/0/0/1  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       2  0/0/0/3/0/0/2  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       3  0/0/0/3/0/0/3  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       4  0/0/0/3/0/0/4  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       5  0/0/0/3/0/0/5  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       6  0/0/0/3/0/0/6  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       7  0/0/0/3/0/0/7  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       8  0/0/0/4/0/0/0  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       9  0/0/0/4/0/0/1  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      10  0/0/0/4/0/0/2  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      11  0/0/0/4/0/0/3  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      12  0/0/0/4/0/0/4  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      13  0/0/0/4/0/0/5  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      14  0/0/0/4/0/0/6  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      15  0/0/0/4/0/0/7  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      16  0/0/0/5/0/0/0/2/0/0/0  iether   CLAIMED     INTERFACE    HP 447881-001  PCIe 4-port 1000Mb/s Mezzanine Adapter
lan      17  0/0/0/5/0/0/0/2/0/0/1  iether   CLAIMED     INTERFACE    HP 447881-001  PCIe 4-port 1000Mb/s Mezzanine Adapter
lan      18  0/0/0/5/0/0/0/4/0/0/0  iether   CLAIMED     INTERFACE    HP 447881-001  PCIe 4-port 1000Mb/s Mezzanine Adapter
lan      19  0/0/0/5/0/0/0/4/0/0/1  iether   CLAIMED     INTERFACE    HP 447881-001  PCIe 4-port 1000Mb/s Mezzanine Adapter
# lanscan
Hardware Station        Crd Hdw   Net-Interface  NM  MAC       HP-DLPI DLPI
Path     Address        In# State NamePPA        ID  Type      Support Mjr#
0/0/0/3/0/0/0 0x643150008850 0   UP    lan0 snap0     1   ETHER     Yes     119
0/0/0/3/0/0/1 0x643150008854 1   UP    lan1 snap1     2   ETHER     Yes     119
0/0/0/3/0/0/2 0x643150008851 2   UP    lan2 snap2     3   ETHER     Yes     119
0/0/0/3/0/0/3 0x643150008855 3   UP    lan3 snap3     4   ETHER     Yes     119
0/0/0/3/0/0/4 0x643150008852 4   UP    lan4 snap4     5   ETHER     Yes     119
0/0/0/3/0/0/5 0x643150008856 5   UP    lan5 snap5     6   ETHER     Yes     119
0/0/0/3/0/0/6 0x643150008853 6   UP    lan6 snap6     7   ETHER     Yes     119
0/0/0/3/0/0/7 0x643150008857 7   UP    lan7 snap7     8   ETHER     Yes     119
0/0/0/4/0/0/0 0x643150008858 8   UP    lan8 snap8     9   ETHER     Yes     119
0/0/0/4/0/0/1 0x64315000885C 9   UP    lan9 snap9     10  ETHER     Yes     119
0/0/0/4/0/0/2 0x643150008859 10  UP    lan10 snap10   11  ETHER     Yes     119
0/0/0/4/0/0/3 0x64315000885D 11  UP    lan11 snap11   12  ETHER     Yes     119
0/0/0/4/0/0/4 0x64315000885A 12  UP    lan12 snap12   13  ETHER     Yes     119
0/0/0/4/0/0/5 0x64315000885E 13  UP    lan13 snap13   14  ETHER     Yes     119
0/0/0/4/0/0/6 0x64315000885B 14  UP    lan14 snap14   15  ETHER     Yes     119
0/0/0/4/0/0/7 0x64315000885F 15  UP    lan15 snap15   16  ETHER     Yes     119
0/0/0/5/0/0/0/2/0/0/0 0x0025B3B44464 16  UP    lan16 snap16   17  ETHER     Yes     119
0/0/0/5/0/0/0/2/0/0/1 0x0025B3B44465 17  UP    lan17 snap17   18  ETHER     Yes     119
0/0/0/5/0/0/0/4/0/0/0 0x0025B3B44466 18  UP    lan18 snap18   19  ETHER     Yes     119
0/0/0/5/0/0/0/4/0/0/1 0x0025B3B44467 19  UP    lan19 snap19   20  ETHER     Yes     119
LinkAgg0 0x000000000000 900 DOWN  lan900 snap900 22  ETHER     Yes     119
LinkAgg1 0x000000000000 901 DOWN  lan901 snap901 23  ETHER     Yes     119
LinkAgg2 0x000000000000 902 DOWN  lan902 snap902 24  ETHER     Yes     119
LinkAgg3 0x000000000000 903 DOWN  lan903 snap903 25  ETHER     Yes     119
LinkAgg4 0x000000000000 904 DOWN  lan904 snap904 26  ETHER     Yes     119
# netstat -in
Name      Mtu  Network         Address         Ipkts              Ierrs Opkts              Oerrs Coll
lan1      1500 192.168.4.0     192.168.4.26    0                  0     0                  0     0
lan0      1500 192.168.5.0     192.168.5.26    0                  0     0                  0     0
lo0      32808 127.0.0.0       127.0.0.1       2848               0     2848               0     0
lan16     1500 192.168.16.0    192.168.16.183  69                 0     37                 0     0
#

Names

hp25 = c3000 
hp26 = BL860c i2 in Bay1
hp27 = BL860c i2 in Bay2
hp28 = BL860c i1 in Bay3

[DonOBrien]

Hi Torsten,

I just wanted to follow up and give closure to this thread. 

You were correct in that the VC mode was the culprit.  My attempts to clear the mode were not successful.  But I ended up opening a HPE ticket and they go in and pretty much did much of the same commands.   They did dig a bit deeper and the system did always show that VC mode was already disabled.  It took them several attempts, but eventually the system came back up with only 4 10Gig NICs.    Once that was done, the NICs pinged just fine. 

Thank you again,

Don