BladeSystem - General
1833446 Members
3077 Online
110052 Solutions
New Discussion

Re: Can't get 10Gig networking working with BL860c i2 blade in a c3000 chassis

 
SOLVED
Go to solution
DonOBrien
Advisor

Can't get 10Gig networking working with BL860c i2 blade in a c3000 chassis

I have recently purchased two BL860c i2 blades and a BL860c i1 blade in a c3000 chassis.   These are Itanium and are running HPUX 11i v3.   The two BL860c i2 blades have an embeded 10Gig network card that shows 16 NICs.   The c3000 has a network passthru module that provides 4 ports to each blade.   I have a 1gig network module in Mezz2 and that goes to a 2nd passthru module. 

I am setting this system up for the first time and I'm running through the hardware makeing sure that everything works.   The SAN works, the 1gig network works.  But when I configure the 10Gig networking, It does not work.

The first 10Gig port is on lan0.  It shows UP when I plug it into a switch, and down when I remove the connection. So I have confirmed that I am in the right port.  When I have blade1 and blade2 10Gig connections plugged into a flat Nexus switch, they can't ping each other.   I then plugged both of them into a Cisco 6509 just incase they didn't like the nexus switch. It still didn't work.   I have two Solaris system with 10Gig NICs that I used as a control group.  I plugged them into the same switches.   They can ping each other, but not the BL860c i2 systems. 

I then took a small fc cable and connected the two BL860c i2 servers to each other without a switch between them.   This worked, they could ping each other.    I then connected a Solaris server directly to one of the BL860c i2 servers.  No ping.

So I have these two BL860c servers that can only ping when they are conencted directly to each other, but fail when connected to any switch or directly to other servers.    I am hoping that there is some 10G setting that I do not know about that I need to use to get this working.   I just used defaults and smh to configure the ports.

# ioscan -funC lan
Class     I  H/W Path       Driver S/W State   H/W Type     Description
========================================================================
lan       0  0/0/0/3/0/0/0  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       1  0/0/0/3/0/0/1  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       2  0/0/0/3/0/0/2  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       3  0/0/0/3/0/0/3  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       4  0/0/0/3/0/0/4  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       5  0/0/0/3/0/0/5  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       6  0/0/0/3/0/0/6  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       7  0/0/0/3/0/0/7  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       8  0/0/0/4/0/0/0  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       9  0/0/0/4/0/0/1  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      10  0/0/0/4/0/0/2  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      11  0/0/0/4/0/0/3  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      12  0/0/0/4/0/0/4  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      13  0/0/0/4/0/0/5  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      14  0/0/0/4/0/0/6  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      15  0/0/0/4/0/0/7  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      16  0/0/0/5/0/0/0/2/0/0/0  iether   CLAIMED     INTERFACE    HP 447881-001  PCIe 4-port 1000Mb/s Mezzanine Adapter
lan      17  0/0/0/5/0/0/0/2/0/0/1  iether   CLAIMED     INTERFACE    HP 447881-001  PCIe 4-port 1000Mb/s Mezzanine Adapter
lan      18  0/0/0/5/0/0/0/4/0/0/0  iether   CLAIMED     INTERFACE    HP 447881-001  PCIe 4-port 1000Mb/s Mezzanine Adapter
lan      19  0/0/0/5/0/0/0/4/0/0/1  iether   CLAIMED     INTERFACE    HP 447881-001  PCIe 4-port 1000Mb/s Mezzanine Adapter
# lanscan
Hardware Station        Crd Hdw   Net-Interface  NM  MAC       HP-DLPI DLPI
Path     Address        In# State NamePPA        ID  Type      Support Mjr#
0/0/0/3/0/0/0 0x643150008850 0   UP    lan0 snap0     1   ETHER     Yes     119
0/0/0/3/0/0/1 0x643150008854 1   UP    lan1 snap1     2   ETHER     Yes     119
0/0/0/3/0/0/2 0x643150008851 2   UP    lan2 snap2     3   ETHER     Yes     119
0/0/0/3/0/0/3 0x643150008855 3   UP    lan3 snap3     4   ETHER     Yes     119
0/0/0/3/0/0/4 0x643150008852 4   UP    lan4 snap4     5   ETHER     Yes     119
0/0/0/3/0/0/5 0x643150008856 5   UP    lan5 snap5     6   ETHER     Yes     119
0/0/0/3/0/0/6 0x643150008853 6   UP    lan6 snap6     7   ETHER     Yes     119
0/0/0/3/0/0/7 0x643150008857 7   UP    lan7 snap7     8   ETHER     Yes     119
0/0/0/4/0/0/0 0x643150008858 8   UP    lan8 snap8     9   ETHER     Yes     119
0/0/0/4/0/0/1 0x64315000885C 9   UP    lan9 snap9     10  ETHER     Yes     119
0/0/0/4/0/0/2 0x643150008859 10  UP    lan10 snap10   11  ETHER     Yes     119
0/0/0/4/0/0/3 0x64315000885D 11  UP    lan11 snap11   12  ETHER     Yes     119
0/0/0/4/0/0/4 0x64315000885A 12  UP    lan12 snap12   13  ETHER     Yes     119
0/0/0/4/0/0/5 0x64315000885E 13  UP    lan13 snap13   14  ETHER     Yes     119
0/0/0/4/0/0/6 0x64315000885B 14  UP    lan14 snap14   15  ETHER     Yes     119
0/0/0/4/0/0/7 0x64315000885F 15  UP    lan15 snap15   16  ETHER     Yes     119
0/0/0/5/0/0/0/2/0/0/0 0x0025B3B44464 16  UP    lan16 snap16   17  ETHER     Yes     119
0/0/0/5/0/0/0/2/0/0/1 0x0025B3B44465 17  UP    lan17 snap17   18  ETHER     Yes     119
0/0/0/5/0/0/0/4/0/0/0 0x0025B3B44466 18  UP    lan18 snap18   19  ETHER     Yes     119
0/0/0/5/0/0/0/4/0/0/1 0x0025B3B44467 19  UP    lan19 snap19   20  ETHER     Yes     119
LinkAgg0 0x000000000000 900 DOWN  lan900 snap900 22  ETHER     Yes     119
LinkAgg1 0x000000000000 901 DOWN  lan901 snap901 23  ETHER     Yes     119
LinkAgg2 0x000000000000 902 DOWN  lan902 snap902 24  ETHER     Yes     119
LinkAgg3 0x000000000000 903 DOWN  lan903 snap903 25  ETHER     Yes     119
LinkAgg4 0x000000000000 904 DOWN  lan904 snap904 26  ETHER     Yes     119
# smh


                              SMH-> Networking and Communications-> Network Interfaces Configuration-> Network Interface Cards
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Interface Name    Subsystem    Hardware Path            Interface State    Interface Type    IPv4 Address      IPv6 Address
-------------------------------------------------------------------------------------------------------------------------------------------------------------
lan0              iexgbe       0/0/0/3/0/0/0            up                 10GBASE-KR        192.168.5.26      Not Configured
lan1              iexgbe       0/0/0/3/0/0/1            up                 10GBASE-KR        192.168.4.26      Not Configured
lan2              iexgbe       0/0/0/3/0/0/2            down               10GBASE-KR        Not Configured    Not Configured
lan3              iexgbe       0/0/0/3/0/0/3            down               10GBASE-KR        Not Configured    Not Configured
lan4              iexgbe       0/0/0/3/0/0/4            down               10GBASE-KR        Not Configured    Not Configured
lan5              iexgbe       0/0/0/3/0/0/5            up                 10GBASE-KR        Not Configured    Not Configured
lan6              iexgbe       0/0/0/3/0/0/6            down               10GBASE-KR        Not Configured    Not Configured
lan7              iexgbe       0/0/0/3/0/0/7            down               10GBASE-KR        Not Configured    Not Configured
lan8              iexgbe       0/0/0/4/0/0/0            up                 10GBASE-KR        Not Configured    Not Configured
lan9              iexgbe       0/0/0/4/0/0/1            up                 10GBASE-KR        Not Configured    Not Configured
lan10             iexgbe       0/0/0/4/0/0/2            down               10GBASE-KR        Not Configured    Not Configured
lan11             iexgbe       0/0/0/4/0/0/3            down               10GBASE-KR        Not Configured    Not Configured
lan12             iexgbe       0/0/0/4/0/0/4            up                 10GBASE-KR        Not Configured    Not Configured
lan13             iexgbe       0/0/0/4/0/0/5            down               10GBASE-KR        Not Configured    Not Configured
lan14             iexgbe       0/0/0/4/0/0/6            down               10GBASE-KR        Not Configured    Not Configured
lan15             iexgbe       0/0/0/4/0/0/7            down               10GBASE-KR        Not Configured    Not Configured
lan16             iether       0/0/0/5/0/0/0/2/0/0/0    up                 1000Mb/s          192.168.16.180    Not Configured
lan17             iether       0/0/0/5/0/0/0/2/0/0/1    down               1000Mb/s          Not Configured    Not Configured
lan18             iether       0/0/0/5/0/0/0/4/0/0/0    down               1000Mb/s          Not Configured    Not Configured
lan19             iether       0/0/0/5/0/0/0/4/0/0/1    down               1000Mb/s          Not Configured    Not Configured
-------------------------------------------------------------------------------------------------------------------------------------------------------------

 

 

19 REPLIES 19
DonOBrien
Advisor

Re: Can't get 10Gig networking working with BL860c i2 blade in a c3000 chassis

I had the network guys check the 6509 Cisco switch and they do not see MAC addresses for my HP systems, but they do for the Solaris systems.   So that tells me I have a layer two issue.   Hmmm.. 

Steven Schweda
Honored Contributor

Re: Can't get 10Gig networking working with BL860c i2 blade in a c3000 chassis

> [...] 192.168.5.26 [...]
> [...] 192.168.4.26 [...]

> [...] 192.168.16.180 [...]

   What's the subnet mask for these interfaces?  Around here (/24),
each would be on its own subnet.

Torsten.
Acclaimed Contributor

Re: Can't get 10Gig networking working with BL860c i2 blade in a c3000 chassis

Maybe the blade was left in virtual connect mode, but you don't have VC.

Worth to try a "clear vc mode", see

https://community.hpe.com/t5/BladeSystem-Virtual-Connect/Reset-Virtual-Connect-to-factory-defaults/td-p/5296553#.XbFKseSP5PY


Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
DonOBrien
Advisor

Re: Can't get 10Gig networking working with BL860c i2 blade in a c3000 chassis

192.168.5.x/24

192.168.4.x/24

192.168.16.x/22

The first two are just "test" networks that I'm using to verify 10Gig NIC functionality.  The 192.168.16.x/22 network is a lab network on the 1gig interface.   That NIC is working just fine.   Just the 10gigs that are giving me issues.

DonOBrien
Advisor

Re: Can't get 10Gig networking working with BL860c i2 blade in a c3000 chassis

Thank you for the reply. I went into the OA and I found that the VC mode is set to disabled.    So I am guessing that this isn't it.

Torsten.
Acclaimed Contributor

Re: Can't get 10Gig networking working with BL860c i2 blade in a c3000 chassis

A  single blade in non-VC mode looks like this (without mezzanine):

# ioscan -Clan
H/W Path Class Description
=================================================================
0/0/0/3/0/0/0 lan HP PCIe 2-p 10GbE Built-in
0/0/0/3/0/0/1 lan HP PCIe 2-p 10GbE Built-in
0/0/0/4/0/0/0 lan HP PCIe 2-p 10GbE Built-in
0/0/0/4/0/0/1 lan HP PCIe 2-p 10GbE Built-in

 

So I think the VC mode is still active for this blade.

 

I have seen the same issue before, now trying to remember how we solved this ...


Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Torsten.
Acclaimed Contributor

Re: Can't get 10Gig networking working with BL860c i2 blade in a c3000 chassis

Try this.

Do a ssh session to the OA, then connect to the MP/ILO of the blade (connect server <bay#>)

switch to CM menu.

Do

CM:hpiLO-> CILON -iloKeepLan

go back to OA.

the do

OA> reset server <bay number>

boot up and check "ioscan -fnClan" again.

 


Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
DonOBrien
Advisor

Re: Can't get 10Gig networking working with BL860c i2 blade in a c3000 chassis

I reset the ILO memory.   I hate to say it, but no change.


[hp26ilo] CM:hpiLO-> CILON -iloKeepLan


CILON -iloKeepLan


WARNING: This is not an HP supported command and it is intended for internal
use only.  Execution of this command is not recommended and can lead to
unpredictable results.

The following command options are available:
   "CILON -ilo"         - Clears MP NVRAM and reboots MP.
   "CILON -iloKeepLan"  - Clears MP NVRAM, except for MP LAN settings,
                          and reboots MP.
   "CILON -sfw"         - Clears SFW NVRAM and reboots MP.
   "CILON -all default" - Clears both MP and SFW NVRAM and reboots MP.


This command will clear MP NVRAM and reboot MP.
All MP parameters will be reset to their default values.

MP LAN Settings will not be cleared.



Are you REALLY sure you want to continue? (Y/[N]) y
y
   -> MP NVRAM has been cleared.


MP is now being reset...




-> Command successful.

[hp26ilo] CM:hpiLO->


Checked the c3000

hp25> show server list

Bay iLO Name                      iLO IP Address  Status   Power   UID Partner
--- ----------------------------- --------------- -------- ------- --- -------
  1 hp26ilo                       192.168.16.177  OK       Off     Off
  2 hp27ilo                       192.168.16.178  OK       On      Off
  3 hp28ilo                       192.168.16.179  OK       On      Off
  4 [Absent]
  5 [Subsumed]
  6 [Subsumed]
  7 [Subsumed]
  8 [Absent]
Totals: 3 server blades installed, 2 powered on.

hp25> reset server 1

WARNING: Resetting the server trips its E-Fuse. This causes all power to be momentarily removed from the server. This command should only be used when physical access to the server is unavailable, and the server must be removed and
reinserted.

Any disk operations on direct attached storage devices will be affected. I/O
will be interrupted on any direct attached I/O devices.

Entering anything other than 'YES' will result in the command not executing.

Do you want to continue ? yes

Successfully reset the E-Fuse for device bay 1.


hp25> show server list

Bay iLO Name                      iLO IP Address  Status   Power   UID Partner
--- ----------------------------- --------------- -------- ------- --- -------
  1 [Unknown]                     N/A             OK       Unknown  ?
  2 hp27ilo                       192.168.16.178  OK       On      Off
  3 hp28ilo                       192.168.16.179  OK       On      Off
  4 [Absent]
  5 [Absent]
  6 [Subsumed]
  7 [Subsumed]
  8 [Absent]
Totals: 3 server blades installed, 2 powered on.

hp25>

<Eventually the server came back up>

hp25> show server list

Bay iLO Name                      iLO IP Address  Status   Power   UID Partner
--- ----------------------------- --------------- -------- ------- --- -------
  1 hp26ilo                       192.168.16.177  OK       Off     Off
  2 hp27ilo                       192.168.16.178  OK       On      Off
  3 hp28ilo                       192.168.16.179  OK       On      Off
  4 [Absent]
  5 [Subsumed]
  6 [Subsumed]
  7 [Subsumed]
  8 [Absent]
Totals: 3 server blades installed, 2 powered on.

hp25>




  It did wipe out my ILO settings.  I had to rebuild the userid and when I booted, it didn't see my local hard disks.  I had to go through the efi menu to boot.   This system was installed without the passthrough modules installed.  I'm going to reinstall the OS and see if that helps.  

PSPrakash
HPE Pro

Re: Can't get 10Gig networking working with BL860c i2 blade in a c3000 chassis

Hello 

Since there is no virtual connect in the ensclosure, please ensure there is no setting/vLANs configured in the upstream switches,. Also when you mentioned no ping., how switch saw these Server NICs , Does the switch saw the MAC of these NICs and Speed patches with the NIC(Auto0Negotiation ON)  ?  By default for IEXGBE the Speed is set to Auto_Negotiation ON (which is a read-only attribute) .  



I work at HPE
HPE Support Center offers support for your HPE services and products when and how you need it. Get started with HPE Support Center today.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
Accept or Kudo
Torsten.
Acclaimed Contributor

Re: Can't get 10Gig networking working with BL860c i2 blade in a c3000 chassis

You have 4 NICs on such blade built-in.
In VC mode each interface is splitted into 4 sub-NIC, hence 16 NICS.
If you don't have VC modules, there is no counterpart for the NICs, hence your problem.

How about the ioscan now?

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   

Re: Can't get 10Gig networking working with BL860c i2 blade in a c3000 chassis

Are you sure you have a 10GbE pass-thru module (538113-B21) and not another 1GbE pass-thru module? (406740-B21) - if you had the 1GbE module it wouldn't surprise me if you had to hard set both ports (NIC and switchj) to 1Gb to get them to work.


I am an HPE Employee
Accept or Kudo
DonOBrien
Advisor

Re: Can't get 10Gig networking working with BL860c i2 blade in a c3000 chassis

I'm pretty sure I have the 10Gig Module.  I did connect to each module thru the c3000.   Here are my stats on the modules.   hp25 is the name of my c3000. 

Module 1 is used for the Mezz1 ports of my blades.  blade1 and blade2 are i2 and have 10gig nics,  blade3 is a i1 and has 1G nics.   (port 3,7,11,15 CU fp).

hp25> show interconnect list

Bay Interconnect Type     Manufacturer      Power   Health   UID  Management IP
--- ----------------- -------------------- ------- --------- --- ---------------
  1 Ethernet          HP                   On      OK        Off 0.0.0.0
  2 Ethernet          HP                   On      OK        Off 0.0.0.0
  3 Fibre Channel     BROCADE              On      OK        Off 192.168.17.31
  4 Fibre Channel     BROCADE              On      OK        Off 192.168.17.33
Totals: 4 interconnect modules installed, 4 powered on.

Module 1

System# info

System:                  HP 10Gbe Pass-Thru
Version:                 1.0.7.0
Board:                   HP 10GbE Pass-Thru Module
Serial:                  MY31375CUH

S
System# /port

[Port Menu]
    auto     -  Port autonegotiation configuration
    info     -  Port information

Port# info

         Uplink                            Downlink
Port Ena Link Speed 1G-Neg  Type  Approved Link Speed Neg
  1  ena down  --    on    SR SFP+   yes   down  --   on
  2  ena down  --    on    SR SFP+   yes   down  --   on
  3  ena down  --    on    CU SFP    yes   down  --   on
  4  dis down  --    on      N/A     no    down  --   on
  5  ena  up  10Gb   on    SR SFP+   yes    up  10Gb  on
  6  ena  up  10Gb   on    SR SFP+   yes    up  10Gb  on
  7  ena  up   1Gb   on    CU SFP    yes    up   1Gb  on
  8  dis down  --    on    SR SFP+   yes   down  --   on
  9  ena down  --    on    SR SFP+   yes   down  --   on
 10  ena down  --    on    SR SFP+   yes   down  --   on
 11  ena down  --    on    CU SFP    yes   down  --   on
 12  dis down  --    on      N/A     no    down  --   on
 13  ena  up  10Gb   on    SR SFP+   yes    up  10Gb  on
 14  ena  up  10Gb   on    SR SFP+   yes    up  10Gb  on
 15  ena down  --    on    CU SFP    yes   down  --   on
 16  dis down  --    on    SR SFP+   yes   down  --   on

Port#

Module 2

System# info

System:                  HP 10Gbe Pass-Thru
Version:                 1.0.7.0
Board:                   HP 10GbE Pass-Thru Module
Serial:                  MY31355BRV

System# /port

[Port Menu]
    auto     -  Port autonegotiation configuration
    info     -  Port information

Port# info

         Uplink                            Downlink
Port Ena Link Speed 1G-Neg  Type  Approved Link Speed Neg
  1  ena  up   1Gb   on    CU SFP    yes    up   1Gb  on
  2  ena  up   1Gb   on    CU SFP    yes    up   1Gb  on
  3  dis down  --    on      N/A     no    down  --   on
  4  dis down  --    on    CU SFP    yes   down  --   on
  5  ena down  --    on    CU SFP    yes   down  --   on
  6  ena down  --    on    CU SFP    yes   down  --   on
  7  dis down  --    on      N/A     no    down  --   on
  8  dis down  --    on    CU SFP    yes   down  --   on
  9  ena down  --    on    CU SFP    yes   down  --   on
 10  ena down  --    on    CU SFP    yes   down  --   on
 11  dis down  --    on      N/A     no    down  --   on
 12  dis down  --    on    CU SFP    yes   down  --   on
 13  ena down  --    on    CU SFP    yes   down  --   on
 14  ena down  --    on    CU SFP    yes   down  --   on
 15  dis down  --    on      N/A     no    down  --   on
 16  dis down  --    on    CU SFP    yes   down  --   on

Port#

 

DonOBrien
Advisor

Re: Can't get 10Gig networking working with BL860c i2 blade in a c3000 chassis

Hi Torsten,

Thank you again for all your help on this.   I tried to clear any VC mode setting, but either I did it incorrectly, or that isn't the issue.   I see what you mean about having 16 nics in the OS, but only four ports available on the passthru. 

After I ran the ILO clear commands (posted earlier) and reset of the server, and a reinstall of HPUX, the 16 nics persist.   I am guessing that the i2 blades may still be in VC mode, while the c3000 is not.    Is there a command that I can run to query the blade/ilo to confirm this?  

 

Here is my sytsem's current output for the network info.

# ioscan -funC lan
Class     I  H/W Path       Driver S/W State   H/W Type     Description
========================================================================
lan       0  0/0/0/3/0/0/0  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       1  0/0/0/3/0/0/1  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       2  0/0/0/3/0/0/2  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       3  0/0/0/3/0/0/3  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       4  0/0/0/3/0/0/4  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       5  0/0/0/3/0/0/5  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       6  0/0/0/3/0/0/6  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       7  0/0/0/3/0/0/7  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       8  0/0/0/4/0/0/0  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       9  0/0/0/4/0/0/1  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      10  0/0/0/4/0/0/2  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      11  0/0/0/4/0/0/3  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      12  0/0/0/4/0/0/4  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      13  0/0/0/4/0/0/5  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      14  0/0/0/4/0/0/6  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      15  0/0/0/4/0/0/7  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      16  0/0/0/5/0/0/0/2/0/0/0  iether   CLAIMED     INTERFACE    HP 447881-001  PCIe 4-port 1000Mb/s Mezzanine Adapter
lan      17  0/0/0/5/0/0/0/2/0/0/1  iether   CLAIMED     INTERFACE    HP 447881-001  PCIe 4-port 1000Mb/s Mezzanine Adapter
lan      18  0/0/0/5/0/0/0/4/0/0/0  iether   CLAIMED     INTERFACE    HP 447881-001  PCIe 4-port 1000Mb/s Mezzanine Adapter
lan      19  0/0/0/5/0/0/0/4/0/0/1  iether   CLAIMED     INTERFACE    HP 447881-001  PCIe 4-port 1000Mb/s Mezzanine Adapter
# lanscan
Hardware Station        Crd Hdw   Net-Interface  NM  MAC       HP-DLPI DLPI
Path     Address        In# State NamePPA        ID  Type      Support Mjr#
0/0/0/3/0/0/0 0x643150008850 0   UP    lan0 snap0     1   ETHER     Yes     119
0/0/0/3/0/0/1 0x643150008854 1   UP    lan1 snap1     2   ETHER     Yes     119
0/0/0/3/0/0/2 0x643150008851 2   UP    lan2 snap2     3   ETHER     Yes     119
0/0/0/3/0/0/3 0x643150008855 3   UP    lan3 snap3     4   ETHER     Yes     119
0/0/0/3/0/0/4 0x643150008852 4   UP    lan4 snap4     5   ETHER     Yes     119
0/0/0/3/0/0/5 0x643150008856 5   UP    lan5 snap5     6   ETHER     Yes     119
0/0/0/3/0/0/6 0x643150008853 6   UP    lan6 snap6     7   ETHER     Yes     119
0/0/0/3/0/0/7 0x643150008857 7   UP    lan7 snap7     8   ETHER     Yes     119
0/0/0/4/0/0/0 0x643150008858 8   UP    lan8 snap8     9   ETHER     Yes     119
0/0/0/4/0/0/1 0x64315000885C 9   UP    lan9 snap9     10  ETHER     Yes     119
0/0/0/4/0/0/2 0x643150008859 10  UP    lan10 snap10   11  ETHER     Yes     119
0/0/0/4/0/0/3 0x64315000885D 11  UP    lan11 snap11   12  ETHER     Yes     119
0/0/0/4/0/0/4 0x64315000885A 12  UP    lan12 snap12   13  ETHER     Yes     119
0/0/0/4/0/0/5 0x64315000885E 13  UP    lan13 snap13   14  ETHER     Yes     119
0/0/0/4/0/0/6 0x64315000885B 14  UP    lan14 snap14   15  ETHER     Yes     119
0/0/0/4/0/0/7 0x64315000885F 15  UP    lan15 snap15   16  ETHER     Yes     119
0/0/0/5/0/0/0/2/0/0/0 0x0025B3B44464 16  UP    lan16 snap16   17  ETHER     Yes     119
0/0/0/5/0/0/0/2/0/0/1 0x0025B3B44465 17  UP    lan17 snap17   18  ETHER     Yes     119
0/0/0/5/0/0/0/4/0/0/0 0x0025B3B44466 18  UP    lan18 snap18   19  ETHER     Yes     119
0/0/0/5/0/0/0/4/0/0/1 0x0025B3B44467 19  UP    lan19 snap19   20  ETHER     Yes     119
LinkAgg0 0x000000000000 900 DOWN  lan900 snap900 22  ETHER     Yes     119
LinkAgg1 0x000000000000 901 DOWN  lan901 snap901 23  ETHER     Yes     119
LinkAgg2 0x000000000000 902 DOWN  lan902 snap902 24  ETHER     Yes     119
LinkAgg3 0x000000000000 903 DOWN  lan903 snap903 25  ETHER     Yes     119
LinkAgg4 0x000000000000 904 DOWN  lan904 snap904 26  ETHER     Yes     119
# netstat -in
Name      Mtu  Network         Address         Ipkts              Ierrs Opkts              Oerrs Coll
lan1      1500 192.168.4.0     192.168.4.26    6                  0     6                  0     0
lan0      1500 192.168.5.0     192.168.5.26    0                  0     0                  0     0
lo0      32808 127.0.0.0       127.0.0.1       50770              0     50770              0     0
lan16     1500 192.168.16.0    192.168.16.184  574708             0     366149             0     0
#

 

 

vc_mode_disabled.JPG

Torsten.
Acclaimed Contributor

Re: Can't get 10Gig networking working with BL860c i2 blade in a c3000 chassis

Did you run the ILO reset command while the blade was powered off?
If not, please try.

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Torsten.
Acclaimed Contributor
Solution

Re: Can't get 10Gig networking working with BL860c i2 blade in a c3000 chassis

For clear VC mode command all blades must be powered off. Please try.

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
DonOBrien
Advisor

Re: Can't get 10Gig networking working with BL860c i2 blade in a c3000 chassis

Ok.  Just to be clear. 

1. I should power off all three blades in my c3000. 
2. Run the "CILON -iloKeepLan" command to clear the nvram, run on each blade. that is showing this problem (the i1 blade dosen't have 10G and its ports are working normally).   Run this command from the CM menu in the ILO.
3. Reset each of the three servers using the "reset server x" command in the OA.

When the servers come up and I run a "ioscan -funC lan" I should only see four 10G NICs and not 16.  

Does it sound like I have this correct? 

Thank you again Torsten. 

Torsten.
Acclaimed Contributor

Re: Can't get 10Gig networking working with BL860c i2 blade in a c3000 chassis

First power off the i2 blade only and do step 2 again.

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
DonOBrien
Advisor

Re: Can't get 10Gig networking working with BL860c i2 blade in a c3000 chassis

Hi Torsten,  Thank you again for all your help on this.   I have tried to reset the blade, but again, the behavior goes unchanged.   It looks like the blade is still behaving as if it is in VC mode.

I shutdown the blade, cleared the config on the ILO, and then reset the blade.  I powered it up, The UEFI menus did not have the local hard disks available to boot from.   So I booted from the DVD and just did a fresh install (hoped that would also allow for a fresh look at the NICs).    Once the HPUX was installed, I logged and I'm still seeing the 16 NICs for the on-board network adapters.

 

1. Shutdown the i2 blade.

[hp26ilo] CM:hpiLO-> ps


PS
For System Processor Status see the SS command.

For Power Supply and Fan status see the BladeSystem Onboard Administrator.
System Power state      : Off
System Power usage      : 34 Watts used in Auxiliary Mode
System Power allocation : 0 Watts allocated for Main Power
Ambient temperature     : 19 C
Temperature status      : Normal

[hp26ilo] CM:hpiLO->


Confirm that the i2 Blade in bay 1 off.

hp25> show server list

Bay iLO Name                      iLO IP Address  Status   Power   UID Partner
--- ----------------------------- --------------- -------- ------- --- -------
  1 hp26ilo                       192.168.16.177  OK       Off     Off
  2 hp27ilo                       192.168.16.178  OK       On      Off
  3 hp28ilo                       192.168.16.179  OK       On      Off
  4 [Absent]
  5 [Subsumed]
  6 [Subsumed]
  7 [Subsumed]
  8 [Absent]
Totals: 3 server blades installed, 2 powered on.

hp25>

------------
------------

Reset the ILO

[hp26ilo] CM:hpiLO-> CILON -iloKeepLan


CILON -iloKeepLan


WARNING: This is not an HP supported command and it is intended for internal
use only.  Execution of this command is not recommended and can lead to
unpredictable results.

The following command options are available:
   "CILON -ilo"         - Clears MP NVRAM and reboots MP.
   "CILON -iloKeepLan"  - Clears MP NVRAM, except for MP LAN settings,
                          and reboots MP.
   "CILON -sfw"         - Clears SFW NVRAM and reboots MP.
   "CILON -all default" - Clears both MP and SFW NVRAM and reboots MP.


This command will clear MP NVRAM and reboot MP.
All MP parameters will be reset to their default values.

MP LAN Settings will not be cleared.



Are you REALLY sure you want to continue? (Y/[N]) y
y
   -> MP NVRAM has been cleared.


MP is now being reset...




-> Command successful.

[hp26ilo] CM:hpiLO->
------------
------------

Reset the server after the ILO has been cleared

hp25> reset server 1

WARNING: Resetting the server trips its E-Fuse. This causes all power to be momentarily removed from the server. This command should only be used when physical access to 

the server is unavailable, and the server must be removed and
reinserted.

Any disk operations on direct attached storage devices will be affected. I/O
will be interrupted on any direct attached I/O devices.

Entering anything other than 'YES' will result in the command not executing.

Do you want to continue ? YES

Successfully reset the E-Fuse for device bay 1.


hp25>

=================

Log back into the ILO using OA manager and configure the admin userid.  Then Log info the ILO an power on the blade.

[hp26ilo] CM:hpiLO-> ps


PS
For System Processor Status see the SS command.

For Power Supply and Fan status see the BladeSystem Onboard Administrator.

System Power state      : Off
System Power usage      : 29 Watts used in Auxiliary Mode
System Power allocation : 0 Watts allocated for Main Power
Ambient temperature     : 15 C
Temperature status      : Normal


[hp26ilo] CM:hpiLO-> pc
PC
Current System Power State: Off


Power Control Menu:
     C   - Power Cycle
     ON  - Power On
     OFF - Power Off
     G   - Graceful Shutdown

Enter menu item or [Q] to Quit: on
on

System will be powered on.
   Confirm? (Y/[N]): y
y
   System power on in progress.

   -> System has been powered on.

[hp26ilo] CM:hpiLO-> ma

===============
Blade boots up.


Local hard disks did not appear in the boot menus.  I ended up booting from the still attached USB DVD drive and re-installling HPUX.  The installer did see the two local 

disks.   This will usually insert the boot path into the UEFI menus. 

After the HPUX was installed, I logged in.

# ioscan -funC lan
Class     I  H/W Path       Driver S/W State   H/W Type     Description
========================================================================
lan       0  0/0/0/3/0/0/0  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       1  0/0/0/3/0/0/1  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       2  0/0/0/3/0/0/2  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       3  0/0/0/3/0/0/3  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       4  0/0/0/3/0/0/4  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       5  0/0/0/3/0/0/5  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       6  0/0/0/3/0/0/6  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       7  0/0/0/3/0/0/7  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       8  0/0/0/4/0/0/0  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       9  0/0/0/4/0/0/1  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      10  0/0/0/4/0/0/2  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      11  0/0/0/4/0/0/3  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      12  0/0/0/4/0/0/4  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      13  0/0/0/4/0/0/5  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      14  0/0/0/4/0/0/6  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      15  0/0/0/4/0/0/7  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      16  0/0/0/5/0/0/0/2/0/0/0  iether   CLAIMED     INTERFACE    HP 447881-001  PCIe 4-port 1000Mb/s Mezzanine Adapter
lan      17  0/0/0/5/0/0/0/2/0/0/1  iether   CLAIMED     INTERFACE    HP 447881-001  PCIe 4-port 1000Mb/s Mezzanine Adapter
lan      18  0/0/0/5/0/0/0/4/0/0/0  iether   CLAIMED     INTERFACE    HP 447881-001  PCIe 4-port 1000Mb/s Mezzanine Adapter
lan      19  0/0/0/5/0/0/0/4/0/0/1  iether   CLAIMED     INTERFACE    HP 447881-001  PCIe 4-port 1000Mb/s Mezzanine Adapter
# lanscan
Hardware Station        Crd Hdw   Net-Interface  NM  MAC       HP-DLPI DLPI
Path     Address        In# State NamePPA        ID  Type      Support Mjr#
0/0/0/3/0/0/0 0x643150008850 0   UP    lan0 snap0     1   ETHER     Yes     119
0/0/0/3/0/0/1 0x643150008854 1   UP    lan1 snap1     2   ETHER     Yes     119
0/0/0/3/0/0/2 0x643150008851 2   UP    lan2 snap2     3   ETHER     Yes     119
0/0/0/3/0/0/3 0x643150008855 3   UP    lan3 snap3     4   ETHER     Yes     119
0/0/0/3/0/0/4 0x643150008852 4   UP    lan4 snap4     5   ETHER     Yes     119
0/0/0/3/0/0/5 0x643150008856 5   UP    lan5 snap5     6   ETHER     Yes     119
0/0/0/3/0/0/6 0x643150008853 6   UP    lan6 snap6     7   ETHER     Yes     119
0/0/0/3/0/0/7 0x643150008857 7   UP    lan7 snap7     8   ETHER     Yes     119
0/0/0/4/0/0/0 0x643150008858 8   UP    lan8 snap8     9   ETHER     Yes     119
0/0/0/4/0/0/1 0x64315000885C 9   UP    lan9 snap9     10  ETHER     Yes     119
0/0/0/4/0/0/2 0x643150008859 10  UP    lan10 snap10   11  ETHER     Yes     119
0/0/0/4/0/0/3 0x64315000885D 11  UP    lan11 snap11   12  ETHER     Yes     119
0/0/0/4/0/0/4 0x64315000885A 12  UP    lan12 snap12   13  ETHER     Yes     119
0/0/0/4/0/0/5 0x64315000885E 13  UP    lan13 snap13   14  ETHER     Yes     119
0/0/0/4/0/0/6 0x64315000885B 14  UP    lan14 snap14   15  ETHER     Yes     119
0/0/0/4/0/0/7 0x64315000885F 15  UP    lan15 snap15   16  ETHER     Yes     119
0/0/0/5/0/0/0/2/0/0/0 0x0025B3B44464 16  UP    lan16 snap16   17  ETHER     Yes     119
0/0/0/5/0/0/0/2/0/0/1 0x0025B3B44465 17  UP    lan17 snap17   18  ETHER     Yes     119
0/0/0/5/0/0/0/4/0/0/0 0x0025B3B44466 18  UP    lan18 snap18   19  ETHER     Yes     119
0/0/0/5/0/0/0/4/0/0/1 0x0025B3B44467 19  UP    lan19 snap19   20  ETHER     Yes     119
LinkAgg0 0x000000000000 900 DOWN  lan900 snap900 22  ETHER     Yes     119
LinkAgg1 0x000000000000 901 DOWN  lan901 snap901 23  ETHER     Yes     119
LinkAgg2 0x000000000000 902 DOWN  lan902 snap902 24  ETHER     Yes     119
LinkAgg3 0x000000000000 903 DOWN  lan903 snap903 25  ETHER     Yes     119
LinkAgg4 0x000000000000 904 DOWN  lan904 snap904 26  ETHER     Yes     119
# netstat -in
Name      Mtu  Network         Address         Ipkts              Ierrs Opkts              Oerrs Coll
lan1      1500 192.168.4.0     192.168.4.26    0                  0     0                  0     0
lan0      1500 192.168.5.0     192.168.5.26    0                  0     0                  0     0
lo0      32808 127.0.0.0       127.0.0.1       2848               0     2848               0     0
lan16     1500 192.168.16.0    192.168.16.183  69                 0     37                 0     0
#

Names

hp25 = c3000 
hp26 = BL860c i2 in Bay1
hp27 = BL860c i2 in Bay2
hp28 = BL860c i1 in Bay3

DonOBrien
Advisor

Re: Can't get 10Gig networking working with BL860c i2 blade in a c3000 chassis

Hi Torsten,


I just wanted to follow up and give closure to this thread. 

 

You were correct in that the VC mode was the culprit.  My attempts to clear the mode were not successful.  But I ended up opening a HPE ticket and they go in and pretty much did much of the same commands.   They did dig a bit deeper and the system did always show that VC mode was already disabled.  It took them several attempts, but eventually the system came back up with only 4 10Gig NICs.    Once that was done, the NICs pinged just fine. 

 

Thank you again,

Don