Can't get 10Gig networking working with BL860c i2 blade in a c3000 chassis

[DonOBrien]

I have recently purchased two BL860c i2 blades and a BL860c i1 blade in a c3000 chassis.   These are Itanium and are running HPUX 11i v3.   The two BL860c i2 blades have an embeded 10Gig network card that shows 16 NICs.   The c3000 has a network passthru module that provides 4 ports to each blade.   I have a 1gig network module in Mezz2 and that goes to a 2nd passthru module. 

I am setting this system up for the first time and I'm running through the hardware makeing sure that everything works.   The SAN works, the 1gig network works.  But when I configure the 10Gig networking, It does not work.

The first 10Gig port is on lan0.  It shows UP when I plug it into a switch, and down when I remove the connection. So I have confirmed that I am in the right port.  When I have blade1 and blade2 10Gig connections plugged into a flat Nexus switch, they can't ping each other.   I then plugged both of them into a Cisco 6509 just incase they didn't like the nexus switch. It still didn't work.   I have two Solaris system with 10Gig NICs that I used as a control group.  I plugged them into the same switches.   They can ping each other, but not the BL860c i2 systems. 

I then took a small fc cable and connected the two BL860c i2 servers to each other without a switch between them.   This worked, they could ping each other.    I then connected a Solaris server directly to one of the BL860c i2 servers.  No ping.

So I have these two BL860c servers that can only ping when they are conencted directly to each other, but fail when connected to any switch or directly to other servers.    I am hoping that there is some 10G setting that I do not know about that I need to use to get this working.   I just used defaults and smh to configure the ports.

# ioscan -funC lan
Class     I  H/W Path       Driver S/W State   H/W Type     Description
========================================================================
lan       0  0/0/0/3/0/0/0  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       1  0/0/0/3/0/0/1  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       2  0/0/0/3/0/0/2  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       3  0/0/0/3/0/0/3  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       4  0/0/0/3/0/0/4  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       5  0/0/0/3/0/0/5  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       6  0/0/0/3/0/0/6  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       7  0/0/0/3/0/0/7  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       8  0/0/0/4/0/0/0  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan       9  0/0/0/4/0/0/1  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      10  0/0/0/4/0/0/2  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      11  0/0/0/4/0/0/3  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      12  0/0/0/4/0/0/4  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      13  0/0/0/4/0/0/5  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      14  0/0/0/4/0/0/6  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      15  0/0/0/4/0/0/7  iexgbe   CLAIMED     INTERFACE    HP  PCIe 2-p 10GbE Built-in  FLEX-10
lan      16  0/0/0/5/0/0/0/2/0/0/0  iether   CLAIMED     INTERFACE    HP 447881-001  PCIe 4-port 1000Mb/s Mezzanine Adapter
lan      17  0/0/0/5/0/0/0/2/0/0/1  iether   CLAIMED     INTERFACE    HP 447881-001  PCIe 4-port 1000Mb/s Mezzanine Adapter
lan      18  0/0/0/5/0/0/0/4/0/0/0  iether   CLAIMED     INTERFACE    HP 447881-001  PCIe 4-port 1000Mb/s Mezzanine Adapter
lan      19  0/0/0/5/0/0/0/4/0/0/1  iether   CLAIMED     INTERFACE    HP 447881-001  PCIe 4-port 1000Mb/s Mezzanine Adapter
# lanscan
Hardware Station        Crd Hdw   Net-Interface  NM  MAC       HP-DLPI DLPI
Path     Address        In# State NamePPA        ID  Type      Support Mjr#
0/0/0/3/0/0/0 0x643150008850 0   UP    lan0 snap0     1   ETHER     Yes     119
0/0/0/3/0/0/1 0x643150008854 1   UP    lan1 snap1     2   ETHER     Yes     119
0/0/0/3/0/0/2 0x643150008851 2   UP    lan2 snap2     3   ETHER     Yes     119
0/0/0/3/0/0/3 0x643150008855 3   UP    lan3 snap3     4   ETHER     Yes     119
0/0/0/3/0/0/4 0x643150008852 4   UP    lan4 snap4     5   ETHER     Yes     119
0/0/0/3/0/0/5 0x643150008856 5   UP    lan5 snap5     6   ETHER     Yes     119
0/0/0/3/0/0/6 0x643150008853 6   UP    lan6 snap6     7   ETHER     Yes     119
0/0/0/3/0/0/7 0x643150008857 7   UP    lan7 snap7     8   ETHER     Yes     119
0/0/0/4/0/0/0 0x643150008858 8   UP    lan8 snap8     9   ETHER     Yes     119
0/0/0/4/0/0/1 0x64315000885C 9   UP    lan9 snap9     10  ETHER     Yes     119
0/0/0/4/0/0/2 0x643150008859 10  UP    lan10 snap10   11  ETHER     Yes     119
0/0/0/4/0/0/3 0x64315000885D 11  UP    lan11 snap11   12  ETHER     Yes     119
0/0/0/4/0/0/4 0x64315000885A 12  UP    lan12 snap12   13  ETHER     Yes     119
0/0/0/4/0/0/5 0x64315000885E 13  UP    lan13 snap13   14  ETHER     Yes     119
0/0/0/4/0/0/6 0x64315000885B 14  UP    lan14 snap14   15  ETHER     Yes     119
0/0/0/4/0/0/7 0x64315000885F 15  UP    lan15 snap15   16  ETHER     Yes     119
0/0/0/5/0/0/0/2/0/0/0 0x0025B3B44464 16  UP    lan16 snap16   17  ETHER     Yes     119
0/0/0/5/0/0/0/2/0/0/1 0x0025B3B44465 17  UP    lan17 snap17   18  ETHER     Yes     119
0/0/0/5/0/0/0/4/0/0/0 0x0025B3B44466 18  UP    lan18 snap18   19  ETHER     Yes     119
0/0/0/5/0/0/0/4/0/0/1 0x0025B3B44467 19  UP    lan19 snap19   20  ETHER     Yes     119
LinkAgg0 0x000000000000 900 DOWN  lan900 snap900 22  ETHER     Yes     119
LinkAgg1 0x000000000000 901 DOWN  lan901 snap901 23  ETHER     Yes     119
LinkAgg2 0x000000000000 902 DOWN  lan902 snap902 24  ETHER     Yes     119
LinkAgg3 0x000000000000 903 DOWN  lan903 snap903 25  ETHER     Yes     119
LinkAgg4 0x000000000000 904 DOWN  lan904 snap904 26  ETHER     Yes     119
# smh


                              SMH-> Networking and Communications-> Network Interfaces Configuration-> Network Interface Cards
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Interface Name    Subsystem    Hardware Path            Interface State    Interface Type    IPv4 Address      IPv6 Address
-------------------------------------------------------------------------------------------------------------------------------------------------------------
lan0              iexgbe       0/0/0/3/0/0/0            up                 10GBASE-KR        192.168.5.26      Not Configured
lan1              iexgbe       0/0/0/3/0/0/1            up                 10GBASE-KR        192.168.4.26      Not Configured
lan2              iexgbe       0/0/0/3/0/0/2            down               10GBASE-KR        Not Configured    Not Configured
lan3              iexgbe       0/0/0/3/0/0/3            down               10GBASE-KR        Not Configured    Not Configured
lan4              iexgbe       0/0/0/3/0/0/4            down               10GBASE-KR        Not Configured    Not Configured
lan5              iexgbe       0/0/0/3/0/0/5            up                 10GBASE-KR        Not Configured    Not Configured
lan6              iexgbe       0/0/0/3/0/0/6            down               10GBASE-KR        Not Configured    Not Configured
lan7              iexgbe       0/0/0/3/0/0/7            down               10GBASE-KR        Not Configured    Not Configured
lan8              iexgbe       0/0/0/4/0/0/0            up                 10GBASE-KR        Not Configured    Not Configured
lan9              iexgbe       0/0/0/4/0/0/1            up                 10GBASE-KR        Not Configured    Not Configured
lan10             iexgbe       0/0/0/4/0/0/2            down               10GBASE-KR        Not Configured    Not Configured
lan11             iexgbe       0/0/0/4/0/0/3            down               10GBASE-KR        Not Configured    Not Configured
lan12             iexgbe       0/0/0/4/0/0/4            up                 10GBASE-KR        Not Configured    Not Configured
lan13             iexgbe       0/0/0/4/0/0/5            down               10GBASE-KR        Not Configured    Not Configured
lan14             iexgbe       0/0/0/4/0/0/6            down               10GBASE-KR        Not Configured    Not Configured
lan15             iexgbe       0/0/0/4/0/0/7            down               10GBASE-KR        Not Configured    Not Configured
lan16             iether       0/0/0/5/0/0/0/2/0/0/0    up                 1000Mb/s          192.168.16.180    Not Configured
lan17             iether       0/0/0/5/0/0/0/2/0/0/1    down               1000Mb/s          Not Configured    Not Configured
lan18             iether       0/0/0/5/0/0/0/4/0/0/0    down               1000Mb/s          Not Configured    Not Configured
lan19             iether       0/0/0/5/0/0/0/4/0/0/1    down               1000Mb/s          Not Configured    Not Configured
-------------------------------------------------------------------------------------------------------------------------------------------------------------

[DonOBrien]

I had the network guys check the 6509 Cisco switch and they do not see MAC addresses for my HP systems, but they do for the Solaris systems.   So that tells me I have a layer two issue.   Hmmm.. 

[Steven Schweda]

> [...] 192.168.5.26 [...]
> [...] 192.168.4.26 [...]

> [...] 192.168.16.180 [...]

   What's the subnet mask for these interfaces?  Around here (/24),
each would be on its own subnet.

[Torsten.]

Maybe the blade was left in virtual connect mode, but you don't have VC.

Worth to try a "clear vc mode", see

https://community.hpe.com/t5/BladeSystem-Virtual-Connect/Reset-Virtual-Connect-to-factory-defaults/td-p/5296553#.XbFKseSP5PY

[DonOBrien]

192.168.5.x/24

192.168.4.x/24

192.168.16.x/22

The first two are just "test" networks that I'm using to verify 10Gig NIC functionality.  The 192.168.16.x/22 network is a lab network on the 1gig interface.   That NIC is working just fine.   Just the 10gigs that are giving me issues.

[DonOBrien]

Thank you for the reply. I went into the OA and I found that the VC mode is set to disabled.    So I am guessing that this isn't it.

[Torsten.]

A  single blade in non-VC mode looks like this (without mezzanine):

# ioscan -Clan
H/W Path Class Description
=================================================================
0/0/0/3/0/0/0 lan HP PCIe 2-p 10GbE Built-in
0/0/0/3/0/0/1 lan HP PCIe 2-p 10GbE Built-in
0/0/0/4/0/0/0 lan HP PCIe 2-p 10GbE Built-in
0/0/0/4/0/0/1 lan HP PCIe 2-p 10GbE Built-in

So I think the VC mode is still active for this blade.

I have seen the same issue before, now trying to remember how we solved this ...

[Torsten.]

Try this.

Do a ssh session to the OA, then connect to the MP/ILO of the blade (connect server <bay#>)

switch to CM menu.

Do

CM:hpiLO-> CILON -iloKeepLan

go back to OA.

the do

OA> reset server <bay number>

boot up and check "ioscan -fnClan" again.

[DonOBrien]

I reset the ILO memory.   I hate to say it, but no change.

[hp26ilo] CM:hpiLO-> CILON -iloKeepLan


CILON -iloKeepLan


WARNING: This is not an HP supported command and it is intended for internal
use only.  Execution of this command is not recommended and can lead to
unpredictable results.

The following command options are available:
   "CILON -ilo"         - Clears MP NVRAM and reboots MP.
   "CILON -iloKeepLan"  - Clears MP NVRAM, except for MP LAN settings,
                          and reboots MP.
   "CILON -sfw"         - Clears SFW NVRAM and reboots MP.
   "CILON -all default" - Clears both MP and SFW NVRAM and reboots MP.


This command will clear MP NVRAM and reboot MP.
All MP parameters will be reset to their default values.

MP LAN Settings will not be cleared.



Are you REALLY sure you want to continue? (Y/[N]) y
y
   -> MP NVRAM has been cleared.


MP is now being reset...




-> Command successful.

[hp26ilo] CM:hpiLO->


Checked the c3000

hp25> show server list

Bay iLO Name                      iLO IP Address  Status   Power   UID Partner
--- ----------------------------- --------------- -------- ------- --- -------
  1 hp26ilo                       192.168.16.177  OK       Off     Off
  2 hp27ilo                       192.168.16.178  OK       On      Off
  3 hp28ilo                       192.168.16.179  OK       On      Off
  4 [Absent]
  5 [Subsumed]
  6 [Subsumed]
  7 [Subsumed]
  8 [Absent]
Totals: 3 server blades installed, 2 powered on.

hp25> reset server 1

WARNING: Resetting the server trips its E-Fuse. This causes all power to be momentarily removed from the server. This command should only be used when physical access to the server is unavailable, and the server must be removed and
reinserted.

Any disk operations on direct attached storage devices will be affected. I/O
will be interrupted on any direct attached I/O devices.

Entering anything other than 'YES' will result in the command not executing.

Do you want to continue ? yes

Successfully reset the E-Fuse for device bay 1.


hp25> show server list

Bay iLO Name                      iLO IP Address  Status   Power   UID Partner
--- ----------------------------- --------------- -------- ------- --- -------
  1 [Unknown]                     N/A             OK       Unknown  ?
  2 hp27ilo                       192.168.16.178  OK       On      Off
  3 hp28ilo                       192.168.16.179  OK       On      Off
  4 [Absent]
  5 [Absent]
  6 [Subsumed]
  7 [Subsumed]
  8 [Absent]
Totals: 3 server blades installed, 2 powered on.

hp25>

<Eventually the server came back up>

hp25> show server list

Bay iLO Name                      iLO IP Address  Status   Power   UID Partner
--- ----------------------------- --------------- -------- ------- --- -------
  1 hp26ilo                       192.168.16.177  OK       Off     Off
  2 hp27ilo                       192.168.16.178  OK       On      Off
  3 hp28ilo                       192.168.16.179  OK       On      Off
  4 [Absent]
  5 [Subsumed]
  6 [Subsumed]
  7 [Subsumed]
  8 [Absent]
Totals: 3 server blades installed, 2 powered on.

hp25>

  It did wipe out my ILO settings.  I had to rebuild the userid and when I booted, it didn't see my local hard disks.  I had to go through the efi menu to boot.   This system was installed without the passthrough modules installed.  I'm going to reinstall the OS and see if that helps.  

[PSPrakash]

Hello 

Since there is no virtual connect in the ensclosure, please ensure there is no setting/vLANs configured in the upstream switches,. Also when you mentioned no ping., how switch saw these Server NICs , Does the switch saw the MAC of these NICs and Speed patches with the NIC(Auto0Negotiation ON)  ?  By default for IEXGBE the Speed is set to Auto_Negotiation ON (which is a read-only attribute) .