HPE STK 11.20 on DL360 Gen10 with Solarflare NICs

[GregCabaj]

Dear All, 

I came across an interesting issue when I installed HPE STK v. 11.20 (Scripting Toolkit) to automate servers deployment. 

The server hardware is HPE ProLiant DL360 Gen10 with two network cards :

SOLARFLARE FLAREON ULTRA SFN8522-PLUS NETWORK ADAPTER PCIE 3.1 X8 10 GIGABIT SFP+

After starting the HPE STK boot image from PXE  (Linux installation) the network devices are not present.

For networking we use only Solarflare cards. The internal 1Gb Ethernet controller is disabled. 

The lspci command shows the cards present, but "ip add"  or "ifconfig -a" show only localhost interface.

The HPE STK User Guide tells to use ./loadnet.sh script but the output of this command is that the interfaces were not found. The sfc module is not loaded.

Do I need to build my own kernel (vmlinuz) and new initrd and add hp tools to it, or am I missing something? 

This seem to be a drivers issue to me. Any hint would be much appreciated. 

Thanks, 
Greg

[Jimmy Vance]

---

@GregCabaj wrote:

Dear All, 

I came across an interesting issue when I installed HPE STK v. 11.20 (Scripting Toolkit) to automate servers deployment. 

The server hardware is HPE ProLiant DL360 Gen10 with two network cards :

SOLARFLARE FLAREON ULTRA SFN8522-PLUS NETWORK ADAPTER PCIE 3.1 X8 10 GIGABIT SFP+

After starting the HPE STK boot image from PXE  (Linux installation) the network devices are not present.

For networking we use only Solarflare cards. The internal 1Gb Ethernet controller is disabled. 

The lspci command shows the cards present, but "ip add"  or "ifconfig -a" show only localhost interface.

The HPE STK User Guide tells to use ./loadnet.sh script but the output of this command is that the interfaces were not found. The sfc module is not loaded.

Do I need to build my own kernel (vmlinuz) and new initrd and add hp tools to it, or am I missing something? 

This seem to be a drivers issue to me. Any hint would be much appreciated. 

Thanks, 
Greg

---

With trying to keep the toolkit kernel somewhat compact, it only contains drivers specific to the HPE server line and HPE options along with a very limited amount of generic drivers.   You can either build your own custom kernel, or try to open the initrd image and poke in the drivers you need for your specific cards.  If you try to add the driver,  v11.20 of the STK is based on SLES12 SP3.  Poking in an additional drivers isn't an easy or fun task.   Might be a lot easier to install using the embedded NIC, once initial configuration and OS install is complete,  use conrep to disable the embedded NIC and run some post install or firstboot script to setup your Solarflare cards.

[Jimmy Vance]

Greg, what driver is used by the Solarflare cards, 'sfc' ?   If it will help with your deployments I can ask that it be added into a future release of the toolkit

[GregCabaj · Accepted solution]

Hi Jim , All, 

Many thanks for the both comments. Unfortunately we have no infrastructure and resources to support onboard NICs. 
You are correct - this was the driver issue and it is resolved now.
I was trying to rebuild the kernel (vmlinuz) and build the driver in, but that was not successful, so I took another approach and unpacked the initrd.img and added the driver to kernel modules. It was a little painful, but hopefully below steps will help others. 

The driver required was OpenOnload driver, which includes sfc NIC drivers. It can be downloaded from : 
https://support.solarflare.com (you will need SF account to download tar.gz file)
filename: SF-109585-LS-30_OpenOnload_Release_Package.zip

For all information this is the solution, which worked for me:

1. Verify the OS version, which HPE STK is running on: 

- launch HPE STK (in this case v. 11.20) from PXE as per documentation
- when STK is up run : cat /etc/*release 
- also run: uname -r  and note down the kernel version
- you will see that it runs on SLES12 SP2 with 4.4.21-69-default kernel

2. Download (needs registration) and install SLES12 SP2 on a PC or server (I used vmware for this)
URL: https://www.suse.com/products/server/download/

3. When the test SLES12 SP2 OS is installed ( in this case let's call it 'testsles12' ),  I copied (winscp) openonload archive to it and followed installation instructions. 

4. On my windows PC I have created HPE_STK_SFC_DRIVERS folder and copied there from testsles12 below content: 
/lib/modules/4.4.21-69-default/extra      <- This directory contains openonload drivers. Copy entire dir. 
/etc/modprobe.d/onload.conf
/boot/vmlinuz-4.4.21-69-default

5. The HPE STK file hpe-scripting-toolkit-linux-11.20-6.tar.gz contains initrd.img file, which should be installed along with vmlinuz on the server with PXE configuration (pxeserver). I have copied the initrd.img file to my homedir in pxeserver and unpacked it:
mkdir /tmp/image
cd /tmp/image
xz -dc /home/greg/initrd.img | cpio -i --make-directories

6. From my windows PC folder 'HPE_STK_SFC_DRIVERS' I copied (winscp) the content to pxeserver:/tmp/HPE_STK_SFC_DRIVERS 

7. Ran below commands on pxeserver. Remember you are in local /tmp/image. Do not use leading slashes in destination. 
cd /tmp/image
mkdir lib/modules/4.4.21-69-default/kernel/drivers/net/onload
mkdir lib/modules/4.4.21-69-default/kernel/drivers/net/ethernet/sfc
cp /tmp/HPE_STK_SFC_DRIVERS/extra/* lib/modules/4.4.21-69-default/kernel/drivers/net/onload/
cp /tmp/HPE_STK_SFC_DRIVERS/extra/sfc.ko lib/modules/4.4.21-69-default/kernel/drivers/net/ethernet/sfc/
cp /tmp/HPE_STK_SFC_DRIVERS/onload.conf etc/modprobe.d/

8. When the drivers and modprobe configuration were in place I packed it back to the new initrd_1.img file. On pxeserver I ran: 
cd /tmp/image
find . 2>/dev/null | cpio --quiet -c -o | xz -9 --format=lzma > ../initrd_1.img

9 . Copied the /tmp/initrd_1.img and vmlinuz_1 on pxeserver:
cp /tmp/HPE_STK_SFC_DRIVERS/vmlinuz-4.4.21-69-default /tftpboot/hpe_stk/vmlinuz_1
cp /tmp/initrd_1.img /tftpboot/hpe_stk/initrd_1.img

10. On the NFS server, in my case pxeserver, in NFS share location with the content from hpe-scripting-toolkit-linux-11.20-6.tar.gz I ran: 
./mkiso.sh
NOTE: This needs to be done every time you modify the initial script (ie. scripts/shell.sh)

11. On pxeserver added HPE Toolkit menu to pxelinux.cfg/default:

LABEL TOOLKIT
MENU LABEL HPE Toolkit
KERNEL hpe_stk/vmlinuz_1
APPEND initrd=hpe_stk/initrd_1.img root=/dev/ram0 rw ramdisk_size=601588 ide=nodma ide=noraid pnpbios=off numa=off network=1 media=net iso1=nfs://10.100.4.200/HPE_STK/hpe-scripting-toolkit-linux-11.20.iso sstk_conf=toolkit.conf sstk_script=/shell.sh hostname=stk- keeppxe

Yes, there are quite few steps, but this should help someone who already spent some time, trying to get this to work. 

Regards,
Greg