SPP-2023.03.00.00 PXE issue

[RandyWe]

When attempting to boot form PXE server, the initrd.img does not see any network adapters on our DL380 Gen10.

It drops to a # prompt.  Running ifconfig -a show no adapters, only the loopback

Version 2022--09 works fine.     As a test , I ran the with the vmlinuz and initrd from the /system directory.   As expected, it also failed to a # prompt.  However, ifconfig -a shows the adapters and the connected adapter has the expected IP as issued from DHCP.    

We are a integrator for HPE equipment and use the PXE enviroment ot update firmware prior to shipping to our  customers.  

Please advise, and I am willing to test as required.

[RandyWe]

FYI.. I found a hack to make this work.

I used the initrd.img from 2022_09 and replace the init , initrd.defaul and initrd.fuctions from the 202303 initrd.img.

I added the modules from /lib/modules/5.14.21-150400.22.1-default from 2023 initrd.img to the 2022 initrd.img

Then I recomppress the initrd and save as initrd.fix   This is working now with the above hacks.

See this article How to unpack/uncompress and repack/re-compress an initial ramdisk (initrd/initramfs) boot image file on RHEL 5,6 ? - Red Hat Customer Portal for maniuplation of the initrd.img file.

Root cause seem to be that in the new 2023 initrd.img, the libz.so.1 does not get the libzstd.so.1 extracted

[ManBha]

Hello,

Thank you for notifying this to us.

We will share the feedback with the relevant team.

Thanks.

[stephane_lapie]

@RandyWe Confirming the same experience about libzstd being missing.

Running the SPP with these kernel options "d3bug console=ttyS0,115200" allows access to a shell via VSP, and one can confirm that lsmod/modprobe (aka kmod) can't find libzstd.so.1.

Because no modules are loaded, it can't even load the squashfs or initialize the network cards, causing the failure we see.

After repacking the initrd to add in a libzstd.so.1 with the appropriate glibc signature (such as one from RedHat 8.x), the SPP works from PXE.

[RandyWe]

Stephane,

Could you help a hacker out.  What is the process for "With the appriopiate Signature" that you described?

Thank you

[stephane_lapie]

You basically need a file that will dynamically link to compatible glibc versions.

Check the version from libz.so.1 in the initrd :

$ strings libz.so.1 | grep GLIBC
GLIBC_2.14
GLIBC_2.4
GLIBC_2.2.5
GLIBC_2.3.4

You therefore must pick a libzstd.so.1 version that has matching signatures, and you can find one such file in RedHat 8.X (not 9.X, which requires a more recent version) :

$ strings libzstd.so.1 | grep GLIBC
GLIBC_2.14
GLIBC_2.3.4
GLIBC_2.4
GLIBC_2.2.5
GA+GLIBCXX_ASSERTIONS
GA+GLIBCXX_ASSERTIONS

[RandyWe]

Thank you!

Randy Weisman
Logistics Engineer
Arrow Electronics | arrow.com

[Lenny2]

## add the modules from /lib/modules/5.14.21-150500.53.2-default from 2023_09 initrd.img to the 2022_09 initrd.img
works also for the 2023_09 G10 SPP :)

extract initrd.img files from 2023_09 and 2022_09 (# xz -dc < xxx.img | cpio -idmv)

add the modules to the extracted 2022_09 initrd.img folder

repack the 2022_09.img from the 2022_09 directory (# find . 2>/dev/null | cpio -o -c -R root:root | xz -9 --format=lzma > ../fixed_initrd.img)

use the new fixed_initrd.img for ipxe boot

[RandyWe]

Yes., That is what I did with 202309., but don't forget to  overwrite the init , initrd.defaul and initrd.fuctions with the files from the 202309 initrd.img.  

The Bigger question, is why HPE has not fixed this issue!?

[stephane_lapie]

The latest SPP 2024.04.00.00 still fails to PXE boot on its own,
however, this time libzstd.so.1 is actually present,
it is failing because of a missing libjitterentropy.so.3!

modprobe: error while loading shared libraries: libjitterentropy.so.3: cannot open shared object file: No such file or directory

You can fix the initrd image with the exact same method, by lifting the library from SUSE 15 (which is linked against the right versions of glibc and everything).

[RandyWe]

This worked great.   My concern is why do we still have to do work-arounds for pxe functionality.   HPE should do better and test this prior to a release.

Thanks for the heads up on the SPP2024.

[nkn-khr]

Thanks to your post, I successfully applied SPP 2024.04.00 Patch Bundle 2 to the my ML30 Gen10 via HTTP Boot.

 Steps i did to fix initrd:

cd /var/tmp/spp/initrd-pxe
xz -cd /mnt/spp2024040002/pxe/spp2024040000/initrd.img | cpio -idm
mount -o loop -t squashfs /mnt/spp2024040002/system/squashfs.img /mnt/squashfs/
cp /mnt/squashfs/usr/lib64/libjitterentropy.* ./usr/lib64
find . 2>/dev/null | cpio -o -c -R root:root | xz --format=lzma > ../fixed_initrd.img

[stephane_lapie]

SPP 2024.09.00.00 seems to have fixed it for Gen11, but not for Gen10.

Also, HPE Japan gave us this advisory: support.hpe.com/hpesc/public/docDisplay?docId=a00137041en_us

"Due to security reasons, PXE boot of SPP is not supported."

This does not make sense, and the messaging is extremely confusing:
- If there is a security concern, why keep on providing the PXE files in the first place?
- There is no explanation of the error message. The PXE initrd image is broken because it is not generated/tested properly. This in itself is not a security problem.
- The handling between Gen10 and Gen11 is not consistent, since Gen11 works.