ProLiant Servers (ML,DL,SL)
1752754 Members
4672 Online
108789 Solutions
New Discussion

Re: DL380 G10 / RHEL Device Order

 
SOLVED
Go to solution
MikeJeezy
Advisor

DL380 G10 / RHEL Device Order

Hi, we just purchased a few DL380 G10 SFF 24 servers.  It has the following controllers:

  • E208i-a SR on the system board (RAID 1 for OS in the rear cage)
  • E208i-p SR - Slot 1 driving front cage 1 (mixed mode JBOD)
  • E208i-p SR - Slot 3 driving front cage 2 (mixed mode JBOD)
  • E208i-p SR - Slot 4 driving front cage 3 (mixed mode JBOD)

All disks except the RAID 1 will be JBOD hence the mixed mode.  The issue is; when we boot the system to RHEL (or to install RHEL), the devices (/dev/sdX) are all out of order.  The rear cage OS drives show up as /dev/sdr instead of /dev/sda.  Is there a way to for the BIOS to present these in some sort of logical order?

Initially I disabled all controllers except for the system board controller, so RHEL would install on the correct device, but when I enabled them again, it reverted back.  This never seemed to be an issue with the G9's.  Any suggestions or advice?

Thank you kindly.

15 REPLIES 15
Jimmy Vance
HPE Pro

Re: DL380 G10 / RHEL Device Order


@MikeJeezy wrote:

Hi, we just purchased a few DL380 G10 SFF 24 servers.  It has the following controllers:

  • E208i-a SR on the system board (RAID 1 for OS in the rear cage)
  • E208i-p SR - Slot 1 driving front cage 1 (mixed mode JBOD)
  • E208i-p SR - Slot 3 driving front cage 2 (mixed mode JBOD)
  • E208i-p SR - Slot 4 driving front cage 3 (mixed mode JBOD)

All disks except the RAID 1 will be JBOD hence the mixed mode.  The issue is; when we boot the system to RHEL (or to install RHEL), the devices (/dev/sdX) are all out of order.  The rear cage OS drives show up as /dev/sdr instead of /dev/sda.  Is there a way to for the BIOS to present these in some sort of logical order?

Initially I disabled all controllers except for the system board controller, so RHEL would install on the correct device, but when I enabled them again, it reverted back.  This never seemed to be an issue with the G9's.  Any suggestions or advice?

Thank you kindly.


What you are seeing is due to the way Linux kernel does PCI bus enumeration. This is one reason RHEL and most other distributions use /dev/disk/by-id (usually UUID) instead of  /dev/sdX device labels to identify the drives.  The device label name of the drives shouldn't matter as /etc/fstab uses UUID.  If you really must have the rear cage appear as 'sda' you need to connect that cage to whichever controller is being enumerated as the first controller during boot.  Another option is to create custom udev rules assigning UUID to specific device labels (not easy to do on multiple sysems).  If the main concern is being able to use /dev/sda for partitioning in a kickstart file you can use a %pre script to figure out which drive you want to be the boot drive, set that as a variable that the kickstart file uses for partitioning.  Or if all your systems are cabled the same way substitute X for what letter is actually showing up. The latter may not always be the same even if cabled the same. If a device is slow to identify for any reason during startup the enumeration can change  

No support by private messages. Please ask the forum! 
MikeJeezy
Advisor

Re: DL380 G10 / RHEL Device Order

Thank you for your response.  Correct, I agree we could use the UUID's but that is easier to do after the OS is installed.  If we can’t use the SDA lun as default to install RHEL, we will have to boot each server into maintenance mode to get UUID and then modify kickstart config file for each server to begin OS installation (we have many servers).  The embedded controller should be enumerated first, then controller slot 1, 2, etc.  It doesn't make any sense to break that order.  This was not an issue with the G9 servers.  Any other bug reports that you are aware?  Or manual BIOS configurations perhaps? Thanks

MikeJeezy
Advisor

Re: DL380 G10 / RHEL Device Order

The attached PDF is showing how the Legacy Boot Order is identifying the controllers.  It is identifying the embedded controller (PCI Slot:0) as Controller #02 instead of Controller #00, which in this case would make it the 17th disk (/dev/sdr).

Jimmy Vance
HPE Pro

Re: DL380 G10 / RHEL Device Order

So the real issue is with kickstart.  You can add a %pre script to kickstart.  I have some examples I need to dig up that will help.  Basically a pre script to figure out which drive you want to use, in your case a 220GB RAID volume, create a partitioning scheme file with that info and use an %include statement to pull that file into the kickstart file

 

No support by private messages. Please ask the forum! 
MikeJeezy
Advisor

Re: DL380 G10 / RHEL Device Order

Well, there seems to be a larger issue here.  I can disable the 3 controller cards so the System Board controller is forced to be "Controller #0" (/dev/sda), and then install RHEL successfully. 

The problem is when I re-enable the cards, the System Board controller reverts to Controller #2 instead of Controller #0. This also changes the original /dev/sda where Linux is installed... it changes to /dev/sdr.  Okay fine.   But the operating system cannot be found even though the correct logical drive is set as "Primary Boot" in the Storage Manager.  It's like the BIOS is only searching for the OS on /dev/sda.  There is no way that I can see to "force" it to boot from /dev/sdr.

Jimmy Vance
HPE Pro

Re: DL380 G10 / RHEL Device Order


@MikeJeezy wrote:

Well, there seems to be a larger issue here.  I can disable the 3 controller cards so the System Board controller is forced to be "Controller #0" (/dev/sda), and then install RHEL successfully. 

The problem is when I re-enable the cards, the System Board controller reverts to Controller #2 instead of Controller #0. This also changes the original /dev/sda where Linux is installed... it changes to /dev/sdr.  Okay fine.   But the operating system cannot be found even though the correct logical drive is set as "Primary Boot" in the Storage Manager.  It's like the BIOS is only searching for the OS on /dev/sda.  There is no way that I can see to "force" it to boot from /dev/sdr.



Any time you add or remove controllers/drives the enumeration order can and most likely will change, that's just how Linux works.   Are you using UEFI or legacy emulation mode?  

Something like what I have osted below in the kickstart file should resolve the problem.  This is just an example, modify to fit your needs, or use something like the 'list-harddrives' tool in Anaconda to create your own pre script.  UEFI is easier to work with in this situation as you don't need to specify the boot drive in RBSU.  This example is modified from an example I found on the Internet some time ago.

%pre
# pre section
#----- partitioning logic below--------------
# pick the first drive that is not removable and is over MINSIZE
DIR="/sys/block"

# minimum size of hard drive needed specified in GIGABYTES
MINSIZE=64

ROOTDRIVE=""

# /sys/block/*/size is in 512 byte chunks

for DEV in sd{a..z}; do
  if [ -d $DIR/$DEV ]; then
    REMOVABLE=`cat $DIR/$DEV/removable`
    if (( $REMOVABLE == 0 )); then
      echo $DEV
      SIZE=`cat $DIR/$DEV/size`
      GB=$(($SIZE/2**21))
      if [ $GB -gt $MINSIZE ]; then
        echo "$(($SIZE/2**21))"
        if [ $GB -lt 250 ]; then
        if [ -z $ROOTDRIVE ]; then
          ROOTDRIVE=$DEV
        fi
        fi
      fi
    fi
  fi
done

echo "ROOTDRIVE=$ROOTDRIVE"

cat << EOF > /tmp/part-include
zerombr
clearpart --all --drives=$ROOTDRIVE --initlabel
bootloader --location=mbr --driveorder=$ROOTDRIVE
part /boot --fstype="xfs" --ondisk=$ROOTDRIVE --size=500
part /boot/efi --fstype="efi" --ondisk=$ROOTDRIVE --size=200
part / --fstype="xfs" --ondisk=$ROOTDRIVE --size=51200 --grow
part swap --fstype="swap" --ondisk=$ROOTDRIVE --size=9876
EOF
%end

#comment out original partitioning scheme and use this include
%include /tmp/part-include

With Gen10 the embedded µSD controller will take a device name also. Hopefully it enumerates as the last device but that is not always the case.  If you are missing a drive letter and can't find an attached device using it, this is likely the cause.

 

No support by private messages. Please ask the forum! 
MikeJeezy
Advisor

Re: DL380 G10 / RHEL Device Order

Thank you for your reply.  The server is set to legacy boot mode.  In further troubleshooting, we basically forced Kixtart to install on /dev/sdr (which is the RAID 1 SSD pair in the rear cage attached to the embedded controller).  We also have this logical volume set as the primary boot device in the SSA utility.  

The problem is.... when the server is restarted after installation, it can't find any OS and reverts to PXE boot.  The server seems to only care about booting from /dev/sda, and when no OS is found there... it fails to boot.  

I do understand using the UUID's, but if GRUB is never found, there is no way to access /etc/fstab to use the UUID's.  What am I missing?  This is a very odd problem to have.

Jimmy Vance
HPE Pro
Solution

Re: DL380 G10 / RHEL Device Order


@MikeJeezy wrote:

Thank you for your reply.  The server is set to legacy boot mode.  In further troubleshooting, we basically forced Kixtart to install on /dev/sdr (which is the RAID 1 SSD pair in the rear cage attached to the embedded controller).  We also have this logical volume set as the primary boot device in the SSA utility.  

The problem is.... when the server is restarted after installation, it can't find any OS and reverts to PXE boot.  The server seems to only care about booting from /dev/sda, and when no OS is found there... it fails to boot.  

I do understand using the UUID's, but if GRUB is never found, there is no way to access /etc/fstab to use the UUID's.  What am I missing?  This is a very odd problem to have.


Please refer to this Customer Advisory, you need to set the "Number of OS bootable drives." to 0 on all the other controllers and be running HPE Smart Array firmware v1.34 (or later)
https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00022532en_us

Can I ask why you are using legacy emulation mode and not UEFI? 

 

No support by private messages. Please ask the forum! 
MikeJeezy
Advisor

Re: DL380 G10 / RHEL Device Order


@Jimmy Vance wrote:


Please refer to this Customer Advisory, you need to set the "Number of OS bootable drives." to 0 on all the other controllers and be running HPE Smart Array firmware v1.34 (or later)
https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00022532en_us

Can I ask why you are using legacy emulation mode and not UEFI? 


There is no technical reason why I suppose... just have not had a compelling reason to make the switch to UEFI.  We are running Smart Array firmware version 1.66. 

It appears, the customer advisory pertains to Legacy mode.   I will try UEFI.