Using ssascripting to load a RAID config results in first array not being sda

[benbaez]

Hi,

We have been using kickstart to auto configure servers since at least Gen 6 from what I have seen.  We have received Gen10 DL360 and DL380s and using our existing raid setup scripts with V3.40 of ssa tools, /dev/sda is not the first array after clearing the config through ssa gui in BIOS or using -reset flag when applying the config over an existing raid config of the same.

/mnt/source/pretools/ssascripting -i /mnt/source/pretools/raid-configs/hp-raid.ini -reset -e /tmp/raid-error.ini

If I don't use the -reset flag (such as when kicking the server again and it is already RAID configured) sda is properly mapped to first array in controller.  The DL360 Gen10 seems to not have this problem, but I have to confirm again to be sure.

hp-raid.ini

; Date captured: Wed Jun 12 23:45:43 2019

; Version: 3.30.14.0 2018-04-24

Action= Configure

Method= Custom

; __________________________ Controller Specifications SLOT 0 ________________________________

;

; Controller HPE Smart Array P816i-a SR Gen10, FirmwareVersion 1.98, License Keys Supported

; SerialNumber: info erased

; DriverName smartpqi

; DriverVersion Linux 1.1.4-115

; SSDSmartPath Supported

Controller= SLOT 0

; PowerMode= MaxPerformance

; SurvivalMode= Enabled

ReadCache= 10

WriteCache= 90

WriteCacheBypassThreshold= 1040

RebuildPriority= High

ExpandPriority= Medium

ParallelSurfaceScanCount= 1

SurfaceScanMode= Idle

SurfaceScanDelay= 3

Latency= Disable

NoBatteryWriteCache= Disabled

MNPDelay= 60

IRPEnable= Disabled

DPOEnable= Disabled

ElevatorSortEnable= Enabled

QueueDepth= Automatic

PredictiveSpareActivation= Disable

SanitizeLock= None

; PendingSanitizeLock= None

; Drive Write Cache Policies - uncomment to modify

; DriveWriteCacheUnconfigured= DEFAULT

; DriveWriteCacheConfigured= DISABLE

; Array Specifications

Array= A

; Array Drive Type is SAS

; Array Block Size is 512 bytes

; Array Free Space    0.0 GBytes

; 1I:3:2 (SAS HDD, 1.2 TB), 1I:3:1 (SAS HDD, 1.2 TB)

Drive= 1I:3:2, 1I:3:1

OnlineSpare= No

; Logical Drive Specifications

LogicalDrive= 1

RAID= 1

Size= 953837

; SizeBlocks= 1953459632

Sectors= 32

StripSize= 256

Caching= Enabled

; VolumeUniqueID= 600508B1001C8F8BD3CBB79719096978

; Array Specifications

Array= B

; Array Drive Type is SAS

; Array Block Size is 512 bytes

; Array Free Space    0.0 GBytes

; 1I:3:4 (SAS HDD, 1.2 TB), 1I:3:3 (SAS HDD, 1.2 TB)

Drive= 1I:3:4, 1I:3:3

OnlineSpare= No

; Logical Drive Specifications

LogicalDrive= 2

RAID= 1

Size= 953837

; SizeBlocks= 1953459632

Sectors= 32

StripSize= 256

Caching= Enabled

; VolumeUniqueID= 600508B1001CD670AD91AAA2CA04EE29

; Array Specifications

Array= C

; Array Drive Type is SAS

; Array Block Size is 512 bytes

; Array Free Space    0.0 GBytes

; 2I:3:5 (SAS HDD, 1.2 TB), 2I:3:6 (SAS HDD, 1.2 TB)

Drive= 2I:3:5, 2I:3:6

OnlineSpare= No

; Logical Drive Specifications

LogicalDrive= 3

RAID= 1

Size= 953837

; SizeBlocks= 1953459632

Sectors= 32

StripSize= 256

Caching= Enabled

; VolumeUniqueID= 600508B1001CF49909165BD97C34296B

; Array Specifications

Array= D

; Array Drive Type is SAS

; Array Block Size is 512 bytes

; Array Free Space    0.0 GBytes

; 2I:3:7 (SAS HDD, 1.2 TB), 2I:3:8 (SAS HDD, 1.2 TB)

Drive= 2I:3:7, 2I:3:8

OnlineSpare= No

; Logical Drive Specifications

LogicalDrive= 4

RAID= 1

Size= 953837

; SizeBlocks= 1953459632

Sectors= 32

StripSize= 256

Caching= Enabled

; VolumeUniqueID= 600508B1001CD71D7ECB3156B5715796

; Array Specifications

Array= E

; Array Drive Type is Solid State SAS

; Array Block Size is 512 bytes

; Array Free Space    0.0 GBytes

; 3I:2:1 (Solid State SAS SSD, 400 GB)

Drive= 3I:2:1

SSDSmartPath= Enable

OnlineSpare= No

; Logical Drive Specifications

LogicalDrive= 5

RAID= 0

Size= 381522

; SizeBlocks= 781357232

Sectors= 32

StripSize= 256

; VolumeUniqueID= 600508B1001CCB52B6CEFADB958C2195

; Array Specifications

Array= F

; Array Drive Type is Solid State SAS

; Array Block Size is 512 bytes

; Array Free Space    0.0 GBytes

; 3I:2:2 (Solid State SAS SSD, 400 GB)

Drive= 3I:2:2

SSDSmartPath= Enable

OnlineSpare= No

; Logical Drive Specifications

LogicalDrive= 6

RAID= 0

Size= 381522

; SizeBlocks= 781357232

Sectors= 32

StripSize= 256

; VolumeUniqueID= 600508B1001CB8EF0CDE495993CEFE5D

; Array Specifications

Array= G

; Array Drive Type is Solid State SAS

; Array Block Size is 512 bytes

; Array Free Space    0.0 GBytes

; 3I:2:3 (Solid State SAS SSD, 400 GB)

Drive= 3I:2:3

SSDSmartPath= Enable

OnlineSpare= No

; Logical Drive Specifications

LogicalDrive= 7

RAID= 0

Size= 381522

; SizeBlocks= 781357232

Sectors= 32

StripSize= 256

; VolumeUniqueID= 600508B1001C9AF77D97B968E2488924

; Array Specifications

Array= H

; Array Drive Type is Solid State SAS

; Array Block Size is 512 bytes

; Array Free Space    0.0 GBytes

; 3I:2:4 (Solid State SAS SSD, 1.3 TB)

Drive= 3I:2:4

SSDSmartPath= Enable

OnlineSpare= No

; Logical Drive Specifications

LogicalDrive= 8

RAID= 0

Size= 1526153

; SizeBlocks= 3125562032

Sectors= 32

StripSize= 256

; VolumeUniqueID= 600508B1001C863E09C43D1D275F655C

; Array Specifications

Array= I

; Array Drive Type is Solid State SAS

; Array Block Size is 512 bytes

; Array Free Space    0.0 GBytes

; 4I:2:5 (Solid State SAS SSD, 1.3 TB)

Drive= 4I:2:5

SSDSmartPath= Enable

OnlineSpare= No

; Logical Drive Specifications

LogicalDrive= 9

RAID= 0

Size= 1526153

; SizeBlocks= 3125562032

Sectors= 32

StripSize= 256

; VolumeUniqueID= 600508B1001C00D3DA5974322DC0705C

; Array Specifications

Array= J

; Array Drive Type is Solid State SAS

; Array Block Size is 512 bytes

; Array Free Space    0.0 GBytes

; 4I:2:6 (Solid State SAS SSD, 1.3 TB)

Drive= 4I:2:6

SSDSmartPath= Enable

OnlineSpare= No

; Logical Drive Specifications

LogicalDrive= 10

RAID= 0

Size= 1526153

; SizeBlocks= 3125562032

Sectors= 32

StripSize= 256

; VolumeUniqueID= 600508B1001CEB17FB1CDD7776FB92D1

; PORT Specifications

Port=1I

; PortMode= Mixed

; PORT Specifications

Port=2I

; PortMode= Mixed

; PORT Specifications

Port=3I

; PortMode= Mixed

; PORT Specifications

Port=4I

; PortMode= Mixed

[Jimmy Vance]

With Gen10, the internal SD device uses a dev label even when there isn't a card present. Because of the way Linux handles PCI enumeration the internal SD card device can sometimes enumerate before any devices on the disk controllers and grab 'sda' 

Refer to this discussion were I explain options in detail https://community.hpe.com/t5/ProLiant-Servers-ML-DL-SL/DL380-G10-RHEL-Device-Order/m-p/7040784#M166176

[benbaez]

How do you attach an image?  I have some iLO screenshots that show the issue.  I haven't had any issues with sd or usb showing up on disk enumeration.

Basically, sda is correctly pointing to 1:1:0:0 during anaconda.  Which is the first logicaldrive.  sdb-j all point to their repsective arrays.  Then anaconda runs ssascripting to load the raid config, and sda points to the logicaldrive 3 typically when the raid config is complete.

Thanks

[Jimmy Vance]

Click the camera icon when in posting mode

[benbaez]

The camera icon is grayed out, but still let's me click it and asks for http url source.  

Anywho, I got a text example.

[setup-raid] ############# /mnt/source/pretools/setup-raid.sh #####################

lrwxrwxrwx. 1 root root 0 Jun 14 00:15 /sys/block/sda/device -> ../../../1:1:0:0
lrwxrwxrwx. 1 root root 0 Jun 14 00:15 /sys/block/sdb/device -> ../../../1:1:0:1
lrwxrwxrwx. 1 root root 0 Jun 14 00:15 /sys/block/sdc/device -> ../../../1:1:0:2
lrwxrwxrwx. 1 root root 0 Jun 14 00:15 /sys/block/sdd/device -> ../../../1:1:0:3
lrwxrwxrwx. 1 root root 0 Jun 14 00:15 /sys/block/sde/device -> ../../../1:1:0:4
lrwxrwxrwx. 1 root root 0 Jun 14 00:15 /sys/block/sdf/device -> ../../../1:1:0:5
lrwxrwxrwx. 1 root root 0 Jun 14 00:15 /sys/block/sdg/device -> ../../../1:1:0:6
lrwxrwxrwx. 1 root root 0 Jun 14 00:15 /sys/block/sdh/device -> ../../../1:1:0:7
lrwxrwxrwx. 1 root root 0 Jun 14 00:15 /sys/block/sdi/device -> ../../../1:1:0:8
lrwxrwxrwx. 1 root root 0 Jun 14 00:15 /sys/block/sdj/device -> ../../../0:0:0:0
lrwxrwxrwx. 1 root root 0 Jun 14 00:15 /sys/block/sdk/device -> ../../../1:1:0:9
[setup-raid] Loading RAID configuration...
[setup-raid] RAID CONFIGURATION LOADED SUCCESSFULLY

lrwxrwxrwx. 1 root root 0 Jun 14 00:15 /sys/block/sda/device -> ../../../1:1:0:1
lrwxrwxrwx. 1 root root 0 Jun 14 00:16 /sys/block/sdb/device -> ../../../1:1:0:2
lrwxrwxrwx. 1 root root 0 Jun 14 00:15 /sys/block/sdc/device -> ../../../1:1:0:0
lrwxrwxrwx. 1 root root 0 Jun 14 00:16 /sys/block/sdd/device -> ../../../1:1:0:3
lrwxrwxrwx. 1 root root 0 Jun 14 00:16 /sys/block/sde/device -> ../../../1:1:0:4
lrwxrwxrwx. 1 root root 0 Jun 14 00:16 /sys/block/sdf/device -> ../../../1:1:0:5
lrwxrwxrwx. 1 root root 0 Jun 14 00:16 /sys/block/sdg/device -> ../../../1:1:0:6
lrwxrwxrwx. 1 root root 0 Jun 14 00:16 /sys/block/sdh/device -> ../../../1:1:0:7
lrwxrwxrwx. 1 root root 0 Jun 14 00:16 /sys/block/sdi/device -> ../../../1:1:0:8
lrwxrwxrwx. 1 root root 0 Jun 14 00:15 /sys/block/sdj/device -> ../../../0:0:0:0
lrwxrwxrwx. 1 root root 0 Jun 14 00:16 /sys/block/sdk/device -> ../../../1:1:0:9

This is the resulting drive layout after the kick:

[root@inf-105-9-syseng log]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 931.5G 0 disk
sdb 8:16 0 931.5G 0 disk
├─sdb1 8:17 0 200M 0 part /boot/efi
├─sdb2 8:18 0 256M 0 part /boot
└─sdb3 8:19 0 931G 0 part
├─mgtvg-rootvol 253:0 0 5G 0 lvm /
├─mgtvg-swapvol 253:1 0 5G 0 lvm [SWAP]
├─mgtvg-varvol 253:2 0 50G 0 lvm /var
└─mgtvg-optvol 253:3 0 5G 0 lvm /opt
sdc 8:32 0 931.5G 0 disk
└─sdc1 8:33 0 931.5G 0 part
└─cass2osvg-cass2os 253:4 0 120G 0 lvm /vm/cass2
sdd 8:48 0 931.5G 0 disk
└─sdd1 8:49 0 931.5G 0 part
└─cass3osvg-cass3os 253:7 0 120G 0 lvm /vm/cass3
sde 8:64 0 372.6G 0 disk
└─sde1 8:65 0 372.6G 0 part
sdf 8:80 0 372.6G 0 disk
└─sdf1 8:81 0 372.6G 0 part
└─cass2logvg-cass2log 253:5 0 370G 0 lvm
sdg 8:96 0 372.6G 0 disk
└─sdg1 8:97 0 372.6G 0 part
└─cass3logvg-cass3log 253:8 0 370G 0 lvm
sdh 8:112 0 1.5T 0 disk
└─sdh1 8:113 0 1.5T 0 part
sdi 8:128 0 1.5T 0 disk
└─sdi1 8:129 0 1.5T 0 part
└─cass2datavg-cass2data 253:6 0 1.4T 0 lvm
sdj 8:144 0 1.5T 0 disk
└─sdj1 8:145 0 1.5T 0 part
└─cass3datavg-cass3data 253:9 0 1.4T 0 lvm

This is the sda layout defined in partition map for kick:

-------------
bootloader --location=mbr --driveorder=sda --append="crashkernel=auto rhgb quiet"
zerombr
clearpart --all --drives=sda,sdb,sdc,sdd,sde,sdf,sdg,sdh,sdi,sdj --initlabel
ignoredisk --only-use=sda
part /boot --fstype=ext4 --size=256 --ondisk=sda
part /boot/efi --fstype=efi --size=50 --grow --maxsize=200
part pv.40 --size=102400 --grow --ondisk=sda
volgroup mgtvg --pesize=32768 pv.40
logvol swap --fstype=swap --name=swapvol --vgname=mgtvg --size=5120
logvol / --fstype=ext4 --name=rootvol --vgname=mgtvg --size=5120
logvol /opt --fstype=ext4 --name=optvol --vgname=mgtvg --size=5120
logvol /var --fstype=ext4 --name=varvol --vgname=mgtvg --size=51200
#NUM_RAID_VOLUMES=10
-------------

[benbaez]

A bit more info to show mapping of sda, this is after the kick.

[root@inf-105-9-syseng log]# fdisk -l | more

Disk /dev/sda: 1000.2 GB, 1000170586112 bytes, 1953458176 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 262144 bytes / 262144 bytes

Disk /dev/sdc: 1000.2 GB, 1000170586112 bytes, 1953458176 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 262144 bytes / 262144 bytes
Disk label type: gpt
Disk identifier: 4F9BEB58-0A23-4E40-8DDD-84DFC3395F73

# Start End Size Type Name
1 2048 1953456127 931.5G Microsoft basic primary

Disk /dev/sdb: 1000.2 GB, 1000170586112 bytes, 1953458176 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 262144 bytes / 262144 bytes
Disk label type: gpt
Disk identifier: 2BF0C21D-7C39-4A98-A6BE-8A2FD35DAEB6

# Start End Size Type Name
1 2048 411647 200M EFI System EFI System Partition
2 411648 935935 256M Microsoft basic
3 935936 1953456127 931G Linux LVM

Disk /dev/sdd: 1000.2 GB, 1000170586112 bytes, 1953458176 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 262144 bytes / 262144 bytes
Disk label type: gpt
Disk identifier: 7453CCF7-62B1-4FAC-8D71-16B334499121

# Start End Size Type Name
1 2048 1953456127 931.5G Microsoft basic primary

Disk /dev/sde: 400.1 GB, 400054812672 bytes, 781357056 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 262144 bytes / 262144 bytes
[root@inf-105-9-syseng log]# ls -ld /sys/block/sd*/device
lrwxrwxrwx 1 root root 0 Jun 14 16:56 /sys/block/sda/device -> ../../../1:1:0:0
lrwxrwxrwx 1 root root 0 Jun 14 16:56 /sys/block/sdb/device -> ../../../1:1:0:1
lrwxrwxrwx 1 root root 0 Jun 14 16:56 /sys/block/sdc/device -> ../../../1:1:0:2
lrwxrwxrwx 1 root root 0 Jun 14 16:56 /sys/block/sdd/device -> ../../../1:1:0:3
lrwxrwxrwx 1 root root 0 Jun 14 16:56 /sys/block/sde/device -> ../../../1:1:0:4
lrwxrwxrwx 1 root root 0 Jun 14 16:56 /sys/block/sdf/device -> ../../../1:1:0:5
lrwxrwxrwx 1 root root 0 Jun 14 16:56 /sys/block/sdg/device -> ../../../1:1:0:6
lrwxrwxrwx 1 root root 0 Jun 14 16:56 /sys/block/sdh/device -> ../../../1:1:0:7
lrwxrwxrwx 1 root root 0 Jun 14 16:56 /sys/block/sdi/device -> ../../../1:1:0:8
lrwxrwxrwx 1 root root 0 Jun 14 16:56 /sys/block/sdj/device -> ../../../1:1:0:9
lrwxrwxrwx 1 root root 0 Jun 14 16:56 /sys/block/sdk/device -> ../../../0:0:0:0

[benbaez · Accepted solution]

The following solution is working so far.  The page RHEL/CentOS Kickstart partition scheme with disk by-path IDs and automation via PXE led me down a path not dependent with /dev/sda for installing the OS.

A bash script is called during anaconda to apply the HP RAID config and has the following snippet.  This snippet is to maintain compatibility with existing profiles.

if [ -e /mnt/source/pretools/raid-configs/hp-raid.ini ]; then
    printMsg "Loading RAID configuration..."

    ## Get the full by-path ID of the OS disk if defined
    SDA_STR=$(grep "BY_PATH_SDA" /tmp/ks.part.txt|awk -F= '{print $2}')
    if [ ! -z "${SDA_STR}" ]; then
      printMsg "Partition map has defined find sda by path."
      SDA_PATH=$(ls -lh /dev/disk/by-path/ | grep "${SDA_STR} " | awk '{print $9}')
      if [ -z "${SDA_PATH}" ]; then
        errorExit "Find sda by path is set in partition map as '${SDA_STR}', but not found."
      else
        printMsg "Using ${SDA_PATH} for SDA_PATH_TAG"
        sed -i "s/SDA_PATH_TAG/${SDA_PATH}/g" /tmp/ks.part.txt
        printMsg "sed exit code: $?"
      fi
    fi

The above reads the target specific partition map below to know it is not a normal "sda" root os install and to replace tags with the 1st disk path found on the hardware.

bootloader --location=mbr --driveorder=disk/by-path/SDA_PATH_TAG --append="crashkernel=auto rhgb quiet"
zerombr
clearpart --all --drives=sda,sdb,sdc,sdd,sde,sdf,sdg,sdh,sdi,sdj --initlabel
#ignoredisk --only-use=sda
part /boot --fstype=ext4 --size=256 --ondisk=disk/by-path/SDA_PATH_TAG
#EFIpart /boot/efi --fstype=efi --size=50 --grow --maxsize=200
part pv.40 --size=102400 --grow --ondisk=disk/by-path/SDA_PATH_TAG
volgroup mgtvg --pesize=32768 pv.40
logvol swap     --fstype=swap --name=swapvol   --vgname=mgtvg --size=5120
logvol /        --fstype=ext4 --name=rootvol   --vgname=mgtvg --size=5120
logvol /opt     --fstype=ext4 --name=optvol    --vgname=mgtvg --size=5120
logvol /var     --fstype=ext4 --name=varvol    --vgname=mgtvg --size=51200
#NUM_RAID_VOLUMES=10
#BY_PATH_SDA=scsi-0:1:0:0

I hope a solution is found to keep sda constant with first disk after using ssascripting.  The DL360 Gen10 seems to not have this problem so far with testing.  We have 10s of thousands of servers for our SaaS cloud, so we want to maintain homogeneity as much as possible.

Thanks.

[Jimmy Vance]

You are going down one of the known optional paths.  The %pre script I posted in the other thread is a similar but different way.  I didn't mean to get hung up on the 'sd' device, but you might also encounter this at some point so I wanted to make you aware of it.  As I originally stated because of the way Linux enumerates the PCI bus as the kernel is loading, there really isn't a solution to keep a specific disk using standard /dev/sdX naming, this is why everyone has moved to use udev rules based from UUID or some other static information from a device (i.e. MAC address for NIC as they can have enumeration issues also). As you have discovered the only time this is an issue is during automated installation scenarios with tools such as kickstart. Once the system is installed having udev rules in place makes it a non issue.  For your configuration, you just need to decide what %pre option is the easiest to work for the installation environment.  You can search 'linux drive enumeration' to read several articles/papers that go into detail on the subject.

[benbaez]

Thanks, I read the enumeration for drives a few weeks ago.  The push for using UUIDs won't work because they don't survive disk logical changes.  We would get many servers in an order and deploying them has to be as automated as possible since people in the DC would just rack and cable them up.  This drive by path method should work well as long as HPE is consistent with bay configuration and cabling.

For networking we can't use MACs either other than the PXE boot NIC.  Again because deployment is automated with little human intervention, we have to rely on mfg card strings.  Thankfully we haven't encountered random differences with reboots regarding NICs.

Thanks again