Community
ProLiant Servers (ML,DL,SL)

External Disk Enclosures attached to P408e is first controller

 
benbaez
Occasional Advisor

‎04-07-2020 01:38 PM

‎04-07-2020 01:38 PM

External Disk Enclosures attached to P408e is first controller

I have a DL380 Gen10 that has (2) disk enclosures  attached to a P408-e card in PCIe Slot1 that is first in boot order over an internal P816i on CentOS 7.  Is there any way of making the internal P816i be the first controller?

The P408e card that has 2 external SAS ports bus path is pci-0000:12:00.0 while the internal P816i is pci-0000:5c:00.0 Since 12 is lower than 5c, I'm assuming udev is making P408e controller first.
[root@inf-105-13 ~]# ls -lh /dev/disk/by-path/
total 0
lrwxrwxrwx 1 root root  9 Apr  3 06:49 pci-0000:12:00.0-scsi-0:1:0:0 -> ../../sda
lrwxrwxrwx 1 root root  9 Apr  3 06:49 pci-0000:12:00.0-scsi-0:1:0:1 -> ../../sdb
lrwxrwxrwx 1 root root  9 Apr  3 06:49 pci-0000:12:00.0-scsi-0:1:0:10 -> ../../sdk
lrwxrwxrwx 1 root root 10 Apr  3 06:49 pci-0000:12:00.0-scsi-0:1:0:1-part1 -> ../../sdb1
lrwxrwxrwx 1 root root  9 Apr  3 06:49 pci-0000:12:00.0-scsi-0:1:0:2 -> ../../sdc
lrwxrwxrwx 1 root root 10 Apr  3 06:49 pci-0000:12:00.0-scsi-0:1:0:2-part1 -> ../../sdc1
lrwxrwxrwx 1 root root  9 Apr  3 06:49 pci-0000:12:00.0-scsi-0:1:0:3 -> ../../sdd
lrwxrwxrwx 1 root root 10 Apr  3 06:49 pci-0000:12:00.0-scsi-0:1:0:3-part1 -> ../../sdd1
lrwxrwxrwx 1 root root  9 Apr  3 06:49 pci-0000:12:00.0-scsi-0:1:0:4 -> ../../sde
lrwxrwxrwx 1 root root 10 Apr  3 06:49 pci-0000:12:00.0-scsi-0:1:0:4-part1 -> ../../sde1
lrwxrwxrwx 1 root root  9 Apr  3 06:49 pci-0000:12:00.0-scsi-0:1:0:5 -> ../../sdf
lrwxrwxrwx 1 root root 10 Apr  3 06:49 pci-0000:12:00.0-scsi-0:1:0:5-part1 -> ../../sdf1
lrwxrwxrwx 1 root root  9 Apr  3 06:49 pci-0000:12:00.0-scsi-0:1:0:6 -> ../../sdg
lrwxrwxrwx 1 root root 10 Apr  3 06:49 pci-0000:12:00.0-scsi-0:1:0:6-part1 -> ../../sdg1
lrwxrwxrwx 1 root root  9 Apr  3 06:49 pci-0000:12:00.0-scsi-0:1:0:7 -> ../../sdh
lrwxrwxrwx 1 root root 10 Apr  3 06:49 pci-0000:12:00.0-scsi-0:1:0:7-part1 -> ../../sdh1
lrwxrwxrwx 1 root root  9 Apr  3 06:49 pci-0000:12:00.0-scsi-0:1:0:8 -> ../../sdi
lrwxrwxrwx 1 root root 10 Apr  3 06:49 pci-0000:12:00.0-scsi-0:1:0:8-part1 -> ../../sdi1
lrwxrwxrwx 1 root root  9 Apr  3 06:49 pci-0000:12:00.0-scsi-0:1:0:9 -> ../../sdj
lrwxrwxrwx 1 root root 10 Apr  3 06:49 pci-0000:12:00.0-scsi-0:1:0:9-part1 -> ../../sdj1
lrwxrwxrwx 1 root root  9 Apr  3 06:49 pci-0000:5c:00.0-scsi-0:1:0:0 -> ../../sdl
lrwxrwxrwx 1 root root 10 Apr  3 06:49 pci-0000:5c:00.0-scsi-0:1:0:0-part1 -> ../../sdl1
lrwxrwxrwx 1 root root 10 Apr  3 06:49 pci-0000:5c:00.0-scsi-0:1:0:0-part2 -> ../../sdl2
lrwxrwxrwx 1 root root 10 Apr  3 06:49 pci-0000:5c:00.0-scsi-0:1:0:0-part3 -> ../../sdl3
lrwxrwxrwx 1 root root 10 Apr  3 06:49 pci-0000:5c:00.0-scsi-0:1:0:0-part4 -> ../../sdl4
lrwxrwxrwx 1 root root 10 Apr  3 06:49 pci-0000:5c:00.0-scsi-0:1:0:0-part5 -> ../../sdl5
lrwxrwxrwx 1 root root 10 Apr  3 06:49 pci-0000:5c:00.0-scsi-0:1:0:0-part6 -> ../../sdl6
lrwxrwxrwx 1 root root 10 Apr  3 06:49 pci-0000:5c:00.0-scsi-0:1:0:0-part7 -> ../../sdl7
lrwxrwxrwx 1 root root  9 Apr  3 06:49 pci-0000:5c:00.0-scsi-0:1:0:1 -> ../../sdm
lrwxrwxrwx 1 root root 10 Apr  3 06:49 pci-0000:5c:00.0-scsi-0:1:0:1-part1 -> ../../sdm1
 
[root@inf-105-13 vmcfg]# lsscsi
[0:0:0:0]    enclosu HPE      D3610            3.64  -
[0:0:1:0]    enclosu HPE      D3610            3.64  -
[0:0:2:0]    enclosu HPE      Smart Adapter    1.98  -
[0:1:0:0]    disk    HPE      LOGICAL VOLUME   1.98  /dev/sda
[0:1:0:1]    disk    HPE      LOGICAL VOLUME   1.98  /dev/sdb
[0:1:0:2]    disk    HPE      LOGICAL VOLUME   1.98  /dev/sdc
[0:1:0:3]    disk    HPE      LOGICAL VOLUME   1.98  /dev/sdd
[0:1:0:4]    disk    HPE      LOGICAL VOLUME   1.98  /dev/sde
[0:1:0:5]    disk    HPE      LOGICAL VOLUME   1.98  /dev/sdf
[0:1:0:6]    disk    HPE      LOGICAL VOLUME   1.98  /dev/sdg
[0:1:0:7]    disk    HPE      LOGICAL VOLUME   1.98  /dev/sdh
[0:1:0:8]    disk    HPE      LOGICAL VOLUME   1.98  /dev/sdi
[0:1:0:9]    disk    HPE      LOGICAL VOLUME   1.98  /dev/sdj
[0:1:0:10]   disk    HPE      LOGICAL VOLUME   1.98  /dev/sdk
[0:2:0:0]    storage HPE      P408e-p SR Gen10 1.98  -
[1:0:0:0]    enclosu HPE      Smart Adapter    1.98  -
[1:1:0:0]    disk    HPE      LOGICAL VOLUME   1.98  /dev/sdl
[1:1:0:1]    disk    HPE      LOGICAL VOLUME   1.98  /dev/sdm
[1:2:0:0]    storage HPE      P816i-a SR Gen10 1.98  -



0 Kudos
Reply
3 REPLIES 3
sudhirsingh
HPE Pro

‎04-07-2020 07:12 PM - edited ‎04-07-2020 08:07 PM

‎04-07-2020 07:12 PM - edited ‎04-07-2020 08:07 PM

Re: External Disk Enclosures attached to P408e is first controller

Hi,

In the BIOS you can select  primary boot controller even delete the boot options from boot order list,

However i believe this is not the case of hardware but this is by design of Linux kernel.

Device numbering it is a behavior of the OS and which device to mount first, purely OS decides.

If you are facing booting issue because this then you can use persistent mount point feature, which allows to fix the mount point to /sda/sdb…etc through UUID

Hope this helps!

regards,

Sudhir

 

While I am an HPE Employee, all of my comments (whether noted or not), are my own and are not any official representation of the company

>Accept or Kudo

0 Kudos
Reply
benbaez
Occasional Advisor

‎04-08-2020 10:10 AM - last edited on ‎04-08-2020 10:36 PM by

‎04-08-2020 10:10 AM - last edited on ‎04-08-2020 10:36 PM by

Re: External Disk Enclosures attached to P408e is first controller

Hi Sudhir,

Thanks for the reply.

Agreed that Linux udev is the main issue, but HPE should have internal RAID controllers with lower PCIe address than slots.  That way udev would properly select the correct device numbering scheme.

I tried the Legacy BIOS Primary boot option but did not work.  Maybe will try UEFI and removing UEFI boot options.

With UUID, how does that work with only knowing the hardware?  I have for instance a hundred of these servers that arrive from HPE configured by the same BOM.  I have a script to automate installation that has worked with previous generations of HPE servers.  Basically the RAID config is loaded during %pre in anaconda. 

Is there a way to assign a label to arrays created on the controllers? 

Can the same VolumeUniqueID be used on different servers so I can use that to target the array?

Using a label or VolumeUniqueID for an array would allow me to use a script to pull the sdX mapping and dynamically set the install of OS and other partitions during anaconda.

Thanks for the help!

Ben

 

 

; Date captured: Thu Apr  2 04:05:55 2020

; Version: 3.40.3.0 2018-12-06

Action= Configure
Method= Custom


; __________________________ Controller Specifications SLOT 0 ________________________________
;
; Controller HPE Smart Array P816i-a SR Gen10, FirmwareVersion 1.98, License Keys Supported
; SerialNumber ---------
; DriverName smartpqi
; DriverVersion Linux 1.1.4-115
; SSDSmartPath Supported
Controller= SLOT 0
; PowerMode= MaxPerformance
; SurvivalMode= Enabled
ReadCache= 10
WriteCache= 90
WriteCacheBypassThreshold= 1040
RebuildPriority= High
ExpandPriority= Medium
ParallelSurfaceScanCount= 1
SurfaceScanMode= Idle
SurfaceScanDelay= 3
Latency= Disable
NoBatteryWriteCache= Disabled
MNPDelay= 60
IRPEnable= Disabled
DPOEnable= Disabled
ElevatorSortEnable= Enabled
QueueDepth= Automatic
PredictiveSpareActivation= Disable
SanitizeLock= None
; PendingSanitizeLock= None

; Drive Write Cache Policies - uncomment to modify
; DriveWriteCacheUnconfigured= DEFAULT
; DriveWriteCacheConfigured= DISABLE


; Array Specifications
Array= A
; Array Drive Type is SAS
; Array Block Size is 512 bytes
; Array Free Space    0.0 GBytes
; 1I:3:2 (SAS HDD, 2.3 TB), 1I:3:1 (SAS HDD, 2.3 TB)
Drive= 1I:3:2, 1I:3:1
OnlineSpare= 1I:3:3

; Logical Drive Specifications
LogicalDrive= 1
RAID= 1
Size= 1907697
; SizeBlocks= 3906963632
Sectors= 32
StripSize= 256
Caching= Enabled
; VolumeUniqueID= 600508B1001C6A742A06DDC8BE044763

; Array Specifications
Array= B
; Array Drive Type is Solid State SAS
; Array Block Size is 512 bytes
; Array Free Space    0.0 GBytes
; 1I:3:4 (Solid State SAS SSD, 1.3 TB), 2I:3:5 (Solid State SAS SSD, 1.3 TB), 2I:3:6 (Solid State SAS SSD, 1.3 TB), 2I:3:7 (Solid State SAS SSD, 1.3 TB), 2I:3:8 (Solid State SAS SSD, 1.3 TB), 3I:2:3 (Solid State SAS SSD, 1.3 TB), 3I:2:2 (Solid State SAS SSD, 1.3 TB), 3I:2:1 (Solid State SAS SSD, 1.3 TB)
Drive= 1I:3:4, 2I:3:5, 2I:3:6, 2I:3:7, 2I:3:8, 3I:2:3, 3I:2:2, 3I:2:1
SSDSmartPath= Enable
OnlineSpare= 3I:2:4

; Logical Drive Specifications
LogicalDrive= 2
RAID= 6
Size= 10988328
; SizeBlocks= 22504096432
Sectors= 32
StripSize= 256
; VolumeUniqueID= 600508B1001CD32248FAF2795FA168E5

; PORT Specifications
Port=1I
; PortMode= Mixed

; PORT Specifications
Port=2I
; PortMode= Mixed

; PORT Specifications
Port=3I
; PortMode= Mixed

; PORT Specifications
Port=4I
; PortMode= Mixed

; __________________________ Controller Specifications SLOT 1 ________________________________
;
; Controller HPE Smart Array P408e-p SR Gen10, FirmwareVersion 1.98, License Keys Supported
; SerialNumber --------
; DriverName smartpqi
; DriverVersion Linux 1.1.4-115
; SSDSmartPath Supported
Controller= SLOT 1
; PowerMode= MaxPerformance
; SurvivalMode= Enabled
ReadCache= 10
WriteCache= 90
WriteCacheBypassThreshold= 1040
RebuildPriority= High
ExpandPriority= Medium
ParallelSurfaceScanCount= 1
SurfaceScanMode= Idle
SurfaceScanDelay= 3
Latency= Disable
NoBatteryWriteCache= Disabled
MNPDelay= 60
IRPEnable= Disabled
DPOEnable= Disabled
ElevatorSortEnable= Enabled
QueueDepth= Automatic
PredictiveSpareActivation= Disable
SanitizeLock= None
; PendingSanitizeLock= None

; Drive Write Cache Policies - uncomment to modify
; DriveWriteCacheUnconfigured= DEFAULT
; DriveWriteCacheConfigured= DISABLE


; Array Specifications
Array= A
; Array Drive Type is SAS
; Array Block Size is 512 bytes
; Array Free Space    0.0 GBytes
; 1E:1:1 (SAS HDD, 4.7 TB), 2E:1:1 (SAS HDD, 4.7 TB)
Drive= 1E:1:1, 2E:1:1
OnlineSpare= 1E:1:12, 2E:1:12

; Logical Drive Specifications
LogicalDrive= 1
RAID= 1
Size= 3815415
; SizeBlocks= 7813971632
Sectors= 32
StripSize= 256
Caching= Enabled
; VolumeUniqueID= 600508B1001C64140ED3A8DC1779E79A

; Array Specifications
Array= B
; Array Drive Type is SAS
; Array Block Size is 512 bytes
; Array Free Space    0.0 GBytes
; 1E:1:2 (SAS HDD, 4.7 TB), 2E:1:2 (SAS HDD, 4.7 TB)
Drive= 1E:1:2, 2E:1:2
OnlineSpare= 1E:1:12, 2E:1:12

; Logical Drive Specifications
LogicalDrive= 2
RAID= 1
Size= 3815415
; SizeBlocks= 7813971632
Sectors= 32
StripSize= 256
Caching= Enabled
; VolumeUniqueID= 600508B1001CDEE7810463451D304B51

; Array Specifications
Array= C
; Array Drive Type is SAS
; Array Block Size is 512 bytes
; Array Free Space    0.0 GBytes
; 1E:1:3 (SAS HDD, 4.7 TB), 2E:1:3 (SAS HDD, 4.7 TB)
Drive= 1E:1:3, 2E:1:3
OnlineSpare= 1E:1:12, 2E:1:12

; Logical Drive Specifications
LogicalDrive= 3
RAID= 1
Size= 3815415
; SizeBlocks= 7813971632
Sectors= 32
StripSize= 256
Caching= Enabled
; VolumeUniqueID= 600508B1001C257648ADCB5DFC4DEC7B

; Array Specifications
Array= D
; Array Drive Type is SAS
; Array Block Size is 512 bytes
; Array Free Space    0.0 GBytes
; 1E:1:4 (SAS HDD, 4.7 TB), 2E:1:4 (SAS HDD, 4.7 TB)
Drive= 1E:1:4, 2E:1:4
OnlineSpare= 1E:1:12, 2E:1:12

; Logical Drive Specifications
LogicalDrive= 4
RAID= 1
Size= 3815415
; SizeBlocks= 7813971632
Sectors= 32
StripSize= 256
Caching= Enabled
; VolumeUniqueID= 600508B1001C0E2E0A3E9EF90F9EC74E

; Array Specifications
Array= E
; Array Drive Type is SAS
; Array Block Size is 512 bytes
; Array Free Space    0.0 GBytes
; 1E:1:5 (SAS HDD, 4.7 TB), 2E:1:5 (SAS HDD, 4.7 TB)
Drive= 1E:1:5, 2E:1:5
OnlineSpare= 1E:1:12, 2E:1:12

; Logical Drive Specifications
LogicalDrive= 5
RAID= 1
Size= 3815415
; SizeBlocks= 7813971632
Sectors= 32
StripSize= 256
Caching= Enabled
; VolumeUniqueID= 600508B1001C9E239116B65EF8589338

; Array Specifications
Array= F
; Array Drive Type is SAS
; Array Block Size is 512 bytes
; Array Free Space    0.0 GBytes
; 1E:1:6 (SAS HDD, 4.7 TB), 2E:1:6 (SAS HDD, 4.7 TB)
Drive= 1E:1:6, 2E:1:6
OnlineSpare= 1E:1:12, 2E:1:12

; Logical Drive Specifications
LogicalDrive= 6
RAID= 1
Size= 3815415
; SizeBlocks= 7813971632
Sectors= 32
StripSize= 256
Caching= Enabled
; VolumeUniqueID= 600508B1001CE902B53861AC763B5773

; Array Specifications
Array= G
; Array Drive Type is SAS
; Array Block Size is 512 bytes
; Array Free Space    0.0 GBytes
; 1E:1:7 (SAS HDD, 4.7 TB), 2E:1:7 (SAS HDD, 4.7 TB)
Drive= 1E:1:7, 2E:1:7
OnlineSpare= 1E:1:12, 2E:1:12

; Logical Drive Specifications
LogicalDrive= 7
RAID= 1
Size= 3815415
; SizeBlocks= 7813971632
Sectors= 32
StripSize= 256
Caching= Enabled
; VolumeUniqueID= 600508B1001C4476F7EA73BC76320105

; Array Specifications
Array= H
; Array Drive Type is SAS
; Array Block Size is 512 bytes
; Array Free Space    0.0 GBytes
; 1E:1:8 (SAS HDD, 4.7 TB), 2E:1:8 (SAS HDD, 4.7 TB)
Drive= 1E:1:8, 2E:1:8
OnlineSpare= 1E:1:12, 2E:1:12

; Logical Drive Specifications
LogicalDrive= 8
RAID= 1
Size= 3815415
; SizeBlocks= 7813971632
Sectors= 32
StripSize= 256
Caching= Enabled
; VolumeUniqueID= 600508B1001CF3ABBC70A423BD0AE0F5

; Array Specifications
Array= I
; Array Drive Type is SAS
; Array Block Size is 512 bytes
; Array Free Space    0.0 GBytes
; 1E:1:9 (SAS HDD, 4.7 TB), 2E:1:9 (SAS HDD, 4.7 TB)
Drive= 1E:1:9, 2E:1:9
OnlineSpare= 1E:1:12, 2E:1:12

; Logical Drive Specifications
LogicalDrive= 9
RAID= 1
Size= 3815415
; SizeBlocks= 7813971632
Sectors= 32
StripSize= 256
Caching= Enabled
; VolumeUniqueID= 600508B1001C8ABF1A9DDD08419506C0

; Array Specifications
Array= J
; Array Drive Type is SAS
; Array Block Size is 512 bytes
; Array Free Space    0.0 GBytes
; 1E:1:10 (SAS HDD, 4.7 TB), 2E:1:10 (SAS HDD, 4.7 TB)
Drive= 1E:1:10, 2E:1:10
OnlineSpare= 1E:1:12, 2E:1:12

; Logical Drive Specifications
LogicalDrive= 10
RAID= 1
Size= 3815415
; SizeBlocks= 7813971632
Sectors= 32
StripSize= 256
Caching= Enabled
; VolumeUniqueID= 600508B1001C10EBF456EB7E076293FB

; Array Specifications
Array= K
; Array Drive Type is SAS
; Array Block Size is 512 bytes
; Array Free Space    0.0 GBytes
; 1E:1:11 (SAS HDD, 4.7 TB), 2E:1:11 (SAS HDD, 4.7 TB)
Drive= 1E:1:11, 2E:1:11
OnlineSpare= 1E:1:12, 2E:1:12

; Logical Drive Specifications
LogicalDrive= 11
RAID= 1
Size= 3815415
; SizeBlocks= 7813971632
Sectors= 32
StripSize= 256
Caching= Enabled
; VolumeUniqueID= 600508B1001CE02AD50D5C588527B8A0

; PORT Specifications
Port=1E
; PortMode= Mixed

; PORT Specifications
Port=2E
; PortMode= Mixed

 

 

 

0 Kudos
Reply
benbaez
Occasional Advisor

‎04-08-2020 04:41 PM

‎04-08-2020 04:41 PM

Re: External Disk Enclosures attached to P408e is first controller

Is there anyway of setting the VolumeUniqueID or labeling each LUN with a user defined string?  I need a user defined parameter that I can load with my RAID config that would identify each LUN.  So that I can script installation of os and other filesystems across hundreds of servers.  SCSI-ID, UUID, WWN,SERIAL are all different on each system.  The path can change depending on controller order udev decides.  I really need to be able to set a string in RAID config per LUN so that I can script a cookie cutter anaconda kick for multiple servers.  I would hope that running a command such as below would return a user defined LUN string.

 

[root@inf-105-13-fongtest ~]# udevadm info -n /dev/sdl
P: /devices/pci0000:5b/0000:5b:00.0/0000:5c:00.0/host2/target2:1:0/2:1:0:0/block/sdl
N: sdl
S: disk/by-id/scsi-3600508b1001ce311885d1303f02b7b5a
S: disk/by-id/wwn-0x600508b1001ce311885d1303f02b7b5a
S: disk/by-path/pci-0000:5c:00.0-scsi-0:1:0:0
E: DEVLINKS=/dev/disk/by-id/scsi-3600508b1001ce311885d1303f02b7b5a /dev/disk/by-id/wwn-0x600508b1001ce311885d1303f02b7b5a /dev/disk/by-path/pci-0000:5c:00.0-scsi-0:1:0:0
E: DEVNAME=/dev/sdl
E: DEVPATH=/devices/pci0000:5b/0000:5b:00.0/0000:5c:00.0/host2/target2:1:0/2:1:0:0/block/sdl
E: DEVTYPE=disk
E: ID_BUS=scsi
E: ID_MODEL=LOGICAL_VOLUME
E: ID_MODEL_ENC=LOGICAL\x20VOLUME\x20\x20
E: ID_PART_TABLE_TYPE=gpt
E: ID_PATH=pci-0000:5c:00.0-scsi-0:1:0:0
E: ID_PATH_TAG=pci-0000_5c_00_0-scsi-0_1_0_0
E: ID_REVISION=1.98
E: ID_SCSI=1
E: ID_SCSI_SERIAL=PEYHD0DRHC00Q3
E: ID_SERIAL=3600508b1001ce311885d1303f02b7b5a
E: ID_SERIAL_SHORT=600508b1001ce311885d1303f02b7b5a
E: ID_TYPE=disk
E: ID_VENDOR=HPE
E: ID_VENDOR_ENC=HPE\x20\x20\x20\x20\x20
E: ID_WWN=0x600508b1001ce311
E: ID_WWN_VENDOR_EXTENSION=0x885d1303f02b7b5a
E: ID_WWN_WITH_EXTENSION=0x600508b1001ce311885d1303f02b7b5a
E: MAJOR=8
E: MINOR=176
E: SUBSYSTEM=block
E: TAGS=:systemd:
E: USEC_INITIALIZED=1000

  

0 Kudos
Reply
The opinions expressed above are the personal opinions of the authors, not of Hewlett Packard Enterprise. By using this site, you accept the Terms of Use and Rules of Participation.