DL380 G9: Enable HDD uid on CentOS using hpssacli

[Ronny_]

Hi there,

I'm having trouble to get the identification led on a DL380G9 working. The server is oacked with H240 Controllers, no LV configured, just presenting 24 1.8TB SAS Disks to the system. Operating System is CentOS 7.

Now, using hpssacli works to get information about controller and disks. But when I try to set the identification LED to on, the server is blocked for several seconds and I get an error (see below). Is there another way to enable this LEDs? I am using hpssacli-2.40-13.0.x86_64.

=> ctrl all show status

Smart HBA H240 in Slot 1 (HBA Mode)
Controller Status: OK

Smart HBA H240 in Slot 2 (HBA Mode)
Controller Status: OK

Smart HBA H240 in Slot 3 (HBA Mode)
Controller Status: OK

=> ctrl slot=1 pd all show status

physicaldrive 1I:3:1 (port 1I:box 3:bay 1, 1800.3 GB): OK
physicaldrive 1I:3:2 (port 1I:box 3:bay 2, 1800.3 GB): OK
physicaldrive 1I:3:3 (port 1I:box 3:bay 3, 1800.3 GB): OK
physicaldrive 1I:3:4 (port 1I:box 3:bay 4, 1800.3 GB): OK
physicaldrive 2I:3:5 (port 2I:box 3:bay 5, 1800.3 GB): OK
physicaldrive 2I:3:6 (port 2I:box 3:bay 6, 1800.3 GB): OK
physicaldrive 2I:3:7 (port 2I:box 3:bay 7, 1800.3 GB): OK
physicaldrive 2I:3:8 (port 2I:box 3:bay 8, 1800.3 GB): OK

=> ctrl slot=1 pd 1I:3:1 modify led=on

Error: This operation is not supported with the current configuration. Use the
"show" command on devices to show additional details about the
configuration.

Thanks in advance

Ronny

[TTr]

I don't have access to a CLI right now but in all the examples I see, the "pd" does not have the port component, only the box and bay. Try 

=> ctrl slot=1 pd 3:1 modify led=on

[Jimmy Vance]

I don't think that command is supported when the controller is in HBA mode

[Ronny_]

Thanks, but no change.

[Ronny_]

Oh :(

Is there any other chance to identifiy disks?

[TTr]

The quickspecks document has numbers on each disk slot on page 65,66. I  

https://www.hpe.com/h20195/v2/getpdf.aspx/c04346247.pdf

But I think these are for counting the total disks because they renumber the disks when more cages are added. Nevertheless they follow left to right numbering.

To be absolutely certain, create sustained reading using the "dd" command on one of the disks and observe the activity light blink continuously. 

[Ronny_]

Because this machines are in production ... I have no chance to try this with dd :)

[TTr]

The dd command in read mode is harmless and you only need to run it for 2-3 seconds to notice the start and stop of sustained i/o. I have done it many times on production systems. Unless of course your disks have already sustained i/o and the activity light is on solid. What other choice do you have?  In every picture I saw, the disks are numbered 1- 8 in each cage left to right. If you wanted to remove disk sdc and I told you to pull out disk #3, would you do it? No. I wouldn't either.

[Jimmy Vance]

The problem you could run into in the case of a failed disk in HBA mode.  If the system is rebooted before the failed disk is replaced, all the disks after the failure will renumber themselves by one. 

Example:

You have 8 drives /dev/sda - /dev/sdh. If disk 3 (sdc) fails, during a reboot sdd becomes sdc, sde becomes sdd and so on

I haven't verified this on an 'H' or 'P' series controller in HBA mode, but this is how the embedded SATA controller functions

[Ronny_]

Sure, you are right. dd in read mode is harmless. Well, we are running ceph on this nodes and the drive LEDs are blinking and running all the times. I just checked this. So I did not see any change when using dd.

[Ronny_]

So it would make sense to have a LV for each disk?

I am running a ceph cluster on this machines. So there is no need to have RAID or something else. But if I can use the UID when using an LV und would not have this issue with renumbering, then this could be a possible config change to avoid those problems.

Does a LV (RAID 0) slow down such a system compared to HBA mode?

[TTr]

Do you use any disk redundancy at the OS level? The H240 is a low end controller and does not have caching acceleration. I would try raid5 with 4 and 8 disks and do some testing and compare these with what you get now. Is your ceph mostly read? 

The only other way to identify the disks is to record their WWN for each slot before you deploy the disks and match the WWN within the OS.

I would think the RAID0 LVs would add another layer of i/o cycles that with HBA mode disks.

[Jimmy Vance]

I'm not aware of any test results that show what the difference is (if any) between HBA mode and individual RAID 0 arrays.  The H240 has no caching so there isn't any advantage there.  If you were using a 'P' series controller the controller cache would come into play (in RAID mode, cache is disabled in HBA mode)

[Jimmy Vance]

---

@TTr wrote:

Do you use any disk redundancy at the OS level?

---

From a previous post, this is a CEPH configuration. Anything but RAID 0 is frowned upon in a CEPH configuration

[Ronny_]

We use disk redundancy only for the OS disks (two disks at the back of the server).

The disks we are using with ceph are just pushed to ceph "as is". No raid, no LV ... That's the recommended configuration by ceph, because you should use one "OSD" process per disk.

I am just investigating in how ceph identifies disks. I do not know if it uses the /dev/sdx or some wwn/uuid.