Solid amber drive lights, but iLO reports OK and drives appear functional

[praabjerg]

We are having a strange and disconcerting issue with the disks in our Apollo 4200 Gen10 server.
Or rather, we do not seem to be having an actual issue, which is what's strange.

We have several drives that are showing solid amber lights indicating a drive failure (if they were just predictive failures they should be blinking). We are currently running them in a ZFS setup and discovered this when we went to replace a drive that ZFS had reported read fails on. The drive ZFS had reported read fails on happens to be one of the drives with amber light.
But several of the drives had amber lights and all of them (including the one with ZFS read fails) are reported as green OK in iLO and by ssacli, and ZFS has reported no issues apart from with the one drive. Our monitoring system uses ssacli and has thus also reported all green.

Since we're currently running them in a ZFS setup, the drives are Unassigned on the controller and exposed to the OS. We are also seeing nothing in the Integrated Management Log to indicate what is going on.

The drives with solid amber lights are:
3I:1:14
3I:1:17
3I:1:21
3I:1:52

So three of them are in the inner drive cage and one is in the back cage.
They are all 14TB HDDs of models MB014000JWTFD or MB014000JZYVP. The ones in the back cage are relatively new, purchased within the past year.

The inner and back drives obviously run hotter and some of them have experienced temperatures in excess of 50 degree celcius, which seems disconcerting but may well be within normal operational temperatures (and we have had no warnings or alerts on this as far as I can tell).

Does anyone have any idea what we should be looking for here? Is it possible to get more information out of the drives themselves?

[praabjerg]

Now that I've double-checked, I can also add that the drive we just switched (3I:1:17) because of the ZFS read fault is still showing solid amber.

I'm starting to wonder if there might be some minor backplane or controller issue that is not being reported in iLO?

[ngnear]

Thank you for reaching out.

This would require some logs to be analyzed.

My suggestion is to get a case logged with HPE via https://support.hpe.com/connect/s/?language=en_US