vSphere: Failed to create VMFS datastore - Cannot change the host configuration.

[Bernd Schlaupitz]

I try to add a LUN as a new Datastore, provided from a HPE MSA 2040 SAS, using a HPE 241 SAS HBA to a vSphere 6.7 System.
The HW is a HPE Proliant 380 Gen9 Server. The MSA2040 is direct attached using SAS Controllers.
The LUN is available in the MSA Utility Software and shown in the "Add Datastore" Wizard as expected.

But during the "Add Datastore" procedure, I get the error "Cannot change the host configuration".

Please find below the summary details of the "Add Datastore" Wizard:
General
Name: MSA2040VDISKA
Type: VMFS
Datastore size: 2.64 TB
Device and Formatting
Disk/LUN: HP Serial Attached SCSI Disk (naa.600c0ff0001e3b26fc879d5f01000000)
Partition Format: GPT
VMFS Version: VMFS 6
Block Size: 1 MB
Space Reclamation Granularity: 1 MB
Space Reclamation Priority Low: Deleted or unmapped blocks are reclaimed on the LUN at low priority

Looking to the device on the ESX Console, is not showing any existing partition on this LUN:
[root@HCDORDEVOPSENP1:~] partedUtil getptbl /vmfs/devices/disks/naa.600c0ff0001e3b26fc879d5f01000000
unknown
352571 255 63 5664055296

The Controller Status of the H241 HBA is the following:

Smart HBA H241 in Slot 3
Bus Interface: PCI
Slot: 3
Serial Number: PDNNL0ARH7Q0IV
Cache Serial Number: PDNNL0ARH7Q0IV
Controller Status: OK
Hardware Revision: B
Firmware Version: 6.60-0
Cache Board Present: False
Controller Temperature (C): 62
Number of Ports: 2 External only
Driver Name: nhpsa
Driver Version: 2.0.30-1OEM
HBA Mode Enabled: True
PCI Address (Domain:Bus:Device.Function): 0000:08:00.0
Negotiated PCIe Data Rate: PCIe 3.0 x8 (7880 MB/s)
Controller Mode: HBA
Pending Controller Mode: HBA
Controller Mode Reboot: Not Required
Port Max Phy Rate Limiting Supported: False
Current Power Mode: MaxPerformance
Survival Mode: Disabled
Host Serial Number: CZ3444FT6M
Sanitize Erase Supported: True
Primary Boot Volume: None
Secondary Boot Volume: None

The MSA 2040 has Version GL210R004 installed.

Does anyone know how to troubleshoot this error?
Is anyone may be able to help or point me to the root cause of this error?

Thank you in advance.
Kind regards Bernd

[Bernd Schlaupitz · Accepted solution]

In the meantime I found the root cause:
The MSA had "unwritable cache" data in its controllers and therefore the LUN was provided only read-only.
Unfortunately this was not shown in the SMU of the MSA2040.
The following command (in a ssh session on the MSA's controller) showed this very clear:
# show unwritable-cache
Unwritable System Cache
-----------------------
Percent of unwritable cache in controller A: 98
Percent of unwritable cache in controller B: 98

As we changed the controllers of this box recently (before connecting to the ESX Server) and created the LUN completely new, there is no useful data in the cache.
Therefore the following command will clear the cache and finally provide the LUN read/writable:
# clear cache
Clearing cache for a volume that is offline or quarantined could result in unrecoverable data loss.
Do you want to continue? (y/n) y
Success: Command completed successfully. - If unwritable cache data existed, it has been cleared.

After some seconds, the "show unwritable-cache" command shows 0 Percent of unwritable data in each controller.
Now adding the Datastore to the vSphere Server worked as expected without any error.

Hope this helps may be others.

kind regards Bernd