Unable to detach Nimble Datastore from the Esxi host

[GaganpreetSingh]

Issue :: Unable to detach Datastore from the Esxi host . 

Nimble iSCSI datastore
Need to remove it
localhost.jgallo.josephfarms.com

Datastore information 

===============

eui.1bbb20f1356db12e6c9ce90015e68b87:1 /vmfs/devices/disks/eui.1bbb20f1356db12e6c9ce90015e68b87:1 564673ca-b2bb0797-7da6-180373f012dd 0 Production Nimble Datastore 1

[root@localhost:~] esxcli storage core device world list -d eui.1bbb20f1356db12e6c9ce90015e68b87
Device World ID Open Count World Name
------------------------------------ -------- ---------- ------------
eui.1bbb20f1356db12e6c9ce90015e68b87 67180 1 hostd-worker
[root@localhost:~] esxcli storage vmfs extent list | grep -i eui.1bbb20f1356db12e6c9ce90015e68b87
[root@localhost:~] esxcli storage vmfs extent list
Volume Name VMFS UUID Extent Number Device Name Partition
-------------------------------- ----------------------------------- ------------- ------------------------------------ ---------
Production Nimble Datastore 2 5652b0e2-15644017-fcd3-180373f012dd 0 eui.7906e9399b849b666c9ce90015e68b87 1
Backup Drobo Datastore 59589fe7-22340e0f-c58c-180373f49f4d 0 naa.6001a620000133313031303030303237 1
datastore1 5a21d563-490a6894-0aec-e0db550a45ce 0 naa.5000c50032309033 3
UniFi Drobo Video Datastore 5bacf581-9484e499-8458-e0db550a45ce 0 naa.6001a620000031363031303030313939 1
Production Nimble Datastore 3 5be294aa-1ed8a8e8-baec-e0db550a45ce 0 eui.06532cc715a170cf6c9ce90015e68b87 1
Production Nimble Datastore 4 5be2c87e-abfea220-fa88-b82a72d3a1b8 0 eui.d72f2bbeb6acd5e66c9ce90015e68b87 1
Production SQL Datastore Data 5be3a373-27330964-84c4-e0db550a45ce 0 eui.64f4f1df84b8505a6c9ce90015e68b87 1
Production SQL Datastore Backups 5be3a39e-78ee2384-5b12-e0db550a45ce 0 eui.3537f4d625c4b7716c9ce90015e68b87 1
[root@localhost:~] esxcfg-scsidevs -m
eui.7906e9399b849b666c9ce90015e68b87:1 /vmfs/devices/disks/eui.7906e9399b849b666c9ce90015e68b87:1 5652b0e2-15644017-fcd3-180373f012dd 0 Production Nimble Datastore 2
naa.6001a620000133313031303030303237:1 /vmfs/devices/disks/naa.6001a620000133313031303030303237:1 59589fe7-22340e0f-c58c-180373f49f4d 0 Backup Drobo Datastore
naa.5000c50032309033:3 /vmfs/devices/disks/naa.5000c50032309033:3 5a21d563-490a6894-0aec-e0db550a45ce 0 datastore1
naa.6001a620000031363031303030313939:1 /vmfs/devices/disks/naa.6001a620000031363031303030313939:1 5bacf581-9484e499-8458-e0db550a45ce 0 UniFi Drobo Video Datastore
eui.06532cc715a170cf6c9ce90015e68b87:1 /vmfs/devices/disks/eui.06532cc715a170cf6c9ce90015e68b87:1 5be294aa-1ed8a8e8-baec-e0db550a45ce 0 Production Nimble Datastore 3
eui.d72f2bbeb6acd5e66c9ce90015e68b87:1 /vmfs/devices/disks/eui.d72f2bbeb6acd5e66c9ce90015e68b87:1 5be2c87e-abfea220-fa88-b82a72d3a1b8 0 Production Nimble Datastore 4
eui.64f4f1df84b8505a6c9ce90015e68b87:1 /vmfs/devices/disks/eui.64f4f1df84b8505a6c9ce90015e68b87:1 5be3a373-27330964-84c4-e0db550a45ce 0 Production SQL Datastore Data
eui.3537f4d625c4b7716c9ce90015e68b87:1 /vmfs/devices/disks/eui.3537f4d625c4b7716c9ce90015e68b87:1 5be3a39e-78ee2384-5b12-e0db550a45ce 0 Production SQL Datastore Backups

[root@localhost:~] esxcli storage filesystem unmount -u 564673ca-b2bb0797-7da6-180373f012dd
No volume with uuid '564673ca-b2bb0797-7da6-180373f012dd' was found

 
[root@localhost:~] esxcli storage filesystem list
Mount Point Volume Name UUID Mounted Type Size Free
------------------------------------------------- -------------------------------- ----------------------------------- ------- ------ -------------- --------------
/vmfs/volumes/5652b0e2-15644017-fcd3-180373f012dd Production Nimble Datastore 2 5652b0e2-15644017-fcd3-180373f012dd true VMFS-5 15392894353408 13655067328512
/vmfs/volumes/59589fe7-22340e0f-c58c-180373f49f4d Backup Drobo Datastore 59589fe7-22340e0f-c58c-180373f49f4d true VMFS-5 17591917608960 9155789193216
/vmfs/volumes/5a21d563-490a6894-0aec-e0db550a45ce datastore1 5a21d563-490a6894-0aec-e0db550a45ce true VMFS-5 138781130752 137761914880
/vmfs/volumes/5bacf581-9484e499-8458-e0db550a45ce UniFi Drobo Video Datastore 5bacf581-9484e499-8458-e0db550a45ce true VMFS-6 35184103653376 4394623959040
/vmfs/volumes/5be294aa-1ed8a8e8-baec-e0db550a45ce Production Nimble Datastore 3 5be294aa-1ed8a8e8-baec-e0db550a45ce true VMFS-6 15392894353408 12692373897216
/vmfs/volumes/5be2c87e-abfea220-fa88-b82a72d3a1b8 Production Nimble Datastore 4 5be2c87e-abfea220-fa88-b82a72d3a1b8 true VMFS-6 15392894353408 9804042993664
/vmfs/volumes/5be3a373-27330964-84c4-e0db550a45ce Production SQL Datastore Data 5be3a373-27330964-84c4-e0db550a45ce true VMFS-6 4397778075648 4396197871616
/vmfs/volumes/5be3a39e-78ee2384-5b12-e0db550a45ce Production SQL Datastore Backups 5be3a39e-78ee2384-5b12-e0db550a45ce true VMFS-6 4397778075648 4396197871616
/vmfs/volumes/5a21d55c-fc63218c-0cf6-e0db550a45ce 5a21d55c-fc63218c-0cf6-e0db550a45ce true vfat 299712512 82952192
/vmfs/volumes/57c4a078-f10ef995-c5fb-53b69388d443 57c4a078-f10ef995-c5fb-53b69388d443 true vfat 261853184 105816064
/vmfs/volumes/23771a7f-67d17bef-0114-b68143ba50e0 23771a7f-67d17bef-0114-b68143ba50e0 true vfat 261853184 107024384
/vmfs/volumes/5a21d564-aea5c004-4ebe-e0db550a45ce 5a21d564-aea5c004-4ebe-e0db550a45ce true vfat 4293591040 4259381248

eui.1bbb20f1356db12e6c9ce90015e68b87:1:


vmkernel.log

2018-11-08T18:30:43.287Z cpu3:67868 opID=23092902)ScsiDevice: 1860: Device eui.1bbb20f1356db12e6c9ce90015e68b87 has been turned off administratively.
2018-11-08T18:44:14.444Z cpu2:65626)WARNING: vmw_psp_rr: psp_rrSelectPathToActivate:1101: Could not select path for device "eui.1bbb20f1356db12e6c9ce90015e68b87".
2018-11-08T18:44:14.445Z cpu22:65625)WARNING: vmw_psp_rr: psp_rrSelectPathToActivate:1101: Could not select path for device "eui.1bbb20f1356db12e6c9ce90015e68b87".
2018-11-08T18:45:21.025Z cpu2:2529522)WARNING: vmw_psp_rr: psp_rrSelectPathToActivate:1101: Could not select path for device "eui.1bbb20f1356db12e6c9ce90015e68b87".
2018-11-08T18:50:21.034Z cpu13:2600641)WARNING: vmw_psp_rr: psp_rrSelectPathToActivate:1101: Could not select path for device "eui.1bbb20f1356db12e6c9ce90015e68b87".
2018-11-08T18:54:04.317Z cpu5:67857 opID=257d822d)ScsiDevice: 1867: Device eui.1bbb20f1356db12e6c9ce90015e68b87 has been turned on administratively.
2018-11-08T18:54:04.317Z cpu3:65851)WARNING: vmw_psp_rr: psp_rrSelectPathToActivate:1101: Could not select path for device "eui.1bbb20f1356db12e6c9ce90015e68b87".
2018-11-08T18:54:04.318Z cpu2:65849)NMP: nmp_UnregisterDeviceEvents:2240: Failed to unregister events 0x180 from device "eui.1bbb20f1356db12e6c9ce90015e68b87". Not found
2018-11-08T18:54:04.319Z cpu2:65849)StorageApdHandler: 1062: Freeing APD handle 0x43051e0865d0 [eui.1bbb20f1356db12e6c9ce90015e68b87]
2018-11-08T20:03:41.995Z cpu12:67857 opID=9e2259fb)ScsiDeviceIO: 7871: QErr is correctly set to 0x0 for device eui.1bbb20f1356db12e6c9ce90015e68b87.
2018-11-08T20:03:41.995Z cpu12:67857 opID=9e2259fb)ScsiDeviceIO: 8368: Could not detect setting of sitpua for device eui.1bbb20f1356db12e6c9ce90015e68b87. Error Not supported.
2018-11-08T20:03:42.006Z cpu22:67857 opID=9e2259fb)ScsiDevice: 4369: Successfully registered device "eui.1bbb20f1356db12e6c9ce90015e68b87" from plugin "NMP" of type 0
2018-11-08T20:03:42.007Z cpu22:67857 opID=9e2259fb)NMP: nmp_DeviceUpdateProtectionInfo:747: Set protection info for device 'eui.1bbb20f1356db12e6c9ce90015e68b87', Enabled: 0 ProtType: 0x0 Guard: 0x0 ProtMask: 0x0
2018-11-08T20:17:23.678Z cpu18:67870 opID=16f5c9a4)ScsiDevice: 1860: Device eui.1bbb20f1356db12e6c9ce90015e68b87 has been turned off administratively.
[root@localhost:/var/log]

2018-11-08T18:23:50.688Z cpu0:67868 opID=7ddd1d90)LVM: 16593: File system '[Production Nimble Datastore 1, 564673ca-b2bb0797-7da6-180373f012dd]' (LV 564673c2-4d9f736d-8a39-18037
3f012dd) un-mounted.
2018-11-08T18:25:21.011Z cp

vobd.log
4673c2-4d9f736d-8a39-180373f012dd) un-mounted.
2018-11-08T18:23:50.689Z: [vmfsCorrelator] 17440839483128us: [esx.audit.vmfs.volume.umounted] File system [Production Nimble Datastore 1, 564673ca-b2bb0797-7da6-180373f012dd] on volume 564673c2-4d9f736d-8a39-180373f012dd has been safely un-mounted. The datastore is no longer accessible on this host.
2018-11-08T18:30:43.288Z: [scsiCorrelator] 17440824866178us: [vob.scsi.device.state.off] Device eui.1bbb20f1356db12e6c9ce90015e68b87 has been turned off administratively.
2018-11-08T18:30:43.288Z: [scsiCorrelator] 17441252082170us: [esx.problem.scsi.device.state.off] Device: eui.1bbb20f1356db12e6c9ce90015e68b87 has been turned off administratively


Steps performed 

===========

Unmounted and detached it from the Vcenter server 

Do see the lun in connected list for the host. 

Cannot remove lun from connected devices 

followed the article 

https://kb.vmware.com/s/article/2004605?docid=2004684 

[mamatadesaiNim]

Assuming you are still seeing devices, keep following steps from the KB.  Use esxcli storage core device detached list to list detached devices and  esxcli storage core device detached remove -d EUI_ID  to remove detached devices.  Also confirm the initiator groups from the array have been removed from the volumes.  Rescan from the ESXi host.

OR... the next time, just use Nimble vCenter Plugin!  This makes deleting a VMFS datastore from ESXi hosts a breeze with one click!

Nimble vCenter Plugin Delete