HPE Ezmeral Data Fabric (MapR) Disk Utilized Became Zero After Restoring VM Snapshots

[AlexHD]

We have 2 environments installed MapR v7.7 on VMs, one is 4 clusters environments and the other one is 5 clusters environment. Both environments having 1 cluster's disk utilization became zero after restoring from VM snapshots, restored all 4 or 5 VMs.

We checked and confirmed there is no data loss and able to access data via Drill or Hadoop command. However, the disk in 0% utilization cluster seems cannot be used. We tried to put about 8 GB data into Hadoop but the utilization is still keeping in 0%, and there is no disk failure logs and error log in warden.

The questions are:

1. Why the disk utilization is 0%? 

    It's the first time happen, we have experience on restoring VMs and it works well.

2. How can we make the disk become “usable”?

[Dave Olker]

When disk utilization is not balanced across the cluster nodes, my first suggestion would be to enable the Disk Balancer.  The documentation begins here: https://docs.ezmeral.hpe.com/datafabric-customer-managed/79/AdministratorGuide/DiskSpaceBalancer.html

There are subsequent docs in the same section that describe how to enable and configure the feature, how to start a balancer execution, etc.

You can also investigate the Volume Balancer: https://docs.ezmeral.hpe.com/datafabric-customer-managed/79/ClusterAdministration/admin/cluster/VolumeBalancer.html

If you are using replication, you should also investigate the Role Balancer: https://docs.ezmeral.hpe.com/datafabric-customer-managed/79/AdministratorGuide/ReplicationRoleBalancer.html

[support_s]

Hello,

Let us know if you were able to resolve the issue.

If you are satisfied with the answers then kindly click the "Accept As Solution" button for the most helpful response so that it is beneficial to all community members.

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

[AlexHD]

@Dave Olker 

Thank you for your reply!

We would like to investigate the problem, but there is no any disk failure log and no error in mfs.log.  Any others can be look into? 

Also, do you think the problem (hard disk became 0 usage, no data) related to VM restoration?   We afraid MapR will classify a hard disk as new or even starting disk formatting after restoration in some situations, and want to know how to prevent this happen again.

The disk is now look like working.  We've follow your solution and tried the disk balancer and role balancer, the disk is being used but it not quite balance. (min 25% usages vs max 45% usages)  The result and setting as below.  Is this alright?  The disk utilization is similar for all nodes before restore, only around 2% different.

[AlexHD]

@Dave Olker 

"cldb.balancer.disk.deltaToRepopulateStoragePoolsBins":"5"

 Is this mean the size different can be around 5%?

[Dave Olker]

Hi AlexHD,

I'm glad the steps you took seem to have resolved the disk balancing issue.  As for your question about that CLDB tunable, that appears to be precisely what it is used to check.  I looked at the CLDB source code and found:

    /*
     * Repopulate the lists if the cluster-average has changed by +/- 5%
     */

That's in the section of code where the CLDB is checking disk utilization and determining whether it needs to do a new round of rebalancing.  I see no reason to change any of those tunables at this time as it seems your disks are now reasonably balanced.

Regards,

Dave