How to check if a MapR volume exists with Python?

[orville1549]

I am struggling with MapR volumes... I am trying to extend a volume structure which has also folders. In other words I want to check if a volume path exists and if not to create it, but I don't know how I can make that check. I am not that familiar with MapR and volumes.

[fred Passeron]

Hi

It might be interesting for you to take a look at the following Workshop-on-Demand .

Browse to the following page and click on register, it's free and availbale 24/7.  This simple workshop may help you in better understand how to uinteract programmatically with a Data Fabric Environment.

HPE Ezmeral Data Fabric 101 – Get to know the basics around the data fabric

Curious about the HPE Ezmeral Data Fabric filesystem? Take this workshop to understand the basics of administering the filesystem through a DevOps experience to create/delete/update volumes and their respective features, as well as setup snapshots for business continuity. In this workshop, you’ll learn to setup mirrors for disaster recovery and create security policies for applications.

A simple curl command on the api server of a data fabric cluster could help you list all volumes existing on it.:

https://docs.datafabric.hpe.com/62/ClusterAdministration/data/volumes/ViewVols.html

CLI : 

maprcli volume list

REST API Call : 

http[s]://<host>:<port>/rest/volume/list[?<parameters>]

https://docs.datafabric.hpe.com/62/ReferenceGuide/volume-list.html

Regards.

Fred

[okalinin]

Hi,

Assuming you want to create a volume and mount it to, say, /path/to/volume1, you need to make sure that /path/to directory exists and /path/to/volume1 does *not* exist. If volume1 directory exists, volume will fail to mount. In MapR-FS, volume creation/mount process will automatically create mount point directory.

In case you would like to check existence of /path/to directory, there are several ways to do that, here are some:

1. Call external command from Python, e.g. 'hadoop fs -test -d /path/to'. It will return 0 if directory exists, and 1 if directory doesn't exist. If directory desn't exist and you would like to create it, you may use 'hadoop fs -mkdir -p /path/to' command. This option will work in case you execute your script on MapR client or cluster node.
2. Install HttpFS service on the cluster and use HttpFS REST API to achieve the same as in (1).
3. Mount MapR-FS via NFS or POSIX client and use conventional means to validate whether directory exists or not.

For volume creation itself, you may use MCS, 'maprcli' command line command or equilvalent REST API. See e.g. this documentation page [1] for details.

Hope this helps.

Cheers,
Alex

[1] https://docs.datafabric.hpe.com/62/ClusterAdministration/data/volumes/CreateVols.html

[Harshkohli]

Usually mapr volume being mounted is nothing but is like a directory being mounted. So as Alex mentioned we are basically checking if the directory exists or not.

I think you can use python subprocess library and then issue the linux commands to check the directory status and then issue maprcli volume create command to create them if they don't exist.

It should be an if statement under which you can get this executed. You can research on subprocess library and its usage over google.

[tdunning]

So, you can check for directories existing or not using the normal Python commands:

So, on my cluster with the data fabric mounted as is conventional at /mapr, I can look to see if a volume is mounted at a particular path or if a directory exists at that path:

>>> import os.path
>>> os.path.exists("/mapr/c0/user/tdunning")
True

But that doesn't distinguish between a volume and a directory. I happen to know that this is a volume, but Python can't tell the difference directly.

I can use the subprocess and json packages as Harsh suggested to help with this:

import os.path
import subprocess
import json
def strip_path(path):
   s = os.path.split(path)
   if (s[0] == "/mapr"):
     return "/"
   else:
     if (s[0] == "/"):
       return path
     else:
       return os.path.join(strip_path(s[0]), s[1])

def check_volume(path):
   s = json.loads(subprocess.run(["maprcli", "volume", "info", "-path", strip_path(path), "-json"], stdout=subprocess.PIPE).stdout)
   return s["status"] == 'OK'

The strip_path call is necessary because maprcli doesn't want to think about the /mapr/c0 prefix.

Another way to do this is to test to see if the magic sub-directory for snapshots exists. This depends, of course, on having the file system mounted via NFS or FUSE client:

import os.path
def check_snaps(path):
    snaps = os.path.join(path, ".snapshot")
    return os.path.exists(snaps)  and  os.path.isdir(snaps)