Better error handiling in Morpheus cypher package

[rajmartha]

HI All

i belive morpheus cypher package needs to have a better error handling,.

we are running the below error almost every day when 5 to 8 jobs run at the same time, which use the morpheus cypher package. the reason might be most of our jobs, i guess 5 to 8 of them are scheduled at the same time, out of 8 jobs one the job is erroring out with below error.

Traceback (most recent call last):
  File "<stdin>", line 44, in <module>
  File "<stdin>", line 25, in main
  File "/var/opt/morpheus/morpheus-local/workspace/py-f85459cd-28c3-4ae8-9dc0-19b435a3b701/lib/python3.11/site-packages/morpheuscypher/__init__.py", line 85, in __init__
    self.set_cypher_endpoint()
  File "/var/opt/morpheus/morpheus-local/workspace/py-f85459cd-28c3-4ae8-9dc0-19b435a3b701/lib/python3.11/site-packages/morpheuscypher/__init__.py", line 103, in set_cypher_endpoint
    if LooseVersion(data['buildVersion']) < LooseVersion('5.3.3'):
                    ~~~~^^^^^^^^^^^^^^^^
KeyError: 'buildVersion'

i downloaded the source code for morpheus cypher package

at line number 103 in def set_cypher_endpoint(self) function,

current code,

    def set_cypher_endpoint(self):
        appliance_url = self.url
        url = appliance_url + "/api/ping"
        headers = {'content-type': 'application/json', 'Authorization': "BEARER %s" % self.token}
        with warnings.catch_warnings():
            warnings.filterwarnings("ignore", category=Warning)
            r = requests.get(url=url, headers=headers, verify=self.ssl_verify)
        data = r.json()
        ***if LooseVersion(data['buildVersion']) < LooseVersion('5.3.3'):***
            self.cypher_endpoint = "/api/cypher/v1/"
        else:
            self.cypher_endpoint = "/api/cypher/"

i think the code should be

def set_cypher_endpoint(self):
    appliance_url = self.url
    url = appliance_url + "/api/ping"
    headers = {'content-type': 'application/json', 'Authorization': "BEARER %s" % self.token}
    
    with warnings.catch_warnings():
        warnings.filterwarnings("ignore", category=Warning)
        r = requests.get(url=url, headers=headers, verify=self.ssl_verify)

    try:
        data = r.json()
        # Check if 'buildVersion' is present
        if 'buildVersion' in data and LooseVersion(data['buildVersion']) >= LooseVersion('5.3.3'):
            self.cypher_endpoint = "/api/cypher/"
        else:
            # Use the fallback endpoint if buildVersion is missing or < 5.3.3
            self.cypher_endpoint = "/api/cypher/v1/"
    except (ValueError, KeyError):
        # Fallback to /api/cypher/v1/ if there's a JSON error or missing 'buildVersion'
        self.cypher_endpoint = "/api/cypher/v1/"

who ever was developing that package, can you please take a look?

[rajmartha]

@cbunge any suggestions on how we can get the apitoken from the below code?

self.token = morpheus['morpheus']['apiAccessToken']

[cbunge]

I believe the var morpheus.executionLeaseToken allows concurrency for cypher access

[cbunge]

That’s a run once token during an execution and immediately expires the next time that variable is called or when the execution completes.

[rajmartha]

support ticket i opened for this issue has been closed,since this needs to be addressed here. can someone from Morpheus fix this issue please?

[rajmartha]

@cbunge any update on this?

[rajmartha]

@ncelebic @cbunge we are keep getting this exceptions, can one of you please help us here?

Traceback (most recent call last): File "<stdin>", line 44, in <module> File "<stdin>", line 25, in main File "/var/opt/morpheus/morpheus-local/workspace/py-605bc1f9-1e72-4027-9244-d75767499d25/lib/python3.11/site-packages/morpheuscypher/__init__.py", line 122, in get raise Exception(data['msg']) Exception: Unauthorized. You must be authenticated to access this api

Traceback (most recent call last): File "<stdin>", line 44, in <module> File "<stdin>", line 25, in main File "/var/opt/morpheus/morpheus-local/workspace/py-a2c37c59-4866-48ff-80c7-31270aeced57/lib/python3.11/site-packages/morpheuscypher/__init__.py", line 122, in get raise Exception(data['msg']) Exception: Invalid Vault Token

[rajmartha]

@Tyler_Boyd did not see the error so far,.

[rajmartha]

@ncelebic now i am seeing this error

Traceback (most recent call last):
  File "<stdin>", line 44, in <module>
  File "<stdin>", line 25, in main
  File "/var/opt/morpheus/morpheus-local/workspace/py-772fabfd-e479-46de-9432-6b28a09530d0/lib/python3.11/site-packages/morpheuscypher/__init__.py", line 122, in get
    raise Exception(data['msg'])
Exception: Invalid Vault Token

[#10]

Hi @rmartha,

This module is not a core Morpheus product, and that’s why support won’t handle it directly. The repository is located at GitHub - tryfan/python-morpheus-cypher . I’d be happy to review a pull request or address an issue. Please note that there is no SLA for fixing this as this falls under our policy located here: Morpheus Support

[rajmartha]

@cbunge can you explan more about it? Do you have any sample code on how we cna use the var morpheus.executionLeaseToken ?

[rajmartha]

Hi @ncelebic

here is the pull request

github.com/tryfan/python-morpheus-cypher

Update __init__.py

tryfan:master ← MarthaRaj:patch-1

opened 06:51PM - 08 Oct 24 UTC

+11 -5

https://discuss.morpheusdata.com/t/better-error-handiling-in-morpheus-cypher-pac…kage/2085

[rajmartha]

@ncelebic we see this error this morning too,

Traceback (most recent call last):
  File "", line 44, in 
  File "", line 25, in main
  File "/var/opt/morpheus/morpheus-local/workspace/py-04d507ad-579b-4f9d-9f14-f1a0e7c5e196/lib/python3.11/site-packages/morpheuscypher/__init__.py", line 122, in get
    raise Exception(data['msg'])
Exception: Unauthorized. You must be authenticated to access this api

While you might not notice the error with a single execution, our scenario involves jobs that repeatedly execute this module more than ten times simultaneously.

[rajmartha]

Tagging @ncelebic @cbunge

[rajmartha]

Thank you somuch @Tyler_Boyd i did see the Pull request, will test and let you know once the code is merged.

[rajmartha]

Thank you @Tyler_Boyd will let you know if i run into any issues

[tyboyd]

You would just use morpheus[“executionLeaseToken”] instead of morpheus[‘morpheus’][‘apiAccessToken’]

executionLeaseToken is valid during the task execution while apiAccessToken is valid until it is called again.

If you are using the cypher module you would use

from morpheuscypher import Cypher

value = Cypher(morpheus=morpheus,ssl_verify=False,token=morpheus["executionLeaseToken"]).get('secret/test')

print(value)

@ncelebic It might be good to update the code to prefer executionLeaseToken over apiAccessToken. Older versions won’t have executionLeaseToken so I could have it fall back to apiAccessToken. I will make a pull request.

[tyboyd]

You should be good to test now.

If you want to specify the version you can set: ‘morpheus-cypher==0.3.1’ in the additional packages field

[#19]

Thanks for the PR! After looking at this entire block more closely, it was causing more trouble than it was worth in testing. This was based on an API path change in version 5.3.3 of Morpheus, which is no longer relevant. I left an escape hatch open in the README, but otherwise just ripped it out.

I just pushed version 0.3.0 to PyPI. Let me know if you have any issues either in this thread, or you can raise an issue in the github repo.

Thanks!

[#20]

I am not getting this error when I run the module. This looks like a token issue in Morpheus as opposed to a module issue. I would need more detail to offer any more insight.