Troubleshooting Oracle Public Cloud integration in Morpheus

[dgaharwar]

While trying to integrate Oracle Public Cloud into Morpheus, after following the steps from our guide: Oracle Cloud — Morpheus Docs documentation, and creating API Key after referring to the guide from the Oracle site: Required Keys and OCIDs, the cloud integration detects the Regions, however, fails to load the Compartments on the Cloud Add form.

Below are examples of 3 different scenarios that need to be checked.

1. Successful call to compartments with permissions set CORRECTLY
   

   

2. Successful call to compartments with BAD permissions. The user is NOT allowed to see any compartments
   

   

3. This one means the keypair/user is bad
   

   

All these calls are considered SUCCESS however, the last one is a failure in the authentication.

So, 3 scenarios need checking,

1. Successful call, shows compartments
2. Successful call, user has no access to compartments
3. Call failure due to authentication

Depending on the error, we will know which approach to take to remediate.

But to check this, you will need to turn the log level down to debug for oracle services in /opt/morpheus/conf/logback.xml and add the below logger.

<logger name="com.morpheus.compute.OracleCloudComputeUtility" level="DEBUG"/>

Then on the cloud edit/create form, wait for the get compartments call. Then check the log message for the compartments call. If it is successful and no compartments come back, it’s a permissions issue. If it’s an error and comes back as ‘Not Authorized’ then it’s a keypair issue.

Apart from the above 3 scenarios, you may see another one which will indicate a clock with an ‘invalid datetime range’ or ‘not within allowed clock skew’, something like the below,

2022-11-04_12:26:23.49913 ''[2022-11-04 12:26:23,618] [http-nio-127.0.0.1-8080-exec-3] DEBUG c.m.c.OracleCloudComputeUtility - FAILURE data to https://identity.ap-singapore-1.oraclecloud.com/20160918/compartments, results: {
 2022-11-04_12:26:23.61918     "code": "NotAuthenticated",
 2022-11-04_12:26:23.61919     "message": "Date 'Fri, 04 Nov 2022 12:26:23 UTC' is not within allowed clock skew. Current 'Fri, 04 Nov 2022 12:20:30 UTC', valid datetime range: ['Fri, 04 Nov 2022 12:15:30 UTC', 'Fri, 04 Nov 2022 12:25:31 UTC']"
 2022-11-04_12:26:23.61919 }
 2022-11-04_12:26:23.61919 ''[2022-11-04 12:26:23,619] [http-nio-127.0.0.1-8080-exec-3] WARN  c.m.c.OracleCloudComputeUtility - error: 401 - [code:NotAuthenticated, message:Date 'Fri, 04 Nov 2022 12:26:23 UTC' is not within allowed clock skew. Current 'Fri, 04 Nov 2022 12:20:30 UTC', valid datetime range: ['Fri, 04 Nov 2022 12:15:30 UTC', 'Fri, 04 Nov 2022 12:25:31 UTC']]

See that message in the not authenticated payload from the above log snippet. It indicates that you may not have your Morpheus server tied to NTP and the clock is incorrect.

Part of the Oracle authentication signing is that the DateTime needs to be correct as well. Every payload you send to the Oracle API takes a bunch of information, combined it together, and then signed using the Private Key. But a part of that information is the current time and if it’s skewed too much from the actual time that OCI thinks it is, then the authentication will fail. The Morpheus application just takes the current system time from the server when it builds the OCI payload so it might not be your user/key at all but the time.

So, you will need to change the system clock on your Morpheus server and make sure it’s accurate.

The oci-cli is also very easy to use, and it uses the same exact API to communicate with oracle cloud that we do so, it’s a pretty good testing harness. Once we install it and set the user/tenancy/keypair in the config, this command does exactly what we do to list compartments.

oci iam compartment list --access-level ACCESSIBLE

Also, might be handy for users to have this when troubleshooting OCI.

[#2]

Excellent write-up @dgaharwar thanks for sharing.