Re: HPE Ezmeral Data Fabric (MapR) Keycloak Service Status

SunnyC · ‎08-26-2024

We have installed a 5-node cluster of HPE Ezmeral Data Fabric (MapR) on RHEL 8.8. We are using Core 7.7.0 and EEP 9.2.2.

We are facing difficulty solving the NODE_ALARM_SERVICE_KEYCLOAK_DOWN alarm in MCS.

The Keycloak service was not configured or started when the installation was performed with intaller version 1.18.0.

We follower below link to reconfig the service but the alarm still exist.

https://support.hpe.com/hpesc/public/docDisplay?docId=sf000102238en_us&docLocale=en_US

Below is the error lines:

We notice that keycloak is trying to start a webservice at 8080 by default which was already in used by hbaserest.

Is this the cause of the alram? Is there any way i can change the port used by keycloak?

Mr_Techie · ‎09-13-2024

Hello @SunnyC

Yes, the conflict with port 8080 could be causing the 'NODE_ALARM_SERVICE_KEYCLOAK_DOWN' alarm. If Keycloak is trying to start on a port that's already in use (by 'hbaserest' in this case), it will fail to bind to that port, which could lead to the service not starting and triggering the alarm.

To resolve this, you can change the port that Keycloak uses.Here's how you can configure Keycloak to use a different port:

Locate the Keycloak configuration file. This is usually found in the standalone.xml or keycloak.conf (depending on your version and installation) under the Keycloak installation directory.

In standalone.xml, look for the HTTP listener configuration:<socket-binding name="http" port="8080"/>

Change the port from 8080 to an available port, for example, 8081: <socket-binding name="http" port="8081"/>

If using keycloak.conf, you can modify the http-port directive: "http-port=8081"

After changing the port, restart Keycloak for the changes to take effect: systemctl restart keycloak

Check Keycloak is now running on the new port by checking the services or using a command like "netstat":netstat -tuln | grep 8081

After these steps, the Keycloak service should no longer conflict with 'hbaserest', and the 'NODE_ALARM_SERVICE_KEYCLOAK_DOWN' alarm should be cleared if the service starts correctly.

I'm not a expert here, just trying to help here by finding the info. I hope this helps and fixes the issue

ParvYadav · ‎09-16-2024

HI @SunnyC

By default keycloak service uses port 6443. Can you please check if there were any changes done in the keycloak.conf file.

I'm an HPE employee.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]

SunnyC · ‎09-16-2024

Thank you for the update. Sorry for the late reply, there's been a lot going on while putting this into a test.

Below is more info about my environment and my finding:

installation packages:

mapr-installer-1.18.0.202405301411.rpm.tgz
mapr-mep-v9.2.2.202404221243.rpm.tgz
mapr-v7.7.0GA.rpm.tgz

After I change the default port from 8080 to 8081 in the keycloak.conf file, I have to rerun configure.sh with the keycloak flag for it to start running.

After the configure.sh run successfully, the openjdk 17 seems broken and my zookeeper failed to restart. I have to re-install my java for it to work again. The java executable change color to red text black background which implies ca - File with capability in my env.

After all these changes, the collectd service in my environment start failing with below traceback in my warden log:

2024-09-17 10:51:07,437 ERROR ServiceRun [collectd_monitor]: Error while running command: [/opt/mapr/collectd/collectd-5.12.0/etc/init.d/collectd, start]

2024-09-17 10:51:07,437 ERROR ServiceRun [collectd_monitor]: CollectD effective IDs are user: mapr group mapr

log4j:WARN Continuable parsing error 22 and column 16

log4j:WARN The content of element type "appender" must match "(errorHandler?,param*,rollingPolicy?,triggeringPolicy?,connectionSource?,layout?,filter*,appender-ref*)".

log4j:WARN Unrecognized element triggeringPolicy

java.lang.reflect.InvocationTargetException

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:568)

at com.mapr.fs.ShimLoader.loadNativeLibrary(ShimLoader.java:351)

at com.mapr.fs.ShimLoader.load(ShimLoader.java:228)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:568)

at org.apache.hadoop.conf.CoreDefaultProperties.<clinit>(CoreDefaultProperties.java:63)

at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:804)

at org.collectd.CreateStreams.main(CreateStreams.java:21)

Caused by: java.lang.UnsatisfiedLinkError: /tmp/mapr-mapr-libMapRClient.7.7.0.0-mapr.so: /tmp/mapr-mapr-libMapRClient.7.7.0.0-mapr.so: failed to map segment from shared object

at java.base/jdk.internal.loader.NativeLibraries.load(Native Method)

at java.base/jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:388)

at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:232)

at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:174)

at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2394)

at java.base/java.lang.Runtime.load0(Runtime.java:755)

at java.base/java.lang.System.load(System.java:1953)

at com.mapr.fs.shim.LibraryLoader.load(LibraryLoader.java:31)

... 13 more

java.lang.reflect.InvocationTargetException

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:568)

at org.apache.hadoop.conf.CoreDefaultProperties.<clinit>(CoreDefaultProperties.java:63)

at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:804)

at org.collectd.CreateStreams.main(CreateStreams.java:21)

Caused by: java.lang.ExceptionInInitializerError

at com.mapr.fs.ShimLoader.load(ShimLoader.java:245)

... 7 more

Caused by: java.lang.reflect.InvocationTargetException

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:568)

at com.mapr.fs.ShimLoader.loadNativeLibrary(ShimLoader.java:351)

at com.mapr.fs.ShimLoader.load(ShimLoader.java:228)

... 7 more

Caused by: java.lang.UnsatisfiedLinkError: /tmp/mapr-mapr-libMapRClient.7.7.0.0-mapr.so: /tmp/mapr-mapr-libMapRClient.7.7.0.0-mapr.so: failed to map segment from shared object

at java.base/jdk.internal.loader.NativeLibraries.load(Native Method)

at java.base/jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:388)

at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:232)

at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:174)

at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2394)

at java.base/java.lang.Runtime.load0(Runtime.java:755)

at java.base/java.lang.System.load(System.java:1953)

at com.mapr.fs.shim.LibraryLoader.load(LibraryLoader.java:31)

... 13 more

java.lang.reflect.InvocationTargetException

at java.base/jdk.internal.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:568)

at com.mapr.fs.ShimLoader.injectNativeLoader(ShimLoader.java:286)

at com.mapr.fs.ShimLoader.load(ShimLoader.java:225)

at com.mapr.baseutils.cldbutils.CLDBRpcCommonUtils.<clinit>(CLDBRpcCommonUtils.java:98)

at com.mapr.baseutils.JVMProperties.<clinit>(JVMProperties.java:21)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:568)

at org.apache.hadoop.conf.CoreDefaultProperties.<clinit>(CoreDefaultProperties.java:80)

at org.apache.hadoop.conf.Configuration.<clinit>(Configuration.java:804)

at org.collectd.CreateStreams.main(CreateStreams.java:21)

Caused by: java.lang.LinkageError: loader 'platform' attempted duplicate class definition for com.mapr.fs.shim.LibraryLoader. (com.mapr.fs.shim.LibraryLoader is in unnamed module of loader 'platform')

at java.base/java.lang.ClassLoader.defineClass1(Native Method)

at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)

... 14 more

java.lang.RuntimeException: Failure loading MapRClient.

at com.mapr.fs.ShimLoader.injectNativeLoader(ShimLoader.java:307)