topic Re: mapr7.2 is not supporting KDC encryption type aes256-cts-hmac-sha1-96 in HPE Ezmeral Software platform

mapr7.2 is not supporting KDC encryption type aes256-cts-hmac-sha1-96

Satish_Infa — Mon, 11 Nov 2024 04:22:46 GMT

[root@invrh88maprk05 conf]# kinit -kt mapr.keytab mapr/mapR-invrh88maprk05@PLATFORMKRB.COM

[root@invrh88maprk05 conf]# klist -kte mapr.keytab
Keytab name: FILE:mapr.keytab
KVNO Timestamp Principal
---- ------------------- ------------------------------------------------------
3 09/16/2024 11:59:59 mapr/mapR-invrh88maprk05@PLATFORMKRB.COM (DEPRECATED:des-cbc-crc)
3 09/16/2024 11:59:59 mapr/mapR-invrh88maprk05@PLATFORMKRB.COM (DEPRECATED:des-cbc-md5)
3 09/16/2024 11:59:59 mapr/mapR-invrh88maprk05@PLATFORMKRB.COM (DEPRECATED:arcfour-hmac)
3 09/16/2024 11:59:59 mapr/mapR-invrh88maprk05@PLATFORMKRB.COM (aes256-cts-hmac-sha1-96)
3 09/16/2024 11:59:59 mapr/mapR-invrh88maprk05@PLATFORMKRB.COM (aes128-cts-hmac-sha1-96)
3 09/16/2024 11:59:59 mapr/invrh88maprk05.informatica.com@PLATFORMKRB.COM (DEPRECATED:des-cbc-crc)
3 09/16/2024 11:59:59 mapr/invrh88maprk05.informatica.com@PLATFORMKRB.COM (DEPRECATED:des-cbc-md5)
3 09/16/2024 11:59:59 mapr/invrh88maprk05.informatica.com@PLATFORMKRB.COM (DEPRECATED:arcfour-hmac)
3 09/16/2024 11:59:59 mapr/invrh88maprk05.informatica.com@PLATFORMKRB.COM (aes256-cts-hmac-sha1-96)
3 09/16/2024 11:59:59 mapr/invrh88maprk05.informatica.com@PLATFORMKRB.COM (aes128-cts-hmac-sha1-96)
3 09/16/2024 11:59:59 HTTP/invrh88maprk05.informatica.com@PLATFORMKRB.COM (DEPRECATED:des-cbc-crc)
3 09/16/2024 11:59:59 HTTP/invrh88maprk05.informatica.com@PLATFORMKRB.COM (DEPRECATED:des-cbc-md5)
3 09/16/2024 11:59:59 HTTP/invrh88maprk05.informatica.com@PLATFORMKRB.COM (DEPRECATED:arcfour-hmac)
3 09/16/2024 11:59:59 HTTP/invrh88maprk05.informatica.com@PLATFORMKRB.COM (aes256-cts-hmac-sha1-96)
3 09/16/2024 12:00:00 HTTP/invrh88maprk05.informatica.com@PLATFORMKRB.COM (aes128-cts-hmac-sha1-96)
[root@invrh88maprk05 conf]#

[root@invrh88maprk05 conf]# maprlogin kerberos

Failure in kerberos handshake Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP-REQ - AES256 CTS mode with HMAC SHA1-96)

Re: mapr7.2 is not supporting KDC encryption type aes256-cts-hmac-sha1-96

okalinin — Wed, 13 Nov 2024 13:17:22 GMT

Kerberos AP-REQ message is a message from the client (maprlogin utility) to CLDB. Looks like CLDB can't decrypt it because it doesn't have the key of enctype aes256-cts-hmac-sha1-96. I would suggest to check keytab on CLDB primary node to make sure aes256-cts-hmac-sha1-96 key is present.

Query: mapr7.2 is not supporting KDC encryption type aes256-cts-hmac-sha1-96

support_s — Fri, 15 Nov 2024 08:05:55 GMT

Hello,

Let us know if you were able to resolve the issue.

If you have no further query, and you are satisfied with the answer then kindly mark the topic as Solved so that it is helpful for all community members.

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

Re: mapr7.2 is not supporting KDC encryption type aes256-cts-hmac-sha1-96

Satish_Infa — Fri, 15 Nov 2024 08:09:39 GMT

below Output is from the primary host of the cluster

Re: mapr7.2 is not supporting KDC encryption type aes256-cts-hmac-sha1-96

okalinin — Mon, 18 Nov 2024 15:35:33 GMT

Side remark: you need to use mapr/invrh88maprk05.informatica.com@PLATFORMKRB.COM principal, i.e.

# kinit -kt mapr.keytab mapr/invrh88maprk05.informatica.com@PLATFORMKRB.COM

Other than that, there could be various reasons for why CLDB doesn't find the key to decrypt AP-REQ message. One option is to restart CLDBs to make sure latest Kerberos configuration and keytab have been picked up. If that still has no effect, lower level Kerberos troubleshooting is required. Uncomment the line in /opt/mapr/conf/env.sh starting with '#MAPR_KERBEROS_DEBUG' on CLDB and on the client sides. Note: to take effect, CLDB must be restarted. Then reproduce the issue and inspect log files cldb.out and maprlogin*<username>*log in /opt/mapr/logs for Kerberos events.

Also, compare Kerberos configurations on all nodes to make sure they are the same and up to date. For example, it is necessary to make sure that enctype aes256-cts-hmac-sha1-96 is actually active in your Kerberos host and KDC configuration. Debugging Kerberos events as mentioned above usually helps to uncover any misconfigurations.

Generally, Ezmeral Data Fabric doesn't imply any limitations on Kerberos authentication as it relies on host Kerberos configuration and implementation. Experience shows, that once Kerberos configurations and keytabs are correct and active across all cluster nodes, Kerberos related issues get resolved.

Re: mapr7.2 is not supporting KDC encryption type aes256-cts-hmac-sha1-96

Satish_Infa — Thu, 21 Nov 2024 06:34:46 GMT

Keytab Initialization is working

[root@invrh88maprk05 conf]# kinit -kt mapr.keytab mapr/mapR-invrh88maprk05@PLATFORMKRB.COM

[root@invrh88maprk05 conf]#

[root@invrh88maprk05 ~]# cat /opt/mapr/conf/env.sh

MAPR_KERBEROS_DEBUG="-Dsun.security.krb5.debug=true -Dsun.security.spnego.debug=true -Djavax.net.debug=all"

[root@invrh88maprk05 logs]# cat maprcli-mapr-5000.log

Header: hostName: invrh88maprk05.informatica.com, Time Zone: India Standard Time, processName: MapRCLI, processId: 3769566, MapR Build Version: 7.2.0.0.20230118195227.GA
2024-11-21 11:59:08,896 INFO com.mapr.cliframework.driver.CLIMainDriver [main]: [node, list, -columns, id, ,ip, -json]
2024-11-21 11:59:09,050 INFO com.mapr.baseutils.cldbutils.CLDBRpcCommonUtils [main]: ZKConnect: invrh88maprk05.informatica.com:5181
2024-11-21 11:59:09,369 INFO com.mapr.util.zookeeper.ZKDataRetrieval [main-EventThread]: Process path: null. Event state: SyncConnected. Event type: None
2024-11-21 11:59:09,371 INFO com.mapr.util.zookeeper.ZKDataRetrieval [main]: Connected to ZK: invrh88maprk05.informatica.com:5181
2024-11-21 11:59:09,372 INFO com.mapr.util.zookeeper.ZKDataRetrieval [main]: Getting servicesMap
2024-11-21 11:59:09,388 INFO com.mapr.util.zookeeper.ZKDataRetrieval [main-EventThread]: Process path: null. Event state: SaslAuthenticated. Event type: None
2024-11-21 11:59:09,416 WARN com.mapr.util.zookeeper.ZKDataRetrieval [main]: Can not get children of /services or it's children with error: KeeperErrorCode = NoNode for /services/hbasethrift/master
2024-11-21 11:59:09,445 WARN com.mapr.util.zookeeper.ZKDataRetrieval [main]: Can not get children of /services or it's children with error: KeeperErrorCode = NoNode for /services/hbaserest/master
2024-11-21 11:59:09,471 INFO com.mapr.util.zookeeper.ZKDataRetrieval [main]: Getting configServicesMap
2024-11-21 11:59:09,554 INFO com.mapr.cli.common.ListCommand [main]: [hasMore] Last fetched record count is 0. no more records to fetch
2024-11-21 11:59:09,586 INFO com.mapr.cliframework.driver.CLIMainDriver [main]: {
"timestamp":1732170549581,
"timeofday":"2024-11-21 11:59:09.581 GMT+0530 AM",
"status":"OK",
"total":1,
"data":[
{
"id":"5838786529940032736",
"ip":"10.65.190.96",
"hostname":"invrh88maprk05.informatica.com"
}
]
}
Header: hostName: invrh88maprk05.informatica.com, Time Zone: India Standard Time, processName: MapRCLI, processId: 3771353, MapR Build Version: 7.2.0.0.20230118195227.GA
2024-11-21 11:59:13,569 INFO com.mapr.cliframework.driver.CLIMainDriver [main]: [node, cldbmaster, -json]
2024-11-21 11:59:13,756 INFO com.mapr.cliframework.driver.CLIMainDriver [main]: {
"timestamp":1732170553753,
"timeofday":"2024-11-21 11:59:13.753 GMT+0530 AM",
"status":"OK",
"total":1,
"data":[
{
"cldbprimary":"ServerID: 5838786529940032736 HostName: invrh88maprk05.informatica.com"
}
]
}

[root@invrh88maprk05 logs]# cat cldb.log

label_and_topo: 1 min_repl: 2 desired_repl: 3 honor_topology: true honor_label: true
2024-11-21 12:01:10,273 INFO CriticallyUnderReplicatedQueue [Repl]: replicating container 2077...total_available: 1 with_desired_label_and_topo: 1 min_repl: 2 desired_repl: 3 honor_topology: true honor_label: true
2024-11-21 12:01:10,273 INFO CriticallyUnderReplicatedQueue [Repl]: replicating container 2062...total_available: 1 with_desired_label_and_topo: 1 min_repl: 2 desired_repl: 3 honor_topology: true honor_label: true
2024-11-21 12:01:10,273 INFO CriticallyUnderReplicatedQueue [Repl]: replicating container 2058...total_available: 1 with_desired_label_and_topo: 1 min_repl: 2 desired_repl: 3 honor_topology: true honor_label: true
2024-11-21 12:01:10,273 INFO CriticallyUnderReplicatedQueue [Repl]: replicating container 2083...total_available: 1 with_desired_label_and_topo: 1 min_repl: 2 desired_repl: 3 honor_topology: true honor_label: true
2024-11-21 12:01:10,273 INFO CriticallyUnderReplicatedQueue [Repl]: replicating container 2084...total_available: 1 with_desired_label_and_topo: 1 min_repl: 2 desired_repl: 3 honor_topology: true honor_label: true
2024-11-21 12:01:10,273 INFO CriticallyUnderReplicatedQueue [Repl]: replicating container 2085...total_available: 1 with_desired_label_and_topo: 1 min_repl: 2 desired_repl: 3 honor_topology: true honor_label: true
2024-11-21 12:01:10,273 INFO CriticallyUnderReplicatedQueue [Repl]: replicating container 2086...total_available: 1 with_desired_label_and_topo: 1 min_repl: 2 desired_repl: 3 honor_topology: true honor_label: true
2024-11-21 12:01:10,273 INFO CriticallyUnderReplicatedQueue [Repl]: replicating container 2087...total_available: 1 with_desired_label_and_topo: 1 min_repl: 2 desired_repl: 3 honor_topology: true honor_label: true
2024-11-21 12:01:10,273 INFO CriticallyUnderReplicatedQueue [Repl]: replicating container 2088...total_available: 1 with_desired_label_and_topo: 1 min_repl: 2 desired_repl: 3 honor_topology: true honor_label: true
2024-11-21 12:01:10,274 INFO CriticallyUnderReplicatedQueue [Repl]: replicating container 2089...total_available: 1 with_desired_label_and_topo: 1 min_repl: 2 desired_repl: 3 honor_topology: true honor_label: true
2024-11-21 12:01:55,347 INFO ReplicationHandlerThread [Repl]: <PRIORITY_REPLICATION> A=552; P=2220; F=2220; QS=552;
2024-11-21 12:01:55,347 INFO ReplicationHandlerThread [Repl]: <UNDER_REPLICATION> A=3; P=14; F=14; QS=3;
2024-11-21 12:03:54,786 ERROR MapRLoginServlet [qtp668760567-197]: Exception handling kerberos
org.ietf.jgss.GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP-REQ - AES256 CTS mode with HMAC SHA1-96)
at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:859) ~[java.security.jgss:?]
at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:361) ~[java.security.jgss:?]
at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:303) ~[java.security.jgss:?]
at sun.security.jgss.spnego.SpNegoContext.GSS_acceptSecContext(SpNegoContext.java:910) ~[java.security.jgss:?]
at sun.security.jgss.spnego.SpNegoContext.acceptSecContext(SpNegoContext.java:559) ~[java.security.jgss:?]
at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:361) ~[java.security.jgss:?]
at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:303) ~[java.security.jgss:?]
at com.mapr.fs.cldb.http.login.MapRLoginServlet$1.run(MapRLoginServlet.java:699) ~[cldb-7.2.0.0-mapr.jar:7.2.0.0-mapr]
at com.mapr.fs.cldb.http.login.MapRLoginServlet$1.run(MapRLoginServlet.java:693) ~[cldb-7.2.0.0-mapr.jar:7.2.0.0-mapr]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at javax.security.auth.Subject.doAs(Subject.java:423) ~[?:?]
at com.mapr.fs.cldb.http.login.MapRLoginServlet.handleKerberosAuth(MapRLoginServlet.java:692) ~[cldb-7.2.0.0-mapr.jar:7.2.0.0-mapr]
at com.mapr.fs.cldb.http.login.MapRLoginServlet.doPost(MapRLoginServlet.java:133) ~[cldb-7.2.0.0-mapr.jar:7.2.0.0-mapr]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) ~[javax.servlet-api-3.1.0.jar:3.1.0]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450) ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) ~[jetty-security-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555) ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410) ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164) ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: sun.security.krb5.KrbException: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP-REQ - AES256 CTS mode with HMAC SHA1-96
at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:278) ~[java.security.jgss:?]
at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:149) ~[java.security.jgss:?]
at sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:139) ~[java.security.jgss:?]
at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:832) ~[java.security.jgss:?]
... 52 more
2024-11-21 12:03:55,500 INFO ReplicationHandlerThread [Repl]: <PRIORITY_REPLICATION> P=4416; F=4416; QS=552;
2024-11-21 12:03:55,500 INFO ReplicationHandlerThread [Repl]: <UNDER_REPLICATION> P=24; F=24; QS=3;

Re: mapr7.2 is not supporting KDC encryption type aes256-cts-hmac-sha1-96

okalinin — Thu, 21 Nov 2024 06:56:19 GMT

The files to share after enabling debug are cldb.out and maprlogin*<user>*log