topic Re: NoSuchMethodError when trying to execute any Sqoop/Oozie command in HPE Ezmeral Software platform

NoSuchMethodError when trying to execute any Sqoop/Oozie command

MaksREDS — Fri, 06 Aug 2021 20:36:04 GMT

When I try to execute any command in Sqoop 1 Editor (via HUE) the following error occurs:

2021-08-05 09:40:06,417 ERROR ActionStartXCommand:517 - SERVER[node3.address.domain] USER[mapr] GROUP[-] TOKEN[] APP[Batch job for query-sqoop1] JOB[0000019-210805045515667-oozie-mapr-W] ACTION[0000019-210805045515667-oozie-mapr-W@sqoop-0d6e] Error,
java.lang.NoSuchMethodError: 'void com.google.common.base.Preconditions.checkArgument(boolean, java.lang.String, java.lang.Object)'
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientConnectionRetry.<init>(TimelineClientImpl.java:158)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.serviceInit(TimelineClientImpl.java:284)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:180)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.oozie.service.HadoopAccessorService$4.run(HadoopAccessorService.java:572)
at org.apache.oozie.service.HadoopAccessorService$4.run(HadoopAccessorService.java:568)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683)
at org.apache.oozie.service.HadoopAccessorService.createYarnClient(HadoopAccessorService.java:568)
at org.apache.oozie.action.hadoop.JavaActionExecutor.createYarnClient(JavaActionExecutor.java:1648)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1047)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1592)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:243)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:68)
at org.apache.oozie.command.XCommand.call(XCommand.java:291)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:363)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:292)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:210)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

Error looks similiar to: https://support.datafabric.hpe.com/s/article/Spark-application-using-HBase-API-fails-with-java-lang-NoSuchMethodError?language=en_US
The currently installed mapr-hbase version is 1.4.13.0.202105170751, so it is newer than specified in above issue.

Output of rpm -qa|grep mapr

mapr-zookeeper-6.2.0.0.20200915234957.GA-1.x86_64
mapr-hive-2.3.202105190211-1.noarch
mapr-opentsdb-2.4.0.202105181725-1.noarch
mapr-hbase-rest-1.4.13.0.202105170751-1.noarch
mapr-historyserver-2.7.5.0.202105131403-1.x86_64
mapr-collectd-5.10.0.0.202104301721-1.x86_64
mapr-librdkafka-0.11.3.202104281722-1.x86_64
mapr-core-6.2.0.0.20200915234957.GA-1.x86_64
mapr-kafka-2.1.1.200.202105140421-1.noarch
mapr-spark-historyserver-2.4.7.100.202105170615-1.noarch
mapr-hivewebhcat-2.3.202105190211-1.noarch
mapr-kafka-rest-5.1.2.200.202105071030-1.noarch
mapr-oozie-5.1.0.202.202105260138-1.noarch
mapr-mastgateway-6.2.0.0.20200915234957.GA-1.x86_64
mapr-hadoop-client-2.7.5.0.202105131403-1.x86_64
mapr-drill-internal-1.16.1.200.202105180709-1.noarch
mapr-livy-0.7.0.0.202105120521-1.noarch
mapr-tez-0.9.202105070958-1.noarch
mapr-kafka-connect-jdbc-5.1.2.0.202009141042-1.noarch
mapr-oozie-internal-5.1.0.202.202105260138-1.noarch
mapr-schema-registry-internal-5.1.2.200.202105130812-1.noarch
mapr-ksql-5.1.2.200.202105140827-1.noarch
mapr-hivemetastore-2.3.202105190211-1.noarch
mapr-sqoop-1.4.7.202105070954-1.noarch
mapr-client-6.2.0.0.20200915234957.GA-1.x86_64
mapr-core-internal-6.2.0.0.20200915234957.GA-1.x86_64
mapr-ksql-internal-5.1.2.200.202105140827-1.noarch
mapr-schema-registry-5.1.2.200.202105130812-1.noarch
mapr-hiveserver2-2.3.202105190211-1.noarch
mapr-kafka-connect-hdfs-5.1.2.200.202105170800-1.noarch
mapr-loopbacknfs-6.2.0.0.20200915234957.GA-1.x86_64
mapr-hbase-1.4.13.0.202105170751-1.noarch
mapr-drill-1.16.1.200.202105180709-1.noarch
mapr-hbase-regionserver-1.4.13.0.202105170751-1.noarch
mapr-timelineserver-2.7.5.0.202105131403-1.x86_64
mapr-pig-0.17.0.0.202009090610-1.noarch
mapr-hadoop-util-2.7.5.0.202105131403-1.x86_64
mapr-patch-6.2.0.9.20210615161337.GA-1.x86_64
mapr-spark-2.4.7.100.202105170615-1.noarch
mapr-hbasethrift-1.4.13.0.202105170751-1.noarch
mapr-nodemanager-2.7.5.0.202105131403-1.x86_64
mapr-httpfs-1.1.0.0.202105130703-1.noarch
mapr-zk-internal-6.2.0.0.20200915234957.GA-1.x86_64
mapr-fileserver-6.2.0.0.20200915234957.GA-1.x86_64
mapr-hadoop-core-2.7.5.0.202105131403-1.x86_64
mapr-asynchbase-1.8.2.202104220851-1.noarch
mapr-spark-thriftserver-2.4.7.100.202105170615-1.noarch
mapr-flume-1.9.0.100.202101270131-1.noarch
mapr-hue-4.6.0.0.202105070812-1.noarch

I read some solutions of similar issues with hadoop ecosystem - to update outdated version of Guava java library, e.g. https://issues.apache.org/jira/browse/HIVE-22915 .

I replaced all guava-11*.jar I had found with guava-28*.jar, but the error still occurs.

Re: NoSuchMethodError when trying to execute any Sqoop/Oozie command

Harshkohli — Fri, 06 Aug 2021 01:10:22 GMT

Thanks for your post. Few questions:

1) What happens when you try to execute sqoop/oozie job from terminal instead of Hue?

2) Can you check the lib path for sqoop, oozie to confirm the version of guava jar it contains.

My idea will be to first isolate the problem and check if it is actually hue/oozie/sqoop issue so that we can focus on the same.

You can also try to do below on guava jar to check its contents and confirm if the version being used has the method or not.

jar tvf <guava jar>

Re: NoSuchMethodError when trying to execute any Sqoop/Oozie command

MaksREDS — Mon, 09 Aug 2021 08:53:49 GMT

Thank you for your advices.

1) Sqoop jobs works when executed from CLI (command "sqoop") - with no errors.

2) guava-28.2-jre.jar contains the Preconditions class (with checkArgument method), so it is probably not the reason.
Maybe the reason is the com.google.common.base.* class, which could be a part of other java lib used by oozie?

Regards.

Re: NoSuchMethodError when trying to execute any Sqoop/Oozie command

Harshkohli — Fri, 13 Aug 2021 04:42:56 GMT

As I understand you were getting this error when submitting job via Hue.

Can you try to submit the job via Oozie command line?

Re: NoSuchMethodError when trying to execute any Sqoop/Oozie command

MaksREDS — Wed, 18 Aug 2021 08:42:06 GMT

Thank you for your advices.
I have been trying to solve this issue in many ways, but with no success.
When I try to run a workflow from Oozie CLI, the following error occurs:

java.lang.IllegalAccessError: class com.mapr.fs.proto.Security$AuthenticationReqFull tried to access private field com.google.protobuf.AbstractMessage.memoizedSize (com.mapr.fs.proto.Security$AuthenticationReqFull and com.google.protobuf.AbstractMessage are in unnamed module of loader 'app') at com.mapr.fs.proto.Security$AuthenticationReqFull.getSerializedSize(Security.java:12103) at com.google.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:62) at com.mapr.security.maprauth.MaprAuthenticator.doMaprHandshake(MaprAuthenticator.java:162) at com.mapr.security.maprauth.MaprAuthenticator.authenticate(MaprAuthenticator.java:92) at org.apache.oozie.client.AuthOozieClient.createConnection(AuthOozieClient.java:173) at org.apache.oozie.client.OozieClient$1.doExecute(OozieClient.java:520) at org.apache.oozie.client.retry.ConnectionRetriableClient.execute(ConnectionRetriableClient.java:44) at org.apache.oozie.client.OozieClient.createRetryableConnection(OozieClient.java:522) at org.apache.oozie.client.OozieClient.getSupportedProtocolVersions(OozieClient.java:397) at org.apache.oozie.client.OozieClient.validateWSVersion(OozieClient.java:357) at org.apache.oozie.client.OozieClient.createURL(OozieClient.java:473) at org.apache.oozie.client.OozieClient.access$000(OozieClient.java:88) at org.apache.oozie.client.OozieClient$ClientCallable.call(OozieClient.java:567) at org.apache.oozie.client.OozieClient.run(OozieClient.java:887) at org.apache.oozie.cli.OozieCLI.jobCommand(OozieCLI.java:1095) at org.apache.oozie.cli.OozieCLI.processCommand(OozieCLI.java:719) at org.apache.oozie.cli.OozieCLI.run(OozieCLI.java:674) at org.apache.oozie.cli.OozieCLI.main(OozieCLI.java:239) Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = Could not authenticate, Exception while getting ticket and key

The CLI command is:

oozie job -oozie="https://oozie_node_hostname:11443/oozie" -config /tmp/job/job.properties -run

I use the same job/workflow files as automatically generated by Hue.

Re: NoSuchMethodError when trying to execute any Sqoop/Oozie command

Harshkohli — Thu, 19 Aug 2021 02:33:20 GMT

Thanks for the update.

1) What do you get when you execute steps mentioned in url below:

https://docs.datafabric.hpe.com/61/Oozie/ManageOozieServicesChkStatus.html?hl=oozie%2Cadmin

2) Has the oozie been configured correctly as mentioned on below url:

https://docs.datafabric.hpe.com/61/Oozie/ConfiguringOozieonaSecureCluster.html

30 What user is being used to run the oozie job? Is it having an active ticket?

Thanks.

Re: NoSuchMethodError when trying to execute any Sqoop/Oozie command

MaksREDS — Mon, 23 Aug 2021 09:15:22 GMT

1) When I use CLI, the same error occurs, i.e.

When I use WEB console - it works ok. I can log in, see the workflow jobs etc.

2) I have checked the configuration - it is as decribed in

https://docs.datafabric.hpe.com/61/Oozie/ConfiguringOozieonaSecureCluster.html

3) I am using mapr user with a valid ticket.

Re: NoSuchMethodError when trying to execute any Sqoop/Oozie command

Harshkohli — Tue, 14 Sep 2021 14:18:21 GMT

Can you confirm if this issue is also fixed after your oozie rpm upgrade?

Re: NoSuchMethodError when trying to execute any Sqoop/Oozie command

Harshkohli — Tue, 05 Oct 2021 18:20:31 GMT

Can you confirm if this issue is also fixed after your oozie rpm upgrade?

Re: NoSuchMethodError when trying to execute any Sqoop/Oozie command

MaksREDS — Mon, 18 Oct 2021 07:23:13 GMT

The issue is fixed after oozie rpm upgrade, as described in https://community.hpe.com/t5/HPE-Ezmeral-Software-platform/Oozie-not-starting/td-p/7138408 .