NoSuchMethodError when trying to execute any Sqoop/Oozie command

[MaksREDS]

When I try to execute any command in Sqoop 1 Editor (via HUE) the following error occurs:

2021-08-05 09:40:06,417 ERROR ActionStartXCommand:517 - SERVER[node3.address.domain] USER[mapr] GROUP[-] TOKEN[] APP[Batch job for query-sqoop1] JOB[0000019-210805045515667-oozie-mapr-W] ACTION[0000019-210805045515667-oozie-mapr-W@sqoop-0d6e] Error,
java.lang.NoSuchMethodError: 'void com.google.common.base.Preconditions.checkArgument(boolean, java.lang.String, java.lang.Object)'
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientConnectionRetry.<init>(TimelineClientImpl.java:158)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.serviceInit(TimelineClientImpl.java:284)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:180)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.oozie.service.HadoopAccessorService$4.run(HadoopAccessorService.java:572)
at org.apache.oozie.service.HadoopAccessorService$4.run(HadoopAccessorService.java:568)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683)
at org.apache.oozie.service.HadoopAccessorService.createYarnClient(HadoopAccessorService.java:568)
at org.apache.oozie.action.hadoop.JavaActionExecutor.createYarnClient(JavaActionExecutor.java:1648)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1047)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1592)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:243)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:68)
at org.apache.oozie.command.XCommand.call(XCommand.java:291)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:363)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:292)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:210)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

Error looks similiar to: https://support.datafabric.hpe.com/s/article/Spark-application-using-HBase-API-fails-with-java-lang-NoSuchMethodError?language=en_US
The currently installed mapr-hbase version is 1.4.13.0.202105170751, so it is newer than specified in above issue.

Output of  rpm -qa|grep mapr

mapr-zookeeper-6.2.0.0.20200915234957.GA-1.x86_64

mapr-hive-2.3.202105190211-1.noarch

mapr-opentsdb-2.4.0.202105181725-1.noarch
mapr-hbase-rest-1.4.13.0.202105170751-1.noarch
mapr-historyserver-2.7.5.0.202105131403-1.x86_64
mapr-collectd-5.10.0.0.202104301721-1.x86_64
mapr-librdkafka-0.11.3.202104281722-1.x86_64
mapr-core-6.2.0.0.20200915234957.GA-1.x86_64
mapr-kafka-2.1.1.200.202105140421-1.noarch
mapr-spark-historyserver-2.4.7.100.202105170615-1.noarch
mapr-hivewebhcat-2.3.202105190211-1.noarch
mapr-kafka-rest-5.1.2.200.202105071030-1.noarch
mapr-oozie-5.1.0.202.202105260138-1.noarch
mapr-mastgateway-6.2.0.0.20200915234957.GA-1.x86_64
mapr-hadoop-client-2.7.5.0.202105131403-1.x86_64
mapr-drill-internal-1.16.1.200.202105180709-1.noarch
mapr-livy-0.7.0.0.202105120521-1.noarch
mapr-tez-0.9.202105070958-1.noarch
mapr-kafka-connect-jdbc-5.1.2.0.202009141042-1.noarch
mapr-oozie-internal-5.1.0.202.202105260138-1.noarch
mapr-schema-registry-internal-5.1.2.200.202105130812-1.noarch
mapr-ksql-5.1.2.200.202105140827-1.noarch
mapr-hivemetastore-2.3.202105190211-1.noarch
mapr-sqoop-1.4.7.202105070954-1.noarch
mapr-client-6.2.0.0.20200915234957.GA-1.x86_64
mapr-core-internal-6.2.0.0.20200915234957.GA-1.x86_64
mapr-ksql-internal-5.1.2.200.202105140827-1.noarch
mapr-schema-registry-5.1.2.200.202105130812-1.noarch
mapr-hiveserver2-2.3.202105190211-1.noarch
mapr-kafka-connect-hdfs-5.1.2.200.202105170800-1.noarch
mapr-loopbacknfs-6.2.0.0.20200915234957.GA-1.x86_64
mapr-hbase-1.4.13.0.202105170751-1.noarch
mapr-drill-1.16.1.200.202105180709-1.noarch
mapr-hbase-regionserver-1.4.13.0.202105170751-1.noarch
mapr-timelineserver-2.7.5.0.202105131403-1.x86_64
mapr-pig-0.17.0.0.202009090610-1.noarch
mapr-hadoop-util-2.7.5.0.202105131403-1.x86_64
mapr-patch-6.2.0.9.20210615161337.GA-1.x86_64
mapr-spark-2.4.7.100.202105170615-1.noarch
mapr-hbasethrift-1.4.13.0.202105170751-1.noarch
mapr-nodemanager-2.7.5.0.202105131403-1.x86_64
mapr-httpfs-1.1.0.0.202105130703-1.noarch
mapr-zk-internal-6.2.0.0.20200915234957.GA-1.x86_64
mapr-fileserver-6.2.0.0.20200915234957.GA-1.x86_64
mapr-hadoop-core-2.7.5.0.202105131403-1.x86_64
mapr-asynchbase-1.8.2.202104220851-1.noarch
mapr-spark-thriftserver-2.4.7.100.202105170615-1.noarch
mapr-flume-1.9.0.100.202101270131-1.noarch
mapr-hue-4.6.0.0.202105070812-1.noarch

I read some solutions of similar issues with hadoop ecosystem - to update outdated version of Guava java library, e.g. https://issues.apache.org/jira/browse/HIVE-22915 .

I replaced all guava-11*.jar I had found with guava-28*.jar, but the error still occurs. 

[Harshkohli]

Thanks for your post. Few questions:

1) What happens when you try to execute sqoop/oozie job from terminal instead of Hue?

2) Can you check the lib path for sqoop, oozie to confirm the version of guava jar it contains.

My idea will be to first isolate the problem and check if it is actually hue/oozie/sqoop issue so that we can focus on the same.

You can also try to do below on guava jar to check its contents and confirm if the version being used has the method or not.

jar tvf <guava jar>

[MaksREDS]

Thank you for your advices.

1) Sqoop jobs works when executed from CLI (command "sqoop") - with no errors.

2) guava-28.2-jre.jar contains the Preconditions class (with checkArgument method), so it is probably not the reason.
Maybe the reason is the com.google.common.base.* class, which could be a part of other java lib used by oozie?

Regards.

[Harshkohli]

As I understand you were getting this error when submitting job via Hue.

Can you try to submit the job via Oozie command line?

[MaksREDS]

Thank you for your advices.
I have been trying to solve this issue in many ways, but with no success.
When I try to run a workflow from Oozie CLI, the following error occurs:

java.lang.IllegalAccessError: class com.mapr.fs.proto.Security$AuthenticationReqFull tried to access private field com.google.protobuf.AbstractMessage.memoizedSize (com.mapr.fs.proto.Security$AuthenticationReqFull and com.google.protobuf.AbstractMessage are in unnamed module of loader 'app')
	at com.mapr.fs.proto.Security$AuthenticationReqFull.getSerializedSize(Security.java:12103)
	at com.google.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:62)
	at com.mapr.security.maprauth.MaprAuthenticator.doMaprHandshake(MaprAuthenticator.java:162)
	at com.mapr.security.maprauth.MaprAuthenticator.authenticate(MaprAuthenticator.java:92)
	at org.apache.oozie.client.AuthOozieClient.createConnection(AuthOozieClient.java:173)
	at org.apache.oozie.client.OozieClient$1.doExecute(OozieClient.java:520)
	at org.apache.oozie.client.retry.ConnectionRetriableClient.execute(ConnectionRetriableClient.java:44)
	at org.apache.oozie.client.OozieClient.createRetryableConnection(OozieClient.java:522)
	at org.apache.oozie.client.OozieClient.getSupportedProtocolVersions(OozieClient.java:397)
	at org.apache.oozie.client.OozieClient.validateWSVersion(OozieClient.java:357)
	at org.apache.oozie.client.OozieClient.createURL(OozieClient.java:473)
	at org.apache.oozie.client.OozieClient.access$000(OozieClient.java:88)
	at org.apache.oozie.client.OozieClient$ClientCallable.call(OozieClient.java:567)
	at org.apache.oozie.client.OozieClient.run(OozieClient.java:887)
	at org.apache.oozie.cli.OozieCLI.jobCommand(OozieCLI.java:1095)
	at org.apache.oozie.cli.OozieCLI.processCommand(OozieCLI.java:719)
	at org.apache.oozie.cli.OozieCLI.run(OozieCLI.java:674)
	at org.apache.oozie.cli.OozieCLI.main(OozieCLI.java:239)
Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = Could not authenticate, Exception while getting ticket and key

The CLI command is:

oozie job -oozie="https://oozie_node_hostname:11443/oozie" -config /tmp/job/job.properties -run

I use the same job/workflow files as automatically generated by Hue.

[Harshkohli]

Thanks for the update.

1) What do you get when you execute steps mentioned in url below:

https://docs.datafabric.hpe.com/61/Oozie/ManageOozieServicesChkStatus.html?hl=oozie%2Cadmin

2) Has the oozie been configured correctly as mentioned on below url:

https://docs.datafabric.hpe.com/61/Oozie/ConfiguringOozieonaSecureCluster.html

30 What user is being used to run the oozie job? Is it having an active ticket?

Thanks.

[MaksREDS]

1) When I use CLI, the same error occurs, i.e.

java.lang.IllegalAccessError: class com.mapr.fs.proto.Security$AuthenticationReqFull tried to access private field com.google.protobuf.AbstractMessage.memoizedSize (com.mapr.fs.proto.Security$AuthenticationReqFull and com.google.protobuf.AbstractMessage are in unnamed module of loader 'app')
	at com.mapr.fs.proto.Security$AuthenticationReqFull.getSerializedSize(Security.java:12103)
	at com.google.protobuf.AbstractMessageLite.toByteArray(AbstractMessageLite.java:62)
	at com.mapr.security.maprauth.MaprAuthenticator.doMaprHandshake(MaprAuthenticator.java:162)
	at com.mapr.security.maprauth.MaprAuthenticator.authenticate(MaprAuthenticator.java:92)
	at org.apache.oozie.client.AuthOozieClient.createConnection(AuthOozieClient.java:173)
	at org.apache.oozie.client.OozieClient$1.doExecute(OozieClient.java:520)
	at org.apache.oozie.client.retry.ConnectionRetriableClient.execute(ConnectionRetriableClient.java:44)
	at org.apache.oozie.client.OozieClient.createRetryableConnection(OozieClient.java:522)
	at org.apache.oozie.client.OozieClient.getSupportedProtocolVersions(OozieClient.java:397)
	at org.apache.oozie.client.OozieClient.validateWSVersion(OozieClient.java:357)
	at org.apache.oozie.client.OozieClient.createURL(OozieClient.java:473)
	at org.apache.oozie.client.OozieClient.access$000(OozieClient.java:88)
	at org.apache.oozie.client.OozieClient$ClientCallable.call(OozieClient.java:567)
	at org.apache.oozie.client.OozieClient.run(OozieClient.java:887)
	at org.apache.oozie.cli.OozieCLI.jobCommand(OozieCLI.java:1095)
	at org.apache.oozie.cli.OozieCLI.processCommand(OozieCLI.java:719)
	at org.apache.oozie.cli.OozieCLI.run(OozieCLI.java:674)
	at org.apache.oozie.cli.OozieCLI.main(OozieCLI.java:239)
Error: IO_ERROR : java.io.IOException: Error while connecting Oozie server. No of retries = 1. Exception = Could not authenticate, Exception while getting ticket and key

When I use WEB console - it works ok. I can log in, see the workflow jobs etc.

2) I have checked the configuration - it is as decribed in

https://docs.datafabric.hpe.com/61/Oozie/ConfiguringOozieonaSecureCluster.html

3) I am using mapr user with a valid ticket.

[Harshkohli]

Can you confirm if this issue is also fixed after your oozie rpm upgrade?

[Harshkohli · Accepted solution]

Can you confirm if this issue is also fixed after your oozie rpm upgrade?

[MaksREDS]

The issue is fixed after oozie rpm upgrade, as described in https://community.hpe.com/t5/HPE-Ezmeral-Software-platform/Oozie-not-starting/td-p/7138408 .