[Hue] Cannot create Spark session when user is not "mapr"

MaksREDS — Thu, 19 Aug 2021 09:28:21 GMT

When I log in Hue as "mapr" and open pySpark editor everything works fine. I am able to run a script.
If I use other user, when I open pySpark editor, a red error pop-up message appears:

[18/Aug/2021 01:57:29 -0700] decorators ERROR Error running create_session Traceback (most recent call last): File "/opt/mapr/hue/hue-4.6.0/desktop/libs/notebook/src/notebook/decorators.py", line 111, in wrapper return f(*args, **kwargs) File "/opt/mapr/hue/hue-4.6.0/desktop/libs/notebook/src/notebook/api.py", line 104, in create_session response['session'] = get_api(request, session).create_session(lang=session['type'], properties=properties) File "/opt/mapr/hue/hue-4.6.0/desktop/libs/notebook/src/notebook/connectors/spark_shell.py", line 229, in create_session raise QueryError(_('The Spark session could not be created in the cluster: %s') % info) QueryError: The Spark session could not be created in the cluster: at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.security.auth.Subject.doAs(Subject.java:423) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:144) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:943) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:952) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) YARN Diagnostics:

I found that Livy session is created and the log is as follow:

21/08/18 04:55:56 ERROR SparkSubmit$$anon$2: User abcd(user id 5003) has been denied access to create application_1628147108709_0127 org.apache.hadoop.security.AccessControlException: User abcd(user id 5003) has been denied access to create application_1628147108709_0127 at com.mapr.fs.MapRFileSystem.makeDir(MapRFileSystem.java:1469) at com.mapr.fs.MapRFileSystem.mkdirs(MapRFileSystem.java:1488) at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1920) at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:641) at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:428) at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:865) at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:179) at org.apache.spark.deploy.yarn.Client.run(Client.scala:1135) at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1530) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:855) at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:146) at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:144) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.security.auth.Subject.doAs(Subject.java:423) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:144) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:943) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:952) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

The user has permission to create file/directory in /apps/spark/ (MapR FS), so " User abcd(user id 5003) has been denied access to create application_1628147108709_0127" probably means something else.

Are there any additional user permissions for starting Spark jobs?

Re: [Hue] Cannot create Spark session when user is not "mapr"

Harshkohli — Thu, 19 Aug 2021 02:16:46 GMT

Hello,

Thanks for your post. Can you confirm on below please?

1) Is this unsecure or secure cluster?

2) Are you trying this with impersonation?

Re: [Hue] Cannot create Spark session when user is not "mapr"

MaksREDS — Thu, 19 Aug 2021 07:19:37 GMT

Hello,

Thank you for asking.

1) The cluster is secure.

2) No, I do not. As far as I know it is not possible to impersonate the mapr user (the only user that is permitted to create Spark session in my case).

Re: [Hue] Cannot create Spark session when user is not "mapr"

Harshkohli — Sat, 21 Aug 2021 01:33:11 GMT

Thanks for the information. I was successfuly able to reproduce your issue and the fix is below.

Create a directory lilke below /mapr/<cluster_name>/user/<customer user>/.sparkStaging on MaprFS and provide 775 permissions as mentioned below.

hadoop dfs -chmod 775 /user/<customer user>/.sparkStaging

After the same try submitting a test job using spark submit on node and then via Hue, it should work hopefully.

/opt/mapr/spark/spark-2.4.4/bin/run-example --master yarn --deploy-mode cluster SparkPi 10

Kindly let me know if it works for you or not.

Re: [Hue] Cannot create Spark session when user is not "mapr"

MaksREDS — Mon, 23 Aug 2021 09:00:27 GMT

It works, thank you.
The directory /mapr/<cluster_name>/user/<customer user>/.sparkStaging must exist before creating a SparkSession. For some Spark Jobs it is also necessary to create /mapr/<cluster_name>/user/<customer user>/.tmp/hive directory.

topic [Hue] Cannot create Spark session when user is not "mapr" in HPE Ezmeral Software platform

[Hue] Cannot create Spark session when user is not "mapr"

Re: [Hue] Cannot create Spark session when user is not "mapr"

Re: [Hue] Cannot create Spark session when user is not "mapr"

Re: [Hue] Cannot create Spark session when user is not "mapr"

Re: [Hue] Cannot create Spark session when user is not "mapr"