[Hue] Cannot create Spark session when user is not "mapr"

[MaksREDS]

When I log in Hue as "mapr" and open pySpark editor everything works fine. I am able to run a script.
If I use other user, when I open pySpark editor, a red error pop-up message appears:

[18/Aug/2021 01:57:29 -0700] decorators   ERROR    Error running create_session
Traceback (most recent call last):
  File "/opt/mapr/hue/hue-4.6.0/desktop/libs/notebook/src/notebook/decorators.py", line 111, in wrapper
    return f(*args, **kwargs)
  File "/opt/mapr/hue/hue-4.6.0/desktop/libs/notebook/src/notebook/api.py", line 104, in create_session
    response['session'] = get_api(request, session).create_session(lang=session['type'], properties=properties)
  File "/opt/mapr/hue/hue-4.6.0/desktop/libs/notebook/src/notebook/connectors/spark_shell.py", line 229, in create_session
    raise QueryError(_('The Spark session could not be created in the cluster: %s') % info)
QueryError: The Spark session could not be created in the cluster: 	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:144)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:943)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:952)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

YARN Diagnostics: 

I found that Livy session is created and the log is as follow:

21/08/18 04:55:56 ERROR SparkSubmit$$anon$2: User abcd(user id 5003)  has been denied access to create application_1628147108709_0127
org.apache.hadoop.security.AccessControlException: User abcd(user id 5003)  has been denied access to create application_1628147108709_0127
	at com.mapr.fs.MapRFileSystem.makeDir(MapRFileSystem.java:1469)
	at com.mapr.fs.MapRFileSystem.mkdirs(MapRFileSystem.java:1488)
	at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1920)
	at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:641)
	at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:428)
	at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:865)
	at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:179)
	at org.apache.spark.deploy.yarn.Client.run(Client.scala:1135)
	at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1530)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:855)
	at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:146)
	at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:144)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:144)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:943)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:952)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

The user has permission to create file/directory in /apps/spark/ (MapR FS), so " User abcd(user id 5003) has been denied access to create application_1628147108709_0127" probably means something else.

Are there any additional user permissions for starting Spark jobs?

[Harshkohli]

Hello,

Thanks for your post. Can you confirm on below please?

1) Is this unsecure or secure cluster?

2) Are you trying this with impersonation?

[MaksREDS]

Hello, 

Thank you for asking.

1) The cluster is secure.

2)  No, I do not. As far as I know it is not possible to impersonate the mapr user (the only user that is permitted to create Spark session in my case).

[Harshkohli · Accepted solution]

Thanks for the information. I was successfuly able to reproduce your issue and the fix is below.

Create a directory lilke below /mapr/<cluster_name>/user/<customer user>/.sparkStaging  on MaprFS and provide 775 permissions as mentioned below.

hadoop dfs -chmod 775 /user/<customer user>/.sparkStaging

After the same try submitting a test job using spark submit on node and then via Hue, it should work hopefully.

/opt/mapr/spark/spark-2.4.4/bin/run-example --master yarn --deploy-mode cluster SparkPi 10

Kindly let me know if it works for you or not.

[MaksREDS]

It works, thank you.
The directory /mapr/<cluster_name>/user/<customer user>/.sparkStaging must exist before creating a SparkSession. For some Spark Jobs it is also necessary to create /mapr/<cluster_name>/user/<customer user>/.tmp/hive directory.