HPE Ezmeral Software platform

[Hue] Cannot create Spark session when user is not "mapr"

 
SOLVED
Go to solution
MaksREDS
Occasional Advisor

[Hue] Cannot create Spark session when user is not "mapr"

When I log in Hue as "mapr" and open pySpark editor everything works fine. I am able to run a script.
If I use other user, when I open pySpark editor, a red error pop-up message appears:

[18/Aug/2021 01:57:29 -0700] decorators   ERROR    Error running create_session
Traceback (most recent call last):
  File "/opt/mapr/hue/hue-4.6.0/desktop/libs/notebook/src/notebook/decorators.py", line 111, in wrapper
    return f(*args, **kwargs)
  File "/opt/mapr/hue/hue-4.6.0/desktop/libs/notebook/src/notebook/api.py", line 104, in create_session
    response['session'] = get_api(request, session).create_session(lang=session['type'], properties=properties)
  File "/opt/mapr/hue/hue-4.6.0/desktop/libs/notebook/src/notebook/connectors/spark_shell.py", line 229, in create_session
    raise QueryError(_('The Spark session could not be created in the cluster: %s') % info)
QueryError: The Spark session could not be created in the cluster: 	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:144)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:943)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:952)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

YARN Diagnostics: 

 

I found that Livy session is created and the log is as follow:

21/08/18 04:55:56 ERROR SparkSubmit$$anon$2: User abcd(user id 5003)  has been denied access to create application_1628147108709_0127
org.apache.hadoop.security.AccessControlException: User abcd(user id 5003)  has been denied access to create application_1628147108709_0127
	at com.mapr.fs.MapRFileSystem.makeDir(MapRFileSystem.java:1469)
	at com.mapr.fs.MapRFileSystem.mkdirs(MapRFileSystem.java:1488)
	at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1920)
	at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:641)
	at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:428)
	at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:865)
	at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:179)
	at org.apache.spark.deploy.yarn.Client.run(Client.scala:1135)
	at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1530)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:855)
	at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:146)
	at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:144)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1683)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:144)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:943)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:952)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

 

The user has permission to create file/directory in /apps/spark/ (MapR FS), so " User abcd(user id 5003) has been denied access to create application_1628147108709_0127" probably means something else.

Are there any additional user permissions for starting Spark jobs?

4 REPLIES 4
Harshkohli
HPE Pro

Re: [Hue] Cannot create Spark session when user is not "mapr"

Hello,

Thanks for your post. Can you confirm on below please?

1) Is this unsecure or secure cluster?

2) Are you trying this with impersonation?

 

 

I work for HPE
MaksREDS
Occasional Advisor

Re: [Hue] Cannot create Spark session when user is not "mapr"

Hello, 

Thank you for asking.

1) The cluster is secure.

2)  No, I do not. As far as I know it is not possible to impersonate the mapr user (the only user that is permitted to create Spark session in my case).

 

Harshkohli
HPE Pro
Solution

Re: [Hue] Cannot create Spark session when user is not "mapr"

Thanks for the information. I was successfuly able to reproduce your issue and the fix is below.

Create a directory lilke below /mapr/<cluster_name>/user/<customer user>/.sparkStaging  on MaprFS and provide 775 permissions as mentioned below.

hadoop dfs -chmod 775 /user/<customer user>/.sparkStaging

After the same try submitting a test job using spark submit on node and then via Hue, it should work hopefully.

/opt/mapr/spark/spark-2.4.4/bin/run-example --master yarn --deploy-mode cluster SparkPi 10

Kindly let me know if it works for you or not.

 

I work for HPE
MaksREDS
Occasional Advisor

Re: [Hue] Cannot create Spark session when user is not "mapr"

It works, thank you.
The directory /mapr/<cluster_name>/user/<customer user>/.sparkStaging must exist before creating a SparkSession. For some Spark Jobs it is also necessary to create /mapr/<cluster_name>/user/<customer user>/.tmp/hive directory.