mapreduce jobs are failing in ezmeral datafabric cluster

[HariprasathAS]

i am getting below error while running mapreduce jobs and it is failing


2024-06-27 18:17:34,726 INFO mapreduce.Job: Job job_1719491780601_0005 failed with state FAILED due to: Application application_1719491780601_0005 failed 2 times due to AM Container for appattempt_1719491780601_0005_000002 exited with exitCode: -1000
Failing this attempt.Diagnostics: [2024-06-27 18:17:33.792]Application application_1719491780601_0005 initialization failed (exitCode=24) with output:
For more detailed output, check the application tracking page: https://node1.ezmeralidc.com:8090/cluster/app/application_1719491780601_0005 Then click on links to logs of each attempt.
. Failing the application.
2024-06-27 18:17:34,741 INFO mapreduce.Job: Counters: 0

below is the log:

2024-06-27 15:20:55,954 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system is set solely by core-default.xml therefore -  ignoring
2024-06-27 15:20:55,955 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at maprfs:/var/mapr/cluster/yarn/rm/staging/mapr/.staging/job_1719334515076_0016/job_1719334515076_0016_1.jhist
2024-06-27 15:20:55,962 WARN [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Unable to parse prior job history, aborting recovery
java.io.FileNotFoundException: /var/mapr/cluster/yarn/rm/staging/mapr/.staging/job_1719334515076_0016/job_1719334515076_0016_1.jhist
  at com.mapr.fs.MapRClientImpl.open(MapRClientImpl.java:354)
  at com.mapr.fs.MapRFileSystem.open(MapRFileSystem.java:1135)
  at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:982)
  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.getPreviousJobHistoryStream(MRAppMaster.java:1430)
  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1434)
  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1336)
  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1224)
  at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1764)
  at java.base/java.security.AccessController.doPrivileged(Native Method)
  at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1988)
  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1760)
  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1694)
2024-06-27 15:20:55,972 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system is set solely by core-default.xml therefore -  ignoring
2024-06-27 15:20:55,974 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at maprfs:/var/mapr/cluster/yarn/rm/staging/mapr/.staging/job_1719334515076_0016/job_1719334515076_0016_1.jhist
2024-06-27 15:20:55,975 WARN [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Could not parse the old history file. Will not have old AMinfos 
java.io.FileNotFoundException: /var/mapr/cluster/yarn/rm/staging/mapr/.staging/job_1719334515076_0016/job_1719334515076_0016_1.jhist
  at com.mapr.fs.MapRClientImpl.open(MapRClientImpl.java:354)
  at com.mapr.fs.MapRFileSystem.open(MapRFileSystem.java:1135)
  at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:982)
  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.getPreviousJobHistoryStream(MRAppMaster.java:1430)
  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.readJustAMInfos(MRAppMaster.java:1482)
  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1344)
  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1224)
  at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1764)
  at java.base/java.security.AccessController.doPrivileged(Native Method)
  at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1988)
  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1760)
  at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1694)
2024-06-27 15:20:55,976 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Starting to clean up previous job's temporary files
2024-06-27 15:20:55,988 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Finished cleaning up previous job temporary files

please help to resolve this issue.

[support_s]

System recommended content:

1. HPE Ezmeral Data Fabric ��� Customer-Managed 7.7.0 Documentation | Hadoop 3.3.4.100 - 2301 (EEP 9.1.0) Release Notes

2. HPE Ezmeral Data Fabric ��� Customer-Managed 7.7.0 Documentation | Hadoop 2.7.6.300 - 2305 (EEP 8.1.1) Release Notes

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[HariprasathAS]

i am still facing this issue. it does not help me to resolve this issue.

[Najmuddin_c]

The errors you pasted are not relevant to the job failure, could you please share NM logs where AM failed to start? Also you'd find useful information in userlogs ('/opt/mapr/hadoop/hadoop-<version>/logs/userlogs/<application_id>/<attempt_id>/' , if centralized logging is enabled, you'd find it under yarn logs location). 

[support_s]

Hello,

Let us know if you were able to resolve the issue.

If you have no further query, and you are satisfied with the answer then kindly mark the topic as Solved so that it is helpful for all community members.

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.