configure.sh with multiple history servers, history server fails

whydo · ‎11-17-2023

Hi All,

I'm setting up a new cluster and when I run configure.sh i'm giving both history servers like below.

The problem is when i try to run.a haddop job it's giving the following error:

2023-11-17 14:08:00,130 ERROR org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer: Error starting JobHistoryServerjava.lang.IllegalArgumentException: Does not contain a valid host:port authority: host0001470.acme.com,host0001476.acme.com:10020 (configuration property mapreduce.jobhistory.address)

History server shuts down at this point.

CONFIGURE COMMAND:

/opt/mapr/server/configure.sh -secure -u mapradmin -g mapradmin -Z host0001467.acme.com:1181,host0001470.acme.com:1181,host0001473.acme.com:1181 -C host0001474.acme.com:7222,host0001475.acme.com:7222,host0001476.acme.com:7222 -H 7443 -N MAPRPOC -dare -HS host0001470.acme.com,host0001476.acme.com --isvm -D /dev/sdb,/dev/sdc -d host0001477.acme.com:3306 -dp password -du user -no-autostart

The resultant lines in mapred-site.xml look like this:

<name>mapreduce.jobhistory.address</name>

</property>

<name>mapreduce.jobhistory.webapp.address</name>

</property>

<name>mapreduce.jobhistory.webapp.https.address</name>

</property>

My question is should i only be providing one history server in the configure.sh command ?

How is the HA functionality handled if I only provide one history server in the configure.sh command?

Thanks!

Dave Olker · ‎11-20-2023

The configure.sh script is designed to only handle a single HS node. Looking at the script itself:

# History Server IP
hs_ip=

...

        -HS)
            shift
            hs_ip="$1"
            ;;

That's why the mapred-site.xml file lists the string you passed to "-HS" and appends the port number rather than have the port number appended to each hostanme individually.

As for how high availability is handled with the HistoryServer, I'll leave that for others with more knowledge of the history server to recommend.

I work at HPE
HPE Support Center offers support for your HPE services and products when and how you need it. Get started with HPE Support Center today.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]

whydo · ‎11-22-2023

Hi Dave,

I modiffied the mapred-site.xml to include history server properties with a ":<port_number> after each node and still seeing the error (pasted below):

I'm wondering if this is something that is not possible?

The new configuratio in mapred-site.xml:

<name>mapreduce.jobhistory.address</name>

</property>

<name>mapreduce.jobhistory.webapp.address</name>

</property>

<name>mapreduce.jobhistory.webapp.https.address</name>

</property>

2023-11-22 12:12:42,956 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer: registered UNIX signal handlers for [TERM, HUP, INT]

2023-11-22 12:12:43,419 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer failed in state INITED

java.lang.IllegalArgumentException: Does not contain a valid host:port authority: host003017.acme.com:10020,host003019.acme.com:10020 (configuration property mapreduce.jobhistory.address)

hat org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:242)

hat org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:198)

hat org.apache.hadoop.yarn.conf.YarnConfiguration.getSocketAddr(YarnConfiguration.java:4779)

hat org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.getBindAddress(JobHistoryServer.java:198)

hat org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.doSecureLogin(JobHistoryServer.java:186)

hat org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceInit(JobHistoryServer.java:127)

hat org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)

hat org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.launchJobHistoryServer(JobHistoryServer.java:231)

hat org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.main(JobHistoryServer.java:241)

2023-11-22 12:12:43,435 ERROR org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer: Error starting JobHistoryServer