HPE Ezmeral Software platform
1824000 Members
3861 Online
109667 Solutions
New Discussion юеВ

configure.sh with multiple history servers, history server fails

 
whydo
Occasional Advisor

configure.sh with multiple history servers, history server fails

Hi All,

I'm setting up a new cluster and when I run configure.sh i'm giving both history servers like below.

The problem is when i try to run.a haddop job it's giving the following error:

2023-11-17 14:08:00,130 ERROR org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer: Error starting JobHistoryServerjava.lang.IllegalArgumentException: Does not contain a valid host:port authority: host0001470.acme.com,host0001476.acme.com:10020 (configuration property mapreduce.jobhistory.address)

History server shuts down at this point.

CONFIGURE COMMAND:

/opt/mapr/server/configure.sh -secure -u mapradmin -g mapradmin -Z host0001467.acme.com:1181,host0001470.acme.com:1181,host0001473.acme.com:1181 -C host0001474.acme.com:7222,host0001475.acme.com:7222,host0001476.acme.com:7222 -H 7443 -N MAPRPOC -dare -HS host0001470.acme.com,host0001476.acme.com --isvm -D /dev/sdb,/dev/sdc -d host0001477.acme.com:3306 -dp password -du user -no-autostart

 

The resultant lines in mapred-site.xml look like this:

<property>

        <name>mapreduce.jobhistory.address</name>

        <value>host0001470.acme.com,host0001476.acme.com:10020</value>

    </property>

    <property>

        <name>mapreduce.jobhistory.webapp.address</name>

        <value>host0001470.acme.com,host0001476.acme.com:19888</value>

    </property>

    <property>

        <name>mapreduce.jobhistory.webapp.https.address</name>

        <value>host0001470.acme.com,host0001476.acme.com:19890</value>

    </property>

 

My question is should i only be providing one history server in the configure.sh command ?

How is the HA functionality handled if I only provide one history server in the configure.sh command?

Thanks!

5 REPLIES 5
Dave Olker
Neighborhood Moderator

Re: configure.sh with multiple history servers, history server fails

The configure.sh script is designed to only handle a single HS node.   Looking at the script itself:

# History Server IP
hs_ip=

...

        -HS)
            shift
            hs_ip="$1"
            ;;

 

That's why the mapred-site.xml file lists the string you passed to "-HS" and appends the port number rather than have the port number appended to each hostanme individually.  

As for how high availability is handled with the HistoryServer, I'll leave that for others with more knowledge of the history server to recommend. 



I work at HPE
HPE Support Center offers support for your HPE services and products when and how you need it. Get started with HPE Support Center today.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
Accept or Kudo
whydo
Occasional Advisor

Re: configure.sh with multiple history servers, history server fails

Hi Dave,

I modiffied the mapred-site.xml to include history server properties with a ":<port_number> after each node and still seeing the error (pasted below):

I'm wondering if this is something that is not possible?

The new configuratio in mapred-site.xml:

    <property>

        <name>mapreduce.jobhistory.address</name>

        <value>host003017.acme.com:10020,host003019.acme.com:10020</value>

    </property>

    <property>

        <name>mapreduce.jobhistory.webapp.address</name>

        <value>host003017.acme.com:19888,host003019.acme.com:19888</value>

    </property>

    <property>

        <name>mapreduce.jobhistory.webapp.https.address</name>

        <value>host003017.acme.com:19890,host003019.acme.com:19890</value>

    </property>

 

 

2023-11-22 12:12:42,956 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer: registered UNIX signal handlers for [TERM, HUP, INT]

2023-11-22 12:12:43,419 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer failed in state INITED

java.lang.IllegalArgumentException: Does not contain a valid host:port authority: host003017.acme.com:10020,host003019.acme.com:10020 (configuration property mapreduce.jobhistory.address)

hat org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:242)

hat org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:198)

hat org.apache.hadoop.yarn.conf.YarnConfiguration.getSocketAddr(YarnConfiguration.java:4779)

hat org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.getBindAddress(JobHistoryServer.java:198)

hat org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.doSecureLogin(JobHistoryServer.java:186)

hat org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceInit(JobHistoryServer.java:127)

hat org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)

hat org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.launchJobHistoryServer(JobHistoryServer.java:231)

hat org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.main(JobHistoryServer.java:241)

2023-11-22 12:12:43,435 ERROR org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer: Error starting JobHistoryServer

java.lang.IllegalArgumentException: Does not contain a valid host:port authority: host003017.acme.com:10020,host003019.acme.com:10020 (configuration property mapreduce.jobhistory.address)

hat org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:242)

hat org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:198)

hat org.apache.hadoop.yarn.conf.YarnConfiguration.getSocketAddr(YarnConfiguration.java:4779)

hat org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.getBindAddress(JobHistoryServer.java:198)

hat org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.doSecureLogin(JobHistoryServer.java:186)

hat org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceInit(JobHistoryServer.java:127)

hat org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)

hat org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.launchJobHistoryServer(JobHistoryServer.java:231)

hat org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.main(JobHistoryServer.java:241)

2023-11-22 12:12:43,437 INFO org.apache.hadoop.util.ExitUtil: Exiting with status -1: Error starting JobHistoryServer

2023-11-22 12:12:43,442 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down JobHistoryServer at host003017.acme.com/172.16.23.139

 

Thanks in advance.

Dave Olker
Neighborhood Moderator

Re: configure.sh with multiple history servers, history server fails

The point I was making is the configure.sh script was only expecting a single HS node.  Does everything work properly when you modify mapred-site.xml and only list a single HS node?



I work at HPE
HPE Support Center offers support for your HPE services and products when and how you need it. Get started with HPE Support Center today.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
Accept or Kudo
whydo
Occasional Advisor

Re: configure.sh with multiple history servers, history server fails

Hi Dave,

Single host listed for __HS_IP__ then yes, everything works.

I was just exploring the HA compatibility as this format is used in some other files.

Thanks.

Dave Olker
Neighborhood Moderator

Re: configure.sh with multiple history servers, history server fails

Cool, just making sure things work when configured with a single HS node.  I'll let others with more experience with HS weigh in on the appropriate way to offer high availability for this service.



I work at HPE
HPE Support Center offers support for your HPE services and products when and how you need it. Get started with HPE Support Center today.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
Accept or Kudo