Ezmeral Datafabric 7.3 installation issue

[HariprasathAS]

I am trying to setup Ezmeral Datafabric 7.3 with 1 master and 6 datanodes. Installation is getting failed at 81 percentage in one node with below error. Followed by this node, installation in other nodes are also getting failed. 

"mapr_logs": "2024-04-23 16:22:18 IST INFO svcs_intersection: ['mapr-gateway-7.3.0', 'mapr-webserver-7.3.0', 'mapr-spark-master-3.3.2', 'mapr-cldb-7.3.0', 'mapr-grafana-7.5.10', 'mapr-apiserver-7.3.0', 'mapr-zookeeper-7.3.0', 'mapr-resourcemanager-3.3.4']\
	2024-04-23 16:22:18 IST INFO all_svcs_set: {'mapr-kafka-connect-jdbc-6.0.0', 'mapr-kafka-connect-hdfs-6.0.0', 'mapr-kafka-schema-registry-6.0.0', 'mapr-webserver-7.3.0', 'mapr-spark-thriftserver-3.3.2', 'mapr-mysql', 'mapr-grafana-7.5.10', 'mapr-librdkafka-2.6.1', 'mapr-hive-client-3.1.3', 'mapr-zookeeper-7.3.0', 'mapr-s3server-7.3.0', 'mapr-resourcemanager-3.3.4', 'mapr-historyserver-3.3.4', 'mapr-spark-historyserver-3.3.2', 'mapr-collectd-5.12.0', 'mapr-hbase-1.4.14', 'mapr-spark-slave-3.3.2', 'mapr-hivewebhcat-3.1.3', 'mapr-kafka-2.6.1', 'mapr-mastgateway-7.3.0', 'mapr-kafka-rest-6.0.0', 'mapr-gateway-7.3.0', 'mapr-asynchbase-1.8.2', 'mapr-opentsdb-2.4.1', 'mapr-fileserver-7.3.0', 'mapr-core-7.3.0', 'mapr-spark-master-3.3.2', 'mapr-cldb-7.3.0', 'mapr-nodemanager-3.3.4', 'mapr-spark-client-3.3.2', 'mapr-nifi-1.19.1', 'mapr-apiserver-7.3.0', 'mapr-hivemetastore-3.1.3', 'mapr-hiveserver2-3.1.3'}\
	2024-04-23 16:22:18 IST INFO host_svcs_set: {'mapr-kafka-connect-jdbc-6.0.0', 'mapr-kafka-connect-hdfs-6.0.0', 'mapr-kafka-schema-registry-6.0.0', 'mapr-spark-thriftserver-3.3.2', 'mapr-mysql', 'mapr-librdkafka-2.6.1', 'mapr-hive-client-3.1.3', 'mapr-historyserver-3.3.4', 'mapr-s3server-7.3.0', 'mapr-spark-historyserver-3.3.2', 'mapr-collectd-5.12.0', 'mapr-hbase-1.4.14', 'mapr-spark-slave-3.3.2', 'mapr-hivewebhcat-3.1.3', 'mapr-mastgateway-7.3.0', 'mapr-kafka-2.6.1', 'mapr-kafka-rest-6.0.0', 'mapr-asynchbase-1.8.2', 'mapr-opentsdb-2.4.1', 'mapr-fileserver-7.3.0', 'mapr-core-7.3.0', 'mapr-nodemanager-3.3.4', 'mapr-spark-client-3.3.2', 'mapr-nifi-1.19.1', 'mapr-hivemetastore-3.1.3', 'mapr-hiveserver2-3.1.3'}\
	2024-04-23 16:22:18 IST WARN These services ['mapr-gateway-7.3.0', 'mapr-webserver-7.3.0', 'mapr-spark-master-3.3.2', 'mapr-cldb-7.3.0', 'mapr-grafana-7.5.10', 'mapr-apiserver-7.3.0', 'mapr-zookeeper-7.3.0', 'mapr-resourcemanager-3.3.4'] are inconsistent between /hosts and /config\
	2024-04-23 16:22:18 IST DEBUG **logline hidden due to sensitive data**\
	2024-04-23 16:22:31 IST DEBUG Command: 'timeout -s HUP 2m hadoop fs -mkdir -p /installer/hive-3.1.3/', Status: '0', Result: '2024-04-23 16:22:21,4493 :1831 peerid 1a0fcfd8d563a300 in binding 7fadadff7270, conn 7fadadff7350, ip 192.168.122.1:5692 doesn't match peerid 3ceb904810f4f0c0 in rpc hdr from 192.168.122.1:5692
	2024-04-23 16:22:22,4922 :1831 peerid 1a0fcfd8d563a300 in binding 7fadadff7270, conn 7fadadff7350, ip 192.168.122.1:5692 doesn't match peerid 3ceb904810f4f0c0 in rpc hdr from 192.168.122.1:5692
	2024-04-23 16:22:24,6636 :1831 peerid 1a0fcfd8d563a300 in binding 7fadadff7270, conn 7fadadff7350, ip 192.168.122.1:5692 doesn't match peerid 3ceb904810f4f0c0 in rpc hdr from 192.168.122.1:5692
	2024-04-23 16:22:27,6708 :1831 peerid 1a0fcfd8d563a300 in binding 7fadadff7270, conn 7fadadff7350, ip 192.168.122.1:5692 doesn't match peerid 3ceb904810f4f0c0 in rpc hdr from 192.168.122.1:5692
	2024-04-23 16:22:31,7502 :1831 peerid 1a0fcfd8d563a300 in binding 7fadadff7270, conn 7fadadff7350, ip 192.168.122.1:5692 doesn't match peerid 3ceb904810f4f0c0 in rpc hdr from 192.168.122.1:5692'\
	2024-04-23 16:22:31 IST INFO Putting /opt/mapr/hive/hive-3.1.3/lib/hive-beeline-3.1.3.200-eep-911.jar via hadoop fs -put to hive-3.1.3/\
	2024-04-23 16:22:31 IST DEBUG **logline hidden due to sensitive data**\
	2024-04-23 16:25:32 IST WARN Command 'b'timeout -s HUP 2m hadoop fs -put -f /opt/mapr/hive/hive-3.1.3/lib/hive-beeline-3.1.3.200-eep-911.jar /installer/hive-3.1.3/'' timed out\
	2024-04-23 16:25:32 IST ERROR Command: 'timeout -s HUP 2m hadoop fs -put -f /opt/mapr/hive/hive-3.1.3/lib/hive-beeline-3.1.3.200-eep-911.jar /installer/hive-3.1.3/', Status: '124', Result: '2024-04-23 16:22:34,4604 :1831 peerid 1a0fcfd8d563a300 in binding 7f227dc864d0, conn 7f227dc865b0, ip 192.168.122.1:5692 doesn't match peerid 3ceb904810f4f0c0 in rpc hdr from 192.168.122.1:5692
	2024-04-23 16:22:35,4691 :1831 peerid 1a0fcfd8d563a300 in binding 7f227dc864d0, conn 7f227dc865b0, ip 192.168.122.1:5692 doesn't match peerid 3ceb904810f4f0c0 in rpc hdr from 192.168.122.1:5692
	2024-04-23 16:22:37,4750 :1831 peerid 1a0fcfd8d563a300 in binding 7f227dc864d0, conn 7f227dc865b0, ip 192.168.122.1:5692 doesn't match peerid 3ceb904810f4f0c0 in rpc hdr from 192.168.122.1:5692
	2024-04-23 16:22:40,4794 :1831 peerid 1a0fcfd8d563a300 in binding 7f227dc864d0, conn 7f227dc865b0, ip 192.168.122.1:5692 doesn't match peerid 3ceb904810f4f0c0 in rpc hdr from 192.168.122.1:5692
	2024-04-23 16:22:44,4848 :1831 peerid 1a0fcfd8d563a300 in binding 7f227dc864d0, conn 7f227dc865b0, ip 192.168.122.1:5692 doesn't match peerid 3ceb904810f4f0c0 in rpc hdr from 192.168.122.1:5692
	2024-04-23 16:22:49,4910 :1831 peerid 1a0fcfd8d563a300 in binding 7f227dc864d0, conn 7f227dc865b0, ip 192.168.122.1:5692 doesn't match peerid 3ceb904810f4f0c0 in rpc hdr from 192.168.122.1:5692
	2024-04-23 16:22:55,4951 :1831 peerid 1a0fcfd8d563a300 in binding 7f227dc864d0, conn 7f227dc865b0, ip 192.168.122.1:5692 doesn't match peerid 3ceb904810f4f0c0 in rpc hdr from 192.168.122.1:5692
	2024-04-23 16:23:02,5055 :1831 peerid 1a0fcfd8d563a300 in binding 7f227dc864d0, conn 7f227dc865b0, ip 192.168.122.1:5692 doesn't match peerid 3ceb904810f4f0c0 in rpc hdr from 192.168.122.1:5692
	2024-04-23 16:23:10,5087 :1831 peerid 1a0fcfd8d563a300 in binding 7f227dc864d0, conn 7f227dc865b0, ip 192.168.122.1:5692 doesn't match peerid 3ceb904810f4f0c0 in rpc hdr from 192.168.122.1:5692
	2024-04-23 16:23:19,5881 :1831 peerid 1a0fcfd8d563a300 in binding 7f227dc864d0, conn 7f227dc865b0, ip 192.168.122.1:5692 doesn't match peerid 3ceb904810f4f0c0 in rpc hdr from 192.168.122.1:5692
	2024-04-23 16:23:28,7146 :1831 peerid 1a0fcfd8d563a300 in binding 7f227dc864d0, conn 7f227dc865b0, ip 192.168.122.1:5692 doesn't match peerid 3ceb904810f4f0c0 in rpc hdr from 192.168.122.1:5692
	2024-04-23 16:22:34,4606 ERROR Client fc/client.cc:12922 Thread: 15112 rpc err Connection reset by peer(104) 28.21 to 172.16.77.112:5692, fid 2049.16.2, upd 0
	2024-04-23 16:23:37,9038 :1831 peerid 1a0fcfd8d563a300 in binding 7f227dc864d0, conn 7f227dc865b0, ip 192.168.122.1:5692 doesn't match peerid 3ceb904810f4f0c0 in rpc hdr from 192.168.122.1:5692
	2024-04-23 16:23:46,9137 :1831 peerid 1a0fcfd8d563a300 in binding 7f227dc864d0, conn 7f227dc865b0, ip 192.168.122.1:5692 doesn't match peerid 3ceb904810f4f0c0 in rpc hdr from 192.168.122.1:5692
	2024-04-23 16:23:57,6428 : 492 Failed to connect to 192.168.122.1:5660 error: 99 sock: 368
	2024-04-23 16:23:55,9476 ERROR Client fc/client.cc:12922 Thread: 15112 rpc err Connection reset by peer(104) 28.1 to 172.16.77.112:5692, fid 2049.56.131270, upd 0
	2024-04-23 16:23:57,3595 ERROR Client fc/client.cc:12922 Thread: 15221 rpc err No such device(19) 28.123 to 172.16.77.34:5660, fid 756250.32.131304, upd 1
	2024-04-23 16:23:57,6423 ERROR Client fc/client.cc:12027 Thread: 15221 <DoWriteRPC: write resp for file hive-beeline-3.1.3.200-eep-911.jar._COPYING_, error 19, fid 756250.32.131304, primary fid 2049.76.524912, node 172.16.77.34:5660, off 65536,, count 119224 rdma 0
	2024-04-23 16:25:01,890 WARN util.ShutdownHookManager: ShutdownHook 'ClientFinalizer' timeout, java.util.concurrent.TimeoutException
	java.util.concurrent.TimeoutException
	\tat java.base/java.util.concurrent.FutureTask.get(FutureTask.java:204)
	\tat org.apache.hadoop.util.ShutdownHookManager.executeShutdown(ShutdownHookManager.java:124)
	\tat org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:95)
	2024-04-23 16:25:31,941 ERROR util.ShutdownHookManager: ShutdownHookManager shutdown forcefully after 30 seconds.'\
	2024-04-23 16:25:32 IST WARN Command timed out : timeout -s HUP 2m hadoop fs -put -f /opt/mapr/hive/hive-3.1.3/lib/hive-beeline-3.1.3.200-eep-911.jar /installer/hive-3.1.3/"
}
Unable to copy /opt/mapr/hive/hive-3.1.3/lib/hive*.jar to hive-3.1.3/ 'in <string>' requires string as left operand, not bytes, stack: Traceback (most recent call last):
  File "/tmp/ansible_mapr_conf.py_payload_y2h_rir4/ansible_mapr_conf.py_payload.zip/ansible/modules/mapr_conf/py.py", line 231, in run
  File "/tmp/ansible_mapr_conf.py_payload_y2h_rir4/ansible_mapr_conf.py_payload.zip/ansible/modules/mapr_conf/py.py", line 152, in copy_from_local
  File "/tmp/ansible_mapr_conf.py_payload_y2h_rir4/ansible_mapr_conf.py_payload.zip/ansible/modules/mapr_conf/py.py", line 169, in cmd_retry
TypeError: 'in <string>' requires string as left operand, not bytes

[support_s]

System recommended content:

1. HPE Ezmeral Data Fabric – Customer-Managed 7.3 Documentation | Known Issues (Release 7.3.0)

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[raymondyan]

From the log you provided, it looks like the initialization of the Ezmeral Data Fabric (MapR) File System didn't succeed.

So when the installation script failed when it tried to put Hive related libraries(JAR files) into the File System.

I would suggest you to install the cluster with only the File System, Object Store components first. And once the cluster is initially set up, then incrementally add your preferred components, e.g. Hive, Spark, etc.

That way the installation script is more stable to work.

[HariprasathAS]

I am installing EDF 7.3 using the installer. It automatically installs the Filesystem as well as the other components such Hive, spark etc. Can you please help how to install only the File System, Object Store components first?

[raymondyan]

Look for the screen like the attached screenshot.

This is a configurable screen to choose componets to be installed. There are several pre-set component groups, you can also extend the selection to manually tick/untick independent components.

Before the installation starts, there is a screen you can verify all components you are going to install. I'm sure you wouldn't miss it. 

Select Components to Install - EDF Installer

[HariprasathAS]

I have also tried the incremenetal install as you mentioned by selecting only Hive service. It is getting failed at 4th node first and then it is failing in all the nodes. Here i used 1 master and 6 datanodes.

if i use 1 master and 2 datanode, doing the same installation is getting success.