Query: HPE Ezmeral "internal error" on Kubernetes node

[Andrii_D]

Hello, team.

I'm QA engineer and I'm working on installation of HPE Ezmeral 5.6. 

We are facing with weird issue on Kubernetes node UI and can't define it.

Setup log writes in the end "Successfully configured a HPE CP kubernetes node.", but UI has red mark with "Inernar error" sign.

0 failed results after precheck script in logs at /tmp folder.

Could you help us with it?

[Andrii_D]

There is a screenshot of our UI with error:
https://app.gemoo.com/share/image-annotation/621781284372832256?codeId=M0BwgGobdla7q&origin=screenshotuploader&card=621781282657325056

[support_s]

System recommended content:

1. HPE Ezmeral Runtime Enterprise 5.4 Documentation | Issues and Workarounds

2. HPE Ezmeral Runtime Enterprise 5.5 Documentation | Issues and Workarounds

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[Andrii_D]

Hello.

Thanks for the links, but the solution from it didn't resolve my issue.

I configured CentOS to sync with NTP Time Servers but still have an "internal error" with red flag on Kubernetes.

What I performed on every node (Controller, Kubernetes and Gateway) is below:

https://thebackroomtech.com/2019/01/17/configure-centos-to-sync-with-ntp-time-servers/

[Shishir_Prakash]

From the error it looks like disk missing by-path.  Typically this might be related to udev not working properly or disk labels changed.

Try udevadm trigger command and see if that helps to resolve the issue.  You probably needs to explore that from linux side. 

What is the OS version? and output of below command ?

ls -ltr /dev/disk/by-path/pci-0000\:00\:0* 

[Andrii_D]

@Shishir_Prakash 

Hello, Shishir.

There are my results:

ls -ltr /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:5:0

lrwxrwxrwx. 1 root root 9 Feb 29 16:46 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:5:0 -> ../../sdf

And version is below:

hostnamectl
Chassis: vm
Virtualization: vmware
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-1160.108.1.el7.x86_64
Architecture: x86-64

BTW,  the command "udevadm trigger" did not help.

[Andrii_D]

@Shishir_Prakash 

Coould you check my output and help with further actions?

[Shishir_Prakash]

Can you please share the /var/log/bluedata/bds-mgmt.log from primary controller  node and database dump file /tmp/ere-database.txt which can be generated using below command.

/opt/bluedata/common-install/bd_mgmt/bin/bd_mgmt dump_db /tmp/ere-database.txt

[Andrii_D]

Hello, @Shishir_Prakash 

I got this files, but they contains sensitive information (passwords, keys etc.).

I deleted all this information and share in such form.

Original file I can share with you via corporate email.

Is it ok for you?
https://drive.google.com/file/d/1HG1IPavHDJQIWSbv_IqUZ73cOH2HXYQd/view?usp=sharing

[Abe-S]

Hi Andrii,

I just requested access to the google drive doc from my gmail account - please can you share it?

thanks

Abe

[GFustagni]

please try ti remove the virtual k8s node, re-image the OS and try to add it again

[GFustagni]

please try ti remove the virtual k8s node, re-image the OS and try to add it again