topic Re: Query: HPE Ezmeral "internal error" on Kubernetes node in HPE Ezmeral Software platform

HPE Ezmeral "internal error" on Kubernetes node

Andrii_D — Fri, 01 Mar 2024 03:49:24 GMT

Hello, team.

I'm QA engineer and I'm working on installation of HPE Ezmeral 5.6.

We are facing with weird issue on Kubernetes node UI and can't define it.

Setup log writes in the end "Successfully configured a HPE CP kubernetes node.", but UI has red mark with "Inernar error" sign.

0 failed results after precheck script in logs at /tmp folder.

Could you help us with it?

Re: HPE Ezmeral "internal error" on Kubernetes node

Andrii_D — Thu, 29 Feb 2024 11:00:41 GMT

There is a screenshot of our UI with error:
https://app.gemoo.com/share/image-annotation/621781284372832256?codeId=M0BwgGobdla7q&origin=screenshotuploader&card=621781282657325056

Query: HPE Ezmeral "internal error" on Kubernetes node

support_s — Thu, 29 Feb 2024 11:56:51 GMT

System recommended content:

1. HPE Ezmeral Runtime Enterprise 5.4 Documentation | Issues and Workarounds

2. HPE Ezmeral Runtime Enterprise 5.5 Documentation | Issues and Workarounds

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

Re: Query: HPE Ezmeral "internal error" on Kubernetes node

Andrii_D — Thu, 29 Feb 2024 15:32:11 GMT

Hello.

Thanks for the links, but the solution from it didn't resolve my issue.

I configured CentOS to sync with NTP Time Servers but still have an "internal error" with red flag on Kubernetes.

What I performed on every node (Controller, Kubernetes and Gateway) is below:

https://thebackroomtech.com/2019/01/17/configure-centos-to-sync-with-ntp-time-servers/

Re: Query: HPE Ezmeral "internal error" on Kubernetes node

Shishir_Prakash — Thu, 29 Feb 2024 18:47:39 GMT

From the error it looks like disk missing by-path. Typically this might be related to udev not working properly or disk labels changed.

Try udevadm trigger command and see if that helps to resolve the issue. You probably needs to explore that from linux side.

What is the OS version? and output of below command ?

ls -ltr /dev/disk/by-path/pci-0000\:00\:0*

Re: Query: HPE Ezmeral "internal error" on Kubernetes node

Andrii_D — Tue, 05 Mar 2024 06:53:34 GMT

@Shishir_Prakash

Hello, Shishir.

There are my results:

ls -ltr /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:5:0

lrwxrwxrwx. 1 root root 9 Feb 29 16:46 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:5:0 -> ../../sdf

And version is below:

hostnamectl
Chassis: vm
Virtualization: vmware
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-1160.108.1.el7.x86_64
Architecture: x86-64

BTW, the command "udevadm trigger" did not help.

Re: Query: HPE Ezmeral "internal error" on Kubernetes node

Andrii_D — Tue, 05 Mar 2024 13:42:40 GMT

@Shishir_Prakash

Coould you check my output and help with further actions?

Re: Query: HPE Ezmeral "internal error" on Kubernetes node

Shishir_Prakash — Fri, 08 Mar 2024 18:31:17 GMT

Can you please share the /var/log/bluedata/bds-mgmt.log from primary controller node and database dump file /tmp/ere-database.txt which can be generated using below command.

/opt/bluedata/common-install/bd_mgmt/bin/bd_mgmt dump_db /tmp/ere-database.txt

Re: Query: HPE Ezmeral "internal error" on Kubernetes node

Andrii_D — Mon, 11 Mar 2024 13:46:51 GMT

Hello, @Shishir_Prakash

I got this files, but they contains sensitive information (passwords, keys etc.).

I deleted all this information and share in such form.

Original file I can share with you via corporate email.

Is it ok for you?
https://drive.google.com/file/d/1HG1IPavHDJQIWSbv_IqUZ73cOH2HXYQd/view?usp=sharing

Re: Query: HPE Ezmeral "internal error" on Kubernetes node

Abe-S — Fri, 15 Mar 2024 02:04:47 GMT

Hi Andrii,

I just requested access to the google drive doc from my gmail account - please can you share it?

thanks

Abe

Re: Query: HPE Ezmeral "internal error" on Kubernetes node

GFustagni — Mon, 16 Sep 2024 09:18:44 GMT

please try ti remove the virtual k8s node, re-image the OS and try to add it again

Re: HPE Ezmeral "internal error" on Kubernetes node

GFustagni — Mon, 16 Sep 2024 09:18:40 GMT

please try ti remove the virtual k8s node, re-image the OS and try to add it again