KVM Cluster Installation on RHEL7 Server

[AungKT]

Hello,

I am facing the below error messages when I tried to install KVM on RHEL server by using Morpheus Cloud type and Morpheus KVM CentOS Cluster Layout type.

RHEL configured subscription and EPEL repo.

Error message from /opt/morpheus-node/embedded/cookbooks/chef-run.log

[2023-01-26T23:25:43+08:00] INFO: directory[/var/run/morpheus-node/morphd] mode changed to 750
[2023-01-26T23:25:43+08:00] INFO: template[/opt/morpheus-node/conf/Config.groovy] backed up to /var/opt/morpheus-node/chef/local-mode-cache/backup/opt/morpheus-node/conf/Config.groovy.chef-20230126232543.593405
[2023-01-26T23:25:43+08:00] INFO: template[/opt/morpheus-node/conf/Config.groovy] updated file contents /opt/morpheus-node/conf/Config.groovy
[2023-01-26T23:25:52+08:00] INFO: Running queued delayed notifications before re-raising exception
[2023-01-26T23:25:52+08:00] ERROR: Running exception handlers
[2023-01-26T23:25:52+08:00] ERROR: Exception handlers complete
[2023-01-26T23:25:52+08:00] FATAL: Stacktrace dumped to /opt/morpheus-node/embedded/cookbooks/cache/chef-stacktrace.out
[2023-01-26T23:25:52+08:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2023-01-26T23:25:52+08:00] FATAL: NoMethodError: execute[enable_rhel_extras_repo] (morpheus-solo-node::docker line 123) had an error: NoMethodError: undefined method `[]' for nil:NilClass




[2023-01-26T23:48:13+08:00] INFO: Started chef-zero at chefzero://localhost:1 with repository at /opt/morpheus-node/embedded
  One version per cookbook

[2023-01-26T23:48:13+08:00] INFO: *** Chef 14.14.29 ***
[2023-01-26T23:48:13+08:00] INFO: Platform: x86_64-linux
[2023-01-26T23:48:13+08:00] INFO: Chef-client pid: 11075
[2023-01-26T23:48:13+08:00] INFO: The plugin path /etc/chef/ohai/plugins does not exist. Skipping...
[2023-01-26T23:48:16+08:00] INFO: Setting the run_list to ["recipe[morpheus-solo-node]"] from CLI options
[2023-01-26T23:48:16+08:00] INFO: Run List is [recipe[morpheus-solo-node]]
[2023-01-26T23:48:16+08:00] INFO: Run List expands to [morpheus-solo-node]
[2023-01-26T23:48:16+08:00] INFO: Starting Chef Run for rhel-kvm-hsot03
[2023-01-26T23:48:16+08:00] INFO: Running start handlers
[2023-01-26T23:48:16+08:00] INFO: Start handlers complete.
[2023-01-26T23:48:16+08:00] INFO: Loading cookbooks [morpheus-solo-node@0.0.1, runit@0.14.2, docker@4.9.2, apt@7.1.1, yum@5.1.0, lvm@4.5.1, parted@2.1.2, chef-yum-docker@3.1.1, apt@7.1.1]
[2023-01-26T23:48:18+08:00] INFO: Don't run Bundler as root. Bundler can ask for sudo if it is needed, and
installing your bundle as root will break this application for all non-root
users on this machine.
Fetching gem metadata from https://www.rubygems.org/.
Resolving dependencies...
Using bundler 1.16.6
Using excon 0.97.2
Using multi_json 1.15.0
Using docker-api 1.34.2
Bundle complete! 1 Gemfile dependency, 4 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.

[2023-01-26T23:48:19+08:00] INFO: template[/opt/morpheus-node/conf/Config.groovy] backed up to /var/opt/morpheus-node/chef/local-mode-cache/backup/opt/morpheus-node/conf/Config.groovy.chef-20230126234819.702190
[2023-01-26T23:48:19+08:00] INFO: template[/opt/morpheus-node/conf/Config.groovy] removed backup at /var/opt/morpheus-node/chef/local-mode-cache/backup/opt/morpheus-node/conf/Config.groovy.chef-20230126195807.693069
[2023-01-26T23:48:19+08:00] INFO: template[/opt/morpheus-node/conf/Config.groovy] updated file contents /opt/morpheus-node/conf/Config.groovy
[2023-01-26T23:48:28+08:00] INFO: Running queued delayed notifications before re-raising exception
[2023-01-26T23:48:28+08:00] ERROR: Running exception handlers
[2023-01-26T23:48:28+08:00] ERROR: Exception handlers complete
[2023-01-26T23:48:28+08:00] FATAL: Stacktrace dumped to /opt/morpheus-node/embedded/cookbooks/cache/chef-stacktrace.out
[2023-01-26T23:48:28+08:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2023-01-26T23:48:28+08:00] FATAL: NoMethodError: execute[enable_rhel_extras_repo] (morpheus-solo-node::docker line 123) had an error: NoMethodError: undefined method `[]' for nil:NilClass

Additional logs from /opt/morpheus-node/embedded/cookbooks/cache/chef-stacktrace.out

NoMethodError: execute[enable_rhel_extras_repo] (morpheus-solo-node::docker line 123) had an error: NoMethodError: undefined method `[]' for nil:NilClass
/opt/morpheus-node/embedded/cookbooks/cache/cookbooks/morpheus-solo-node/recipes/docker.rb:125:in `block (2 levels) in from_file'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource/conditional.rb:106:in `evaluate_block'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource/conditional.rb:95:in `evaluate'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource/conditional.rb:86:in `continue?'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource.rb:1526:in `block in should_skip?'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource.rb:1525:in `each'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource.rb:1525:in `find'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource.rb:1525:in `should_skip?'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource.rb:577:in `run_action'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/runner.rb:74:in `run_action'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/runner.rb:108:in `each'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/runner.rb:108:in `run_all_actions'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/runner.rb:132:in `block in converge'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource_collection/resource_list.rb:94:in `block in execute_each_resource'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource_collection/resource_list.rb:92:in `execute_each_resource'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/runner.rb:130:in `converge'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/client.rb:720:in `block in converge'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/client.rb:715:in `catch'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/client.rb:715:in `converge'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/client.rb:754:in `converge_and_save'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/client.rb:286:in `run'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/application.rb:303:in `run_with_graceful_exit_option'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/application.rb:279:in `block in run_chef_client'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/local_mode.rb:44:in `with_server_connectivity'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/application.rb:261:in `run_chef_client'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/application/client.rb:449:in `run_application'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/application.rb:66:in `run'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/bin/chef-client:25:in `<top (required)>'
/opt/morpheus-node/embedded/bin/chef-client:153:in `load'
/opt/morpheus-node/embedded/bin/chef-client:153:in `<main>'

>>>> Caused by NoMethodError: undefined method `[]' for nil:NilClass
/opt/morpheus-node/embedded/cookbooks/cache/cookbooks/morpheus-solo-node/recipes/docker.rb:125:in `block (2 levels) in from_file'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource/conditional.rb:106:in `evaluate_block'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource/conditional.rb:95:in `evaluate'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource/conditional.rb:86:in `continue?'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource.rb:1526:in `block in should_skip?'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource.rb:1525:in `each'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource.rb:1525:in `find'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource.rb:1525:in `should_skip?'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource.rb:577:in `run_action'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/runner.rb:74:in `run_action'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/runner.rb:108:in `block in run_all_actions'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/runner.rb:108:in `each'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/runner.rb:108:in `run_all_actions'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/runner.rb:132:in `block in converge'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource_collection/resource_list.rb:94:in `block in execute_each_resource'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/resource_collection/resource_list.rb:92:in `execute_each_resource'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/runner.rb:130:in `converge'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/client.rb:720:in `block in converge'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/client.rb:715:in `catch'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/client.rb:715:in `converge'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/client.rb:754:in `converge_and_save'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/client.rb:286:in `run'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/application.rb:303:in `run_with_graceful_exit_option'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/application.rb:279:in `block in run_chef_client'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/local_mode.rb:44:in `with_server_connectivity'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/application.rb:261:in `run_chef_client'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/application/client.rb:449:in `run_application'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/lib/chef/application.rb:66:in `run'
/opt/morpheus-node/embedded/lib/ruby/gems/2.5.0/gems/chef-14.14.29/bin/chef-client:25:in `<top (required)>'
/opt/morpheus-node/embedded/bin/chef-client:153:in `load'

[AungKT]

Hello @alipscombe, I am using 7.9 as below.

NAME="Red Hat Enterprise Linux Server"
VERSION="7.9 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.9"
PRETTY_NAME="Red Hat Enterprise Linux"
ANSI_COLOR="0;31"

It says repository not found for rhel-7-server-rhui-extras-rpms when I tried to enable the repo and didn’t see this repo in the available list. below are my enabled repos.

Repo ID:   rhel-7-server-e4s-optional-rpms
Repo Name: Red Hat Enterprise Linux 7 Server - Update Services for SAP Solutions - Optional (RPMs)
Repo URL:  https://cdn.redhat.com/content/e4s/rhel/server/7/$releasever/$basearch/optional/os
Enabled:   1
--
Repo ID:   rhel-7-server-extras-rpms
Repo Name: Red Hat Enterprise Linux 7 Server - Extras (RPMs)
Repo URL:  https://cdn.redhat.com/content/dist/rhel/server/7/7Server/$basearch/extras/os
Enabled:   1
--
Repo ID:   rhel-7-server-eus-optional-rpms
Repo Name: Red Hat Enterprise Linux 7 Server - Extended Update Support - Optional (RPMs)
Repo URL:  https://cdn.redhat.com/content/eus/rhel/server/7/$releasever/$basearch/optional/os
Enabled:   1
--
Repo ID:   rhel-7-server-optional-rpms
Repo Name: Red Hat Enterprise Linux 7 Server - Optional (RPMs)
Repo URL:  https://cdn.redhat.com/content/dist/rhel/server/7/$releasever/$basearch/optional/os
Enabled:   1
--
Repo ID:   rhel-ha-for-rhel-7-server-rpms
Repo Name: Red Hat Enterprise Linux High Availability (for RHEL 7 Server) (RPMs)
Repo URL:  https://cdn.redhat.com/content/dist/rhel/server/7/$releasever/$basearch/highavailability/os
Enabled:   1
--
Repo ID:   rhel-7-server-rpms
Repo Name: Red Hat Enterprise Linux 7 Server (RPMs)
Repo URL:  https://cdn.redhat.com/content/dist/rhel/server/7/$releasever/$basearch/os
Enabled:   1

Thanks,
Aung

[AungKT]

@alipscombe will be great if you can also provide some suggestions.

Thanks,
Aung

[#4]

hello @aungkyawthu It appears that your deployment is failing when attempting to enable the rhel-7-server-rhui-extras-rpms repository on the KVM host.

Please can you manually execute sudo yum-config-manager --enable rhel-7-server-rhui-extras-rpms on the host. can you also share the version of RHEL 7 you are using.

[AungKT]

Hello @alipscombe, yes I raised a case with the product support team today.

Thanks for your help.

Regards,
Aung

[AungKT]

Yes, sure. here is morpheus-node-3.2.10-1.el7.x86_64

Morpheus is V 5.5.3.

[#7]

Hello,

Can you tell me which morpheus-node package version you are installing? You can get it by running

rpm -q morpheus-node

I can try to reproduce this on RHEL 7.

[#8]

Hello @aungkyawthu

The rhel-7-server-rhui-extras-rpms repository requires an RHUA instance running within your infrastructure.

When registering the RHEL 7 host with subscription-manager you need to pass in the argument --type=rhui command example: subscription-manager register --type=rhui --username <admin-example> --password <secret>

I believe the rhel-7-server-rhui-extras-rpms is a copy of the rhel-7-server-extras-rpms repository.

Please can you raise a case with the product support team? So that we can investigate further.

Thanks
Adam

[AungKT]

Hello @alipscombe , I tested again with a new fresh server and got same error.

[2023-01-30T14:05:16+08:00] INFO: directory[/var/run/morpheus-node/morphd] mode changed to 750
[2023-01-30T14:05:16+08:00] INFO: template[/opt/morpheus-node/conf/Config.groovy] backed up to /var/opt/morpheus-node/chef/local-mode-cache/backup/opt/morpheus-node/conf/Config.groovy.chef-20230130140516.675346
[2023-01-30T14:05:16+08:00] INFO: template[/opt/morpheus-node/conf/Config.groovy] updated file contents /opt/morpheus-node/conf/Config.groovy
[2023-01-30T14:05:25+08:00] INFO: template[/etc/yum.repos.d/Docker.repo] created file /etc/yum.repos.d/Docker.repo
[2023-01-30T14:05:25+08:00] INFO: template[/etc/yum.repos.d/Docker.repo] updated file contents /etc/yum.repos.d/Docker.repo
[2023-01-30T14:05:25+08:00] INFO: template[/etc/yum.repos.d/Docker.repo] mode changed to 644
[2023-01-30T14:05:25+08:00] INFO: template[/etc/yum.repos.d/Docker.repo] sending run action to execute[yum clean metadata Docker] (immediate)
[2023-01-30T14:05:30+08:00] INFO: execute[yum clean metadata Docker] ran successfully
[2023-01-30T14:05:30+08:00] INFO: template[/etc/yum.repos.d/Docker.repo] sending run action to execute[yum-makecache-Docker] (immediate)
[2023-01-30T14:05:37+08:00] INFO: execute[yum-makecache-Docker] ran successfully
[2023-01-30T14:05:37+08:00] INFO: template[/etc/yum.repos.d/Docker.repo] sending create action to ruby_block[package-cache-reload-Docker] (immediate)
[2023-01-30T14:05:37+08:00] INFO: ruby_block[package-cache-reload-Docker] called
[2023-01-30T14:05:37+08:00] INFO: Running queued delayed notifications before re-raising exception
[2023-01-30T14:05:37+08:00] ERROR: Running exception handlers
[2023-01-30T14:05:37+08:00] ERROR: Exception handlers complete
[2023-01-30T14:05:37+08:00] FATAL: Stacktrace dumped to /opt/morpheus-node/embedded/cookbooks/cache/chef-stacktrace.out
[2023-01-30T14:05:37+08:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2023-01-30T14:05:37+08:00] FATAL: NoMethodError: execute[enable_rhel_extras_repo] (morpheus-solo-node::docker line 123) had an error: NoMethodError: undefined method `[]' for nil:NilClass

# yum repolist
Loaded plugins: product-id, search-disabled-repos, subscription-manager
repo id                                                                                 repo name                                                                                                       status
Docker/7Server/x86_64                                                                   Yum Repository                                                                                                     198
epel/x86_64                                                                             Extra Packages for Enterprise Linux 7 - x86_64                                                                  13,740
morpheus/7Server/x86_64                                                                 Morpheus: Next-Gen Platform as a Service (PaaS) Software                                                             4
rhel-7-server-rpms/7Server/x86_64                                                       Red Hat Enterprise Linux 7 Server (RPMs)                                                                        33,552
repolist: 47,494