Failed to create HVM cluster in 8.0.10

[PeterTzvetanov]

Hi Dya,

for the moment there are only 2 systems known with that type of issue:
1. Yours
and
2. From a co-worker of mine.

All other cases were related to other type of issues where after resolving them, everything worked fine.

With regards

[dya]

So other people are not having any problems.
I was able to create clusters many times in past versions, but I'm really surprised that I can't do it in 8.0.10 without implementing a workaround.

I guess I'll have to give up.

Thank you.

[PeterTzvetanov]

Hi Dya,

so far I know my co-worker will get it to the developers and they might get some light on that. How quickly it will be, I can't say for now as the workaround is allowing you to proceed, so that there is no full system down from the priority point of view and the rareness of the issue.

With regards

[dya]

I believe the priority determination is reasonable.

It appears the following error is output to cloud-init-out.log. It's unclear whether this is normally output, but it is concerning.

root@vme-iscsi-manager:~# grep “Error executing action” /var/log/cloud-init-output.log
Error executing action `create` on resource ‘ruby_block[reload_log_service]’
Error executing action `enable` on resource ‘runit_service[morpheus-ui]’
root@vme-iscsi-manager:~#

I don't know the cause, but it seems the following part of the user-data executed by cloud-init is not running correctly.

runcmd:
# Change Morpheus Appliance URL
- echo "appliance_url 'https://vme-iscsi-manager.home.example'" > /etc/morpheus/morpheus.rb
# Shutdown and reconfigure Morpheus
- systemctl enable --now morpheus-runsvdir
- morpheus-ctl stop morpheus-ui
- morpheus-ctl reconfigure
- morpheus-ctl restart nginx
- morpheus-ctl start morpheus-ui

[dya]

I verified this to understand the mechanism.

・I confirmed that Morpheus uses Chef internally.
・Chef is running during the execution of “morpheus-ctl reconfigure”, but an error occurs there.
・I have absolutely no idea why it fails when processed via cloud-init, yet completes successfully when executed after logging into the manager VM post-deployment.
・This does not happen in 8.0.9.

This is difficult...

■Part of user-data

root@vme-iscsi-manager:~# grep -A 10 "runcmd:" /var/lib/cloud/instances/nocloud/user-data.txt
runcmd:
# Change Morpheus Appliance URL
- echo "appliance_url 'https://vme-iscsi-manager.home.example'" > /etc/morpheus/morpheus.rb
# Shutdown and reconfigure Morpheus
- systemctl enable --now morpheus-runsvdir
- morpheus-ctl stop morpheus-ui
- morpheus-ctl reconfigure
- morpheus-ctl restart nginx
- morpheus-ctl start morpheus-ui
root@vme-iscsi-manager:~#

■/var/log/cloud-init-output.log after manager deployment

root@vme-iscsi-manager:~# tail -n 7 /var/log/cloud-init-output.log

Running handlers:
Running handlers complete
Chef Client failed. 318 resources updated in 11 minutes 23 seconds
ok: run: nginx: (pid 32762) 1s
ok: run: morpheus-ui: (pid 32742) 6s
Cloud-init v. 25.2-0ubuntu1~24.04.1 finished at Sat, 25 Oct 2025 02:38:55 +0000. Datasource DataSourceNoCloud [seed=/dev/sr0]. Up 737.40 seconds
root@vme-iscsi-manager:~#

■/opt/morpheus/embedded/cookbooks/chef-run.log after manager deployment

[2025-10-25T02:38:51+00:00] INFO: template[/opt/morpheus/embedded/nginx/conf/nginx.conf] sending restart action to runit_service[nginx] (delayed)
[2025-10-25T02:38:52+00:00] INFO: bash[build_guacamole-server] sending restart action to runit_service[guacd] (delayed)
[2025-10-25T02:38:52+00:00] ERROR: Running exception handlers
[2025-10-25T02:38:52+00:00] ERROR: Exception handlers complete
[2025-10-25T02:38:52+00:00] FATAL: Stacktrace dumped to /opt/morpheus/embedded/cookbooks/cache/chef-stacktrace.out
[2025-10-25T02:38:52+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2025-10-25T02:38:52+00:00] FATAL: Chef::Exceptions::MultipleFailures: Multiple failures occurred:
* Mixlib::ShellOut::ShellCommandFailed occurred in delayed notification: ruby_block[restart_log_service] (/opt/morpheus/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb line 69) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/morpheus/embedded/bin/sv restart /opt/morpheus/service/morpheus-ui/log ----
STDOUT: timeout: run: /opt/morpheus/service/morpheus-ui/log: (pid 32734) 8s
STDERR:
---- End output of /opt/morpheus/embedded/bin/sv restart /opt/morpheus/service/morpheus-ui/log ----
Ran /opt/morpheus/embedded/bin/sv restart /opt/morpheus/service/morpheus-ui/log returned 1
* Mixlib::ShellOut::ShellCommandFailed occurred in delayed notification: ruby_block[reload_log_service] (/opt/morpheus/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb line 77) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/morpheus/embedded/bin/sv force-reload /opt/morpheus/service/morpheus-ui/log ----
STDOUT: kill: run: /opt/morpheus/service/morpheus-ui/log: (pid 32734) 17s
STDERR:
---- End output of /opt/morpheus/embedded/bin/sv force-reload /opt/morpheus/service/morpheus-ui/log ----
Ran /opt/morpheus/embedded/bin/sv force-reload /opt/morpheus/service/morpheus-ui/log returned 1

[dya]

Is there any new information?

While some details remain unclear, certain aspects have clearly changed in 8.0.10, and I'm concerned about how things will turn out in the next release.

[tsuneo]

こんにちは

verison 8.0.11 でも同様の事象が発生しており、Manager のリポジトリがダメになっていると思うのですが、他の方も発生していますか？
前回同様クラスター作成時に失敗します。

[dya]

The same issue occurred in 8.0.11.

After installing the manager via hpe-vm, I performed the following actions. This is the same as for 8.0.10.

1. Connected to the manager via console (virsh console)
2. Verified cloud-init processing completed successfully
# tail -n 10 /var/log/cloud-init-output.log
　✔ Cloud-init processing must be finished
　　→ Output like “Cloud-init v. 25.2-0ubuntu1~24.04.1 finished” is acceptable
　✔Chef processing has completed successfully
　　→“Chef Client failed” is NG
　　→“Chef Client finished” is OK
3. Handling when cloud-init completes but Chef fails
　　# morpheus-ctl stop morpheus-ui
　　# morpheus-ctl reconfigure
　　✔If “Chef Client finished” is output, it is OK
　　# ls -l /var/opt/morpheus/package-repos/apt/dists/morpheus/
　　✔ If the “Release” file exists, it's OK
　　# morpheus-ctl start morpheus-ui
4. Verify service startup (morpheus-ui)
　　# tail -f /var/log/morpheus/morpheus-ui/current
　　✔ If the “morpheus” logo appears, it's OK

That's all.

[tsuneo]

返信いただき、ありがとうございます。
やはり、同じ現象が発生しているのですね、、、
version 8.0.11 で改善されると考えていたので、早急な修正を期待します。

[msamaniego1991]

Yes, I was experiencing the same issue. I have certain restrictions on internet access for the devices, and some exceptions were made. However, when I clicked "retry the process," it would run internally again, and after a couple of attempts, it would pass one stage to add the node but then stop at another. It would retry from where it had stopped, and so on, repeating this process for several stages until the node was finally fully added to the cluster. I'm not sure why this happened; it must be a network restriction issue, or perhaps it's trying other repositories internally.