Failed to create HVM cluster in 8.0.10

[PeterTzvetanov]

Hi Dya,

for the moment there are only 2 systems known with that type of issue:
1. Yours
and
2. From a co-worker of mine.

All other cases were related to other type of issues where after resolving them, everything worked fine.

With regards

[dya]

So other people are not having any problems.
I was able to create clusters many times in past versions, but I'm really surprised that I can't do it in 8.0.10 without implementing a workaround.

I guess I'll have to give up.

Thank you.

[PeterTzvetanov]

Hi Dya,

so far I know my co-worker will get it to the developers and they might get some light on that. How quickly it will be, I can't say for now as the workaround is allowing you to proceed, so that there is no full system down from the priority point of view and the rareness of the issue.

With regards

[dya]

I believe the priority determination is reasonable.

It appears the following error is output to cloud-init-out.log. It's unclear whether this is normally output, but it is concerning.

root@vme-iscsi-manager:~# grep “Error executing action” /var/log/cloud-init-output.log
Error executing action `create` on resource ‘ruby_block[reload_log_service]’
Error executing action `enable` on resource ‘runit_service[morpheus-ui]’
root@vme-iscsi-manager:~#

I don't know the cause, but it seems the following part of the user-data executed by cloud-init is not running correctly.

runcmd:
# Change Morpheus Appliance URL
- echo "appliance_url 'https://vme-iscsi-manager.home.example'" > /etc/morpheus/morpheus.rb
# Shutdown and reconfigure Morpheus
- systemctl enable --now morpheus-runsvdir
- morpheus-ctl stop morpheus-ui
- morpheus-ctl reconfigure
- morpheus-ctl restart nginx
- morpheus-ctl start morpheus-ui

[dya]

I verified this to understand the mechanism.

・I confirmed that Morpheus uses Chef internally.
・Chef is running during the execution of “morpheus-ctl reconfigure”, but an error occurs there.
・I have absolutely no idea why it fails when processed via cloud-init, yet completes successfully when executed after logging into the manager VM post-deployment.
・This does not happen in 8.0.9.

This is difficult...

■Part of user-data

root@vme-iscsi-manager:~# grep -A 10 "runcmd:" /var/lib/cloud/instances/nocloud/user-data.txt
runcmd:
# Change Morpheus Appliance URL
- echo "appliance_url 'https://vme-iscsi-manager.home.example'" > /etc/morpheus/morpheus.rb
# Shutdown and reconfigure Morpheus
- systemctl enable --now morpheus-runsvdir
- morpheus-ctl stop morpheus-ui
- morpheus-ctl reconfigure
- morpheus-ctl restart nginx
- morpheus-ctl start morpheus-ui
root@vme-iscsi-manager:~#

■/var/log/cloud-init-output.log after manager deployment

root@vme-iscsi-manager:~# tail -n 7 /var/log/cloud-init-output.log

Running handlers:
Running handlers complete
Chef Client failed. 318 resources updated in 11 minutes 23 seconds
ok: run: nginx: (pid 32762) 1s
ok: run: morpheus-ui: (pid 32742) 6s
Cloud-init v. 25.2-0ubuntu1~24.04.1 finished at Sat, 25 Oct 2025 02:38:55 +0000. Datasource DataSourceNoCloud [seed=/dev/sr0]. Up 737.40 seconds
root@vme-iscsi-manager:~#

■/opt/morpheus/embedded/cookbooks/chef-run.log after manager deployment

[2025-10-25T02:38:51+00:00] INFO: template[/opt/morpheus/embedded/nginx/conf/nginx.conf] sending restart action to runit_service[nginx] (delayed)
[2025-10-25T02:38:52+00:00] INFO: bash[build_guacamole-server] sending restart action to runit_service[guacd] (delayed)
[2025-10-25T02:38:52+00:00] ERROR: Running exception handlers
[2025-10-25T02:38:52+00:00] ERROR: Exception handlers complete
[2025-10-25T02:38:52+00:00] FATAL: Stacktrace dumped to /opt/morpheus/embedded/cookbooks/cache/chef-stacktrace.out
[2025-10-25T02:38:52+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2025-10-25T02:38:52+00:00] FATAL: Chef::Exceptions::MultipleFailures: Multiple failures occurred:
* Mixlib::ShellOut::ShellCommandFailed occurred in delayed notification: ruby_block[restart_log_service] (/opt/morpheus/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb line 69) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/morpheus/embedded/bin/sv restart /opt/morpheus/service/morpheus-ui/log ----
STDOUT: timeout: run: /opt/morpheus/service/morpheus-ui/log: (pid 32734) 8s
STDERR:
---- End output of /opt/morpheus/embedded/bin/sv restart /opt/morpheus/service/morpheus-ui/log ----
Ran /opt/morpheus/embedded/bin/sv restart /opt/morpheus/service/morpheus-ui/log returned 1
* Mixlib::ShellOut::ShellCommandFailed occurred in delayed notification: ruby_block[reload_log_service] (/opt/morpheus/embedded/cookbooks/cache/cookbooks/runit/libraries/provider_runit_service.rb line 77) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/morpheus/embedded/bin/sv force-reload /opt/morpheus/service/morpheus-ui/log ----
STDOUT: kill: run: /opt/morpheus/service/morpheus-ui/log: (pid 32734) 17s
STDERR:
---- End output of /opt/morpheus/embedded/bin/sv force-reload /opt/morpheus/service/morpheus-ui/log ----
Ran /opt/morpheus/embedded/bin/sv force-reload /opt/morpheus/service/morpheus-ui/log returned 1