Problem with appliance backup

[jkre]

Hello,

since upgrading to 9.0.2 the backup job "Morpheus Appliance" is failing. Does anyone know the reason?

Jirka 

[Cparra_]

Hi 

I've seen a few cases where backup jobs start failing right after an upgrade, but it's difficult to say if this is a general 9.0.2 issue or something specific to your environment.

Could you share the exact error message from the Morpheus Appliance backup job or any related logs? That would help narrow it down.

Also, did the backup work normally before the upgrade, and did anything else change besides moving to 9.0.2?

A little more detail should help the community point you in the right direction.

Regards

[CalvinZito]

Agree with @Cparra_ that more information is needed to understand what is going on. I asked my Copilot notebook based on v9.0.1 documentation and it suggests (along with more details) the following:

There are a few things I'd check:

• The appliance backup job backs up the Morpheus database and is controlled from Administration > Settings > Backups > Backup Appliance. The docs note that appliance backups have a hard 2-hour timeout, and backups exceeding that limit will be abandoned and marked failed.
• The v9.0.1 release notes include several upgrade-related fixes, including an issue where upgrades could fail when database connections were closed unexpectedly and another where appliance upgrades could fail because of service/plugin issues. While not backup-specific, they suggest checking appliance health after the upgrade, especially database and queue health.
• I'd also verify the backup target (bucket or file share). Appliance backups write the Morpheus database dump to the configured storage target, so connectivity, credentials, or storage availability issues could cause failures.
• Check Administration > Health for database, storage, and RabbitMQ queue warnings, and review the appliance logs under Administration > Health > Morpheus Logs around the time the backup job runs. The documentation specifically points to these locations for troubleshooting appliance issues.
• If the environment was just upgraded, I'd also confirm that all host/agent updates recommended after the upgrade were completed. The release notes specifically recommend verifying HVM host agent upgrades after upgrading the appliance.

One additional thing to check is appliance backup health. The documentation notes that appliance backups have a hard two-hour execution limit, and any backup that exceeds that limit will fail. If the appliance database has grown significantly or storage performance/access characteristics have changed over time, it's possible a backup that previously completed successfully is now timing out.

It may also be worth reviewing Administration > Health > VM Essentials Health for any warnings or errors. The health checks specifically monitor database connectivity, storage utilization, memory usage, Elasticsearch/OpenSearch health, and RabbitMQ queue backlogs, any of which could contribute to backup-related failures.

Was the backup failure immediate after the upgrade, or did it begin sometime later? Are there any relevant messages in the appliance logs around the time the backup job fails?

Can you share the actual error message from the failed backup job? That will probably narrow it down quickly.

[jkre]

Thank you for your comments. The exact error is:

mysqldump: Got error: 1045: Access denied for user 'morpheus'@'localhost' (using password: YES) when trying to connect.

Health status is OK

[CalvinZito]

I gave my Copilot notebook the additional information and here's what it comes back with.

This suggests the backup job is failing because the appliance cannot authenticate to the local database using the morpheus database account. The backup process relies on creating a MySQL dump of the appliance database, so if database authentication fails, the appliance backup cannot start.

A few things I'd check:

1. Was the problem introduced immediately after upgrading to 9.0.2?
   • If backups worked prior to the upgrade and failed immediately afterward, it's possible the database credentials stored for the backup process no longer match the credentials configured in MySQL.
2. Verify appliance health
   • Navigate to Administration > Health and check the Database status. The VM Essentials Health page specifically monitors database connectivity and database health indicators. Any database access or connection issues should show up there.
3. Check the Morpheus secrets file
   • The documentation shows that the appliance stores the Morpheus database password in: /etc/morpheus/morpheus-secrets.json
   During restore procedures, the documented method is to retrieve the morpheus_password value from that file and use it to connect to the database.
4. Test the database account manually
   • From the appliance shell, try connecting using the credentials stored in morpheus-secrets.json:
   mysql -u morpheus -p -h 127.0.0.1 morpheus
   If that login fails, you've likely identified the root cause: the password in the appliance configuration and the password configured in MySQL are no longer in sync.
5. Review appliance logs
   • Check Administration > Health > Morpheus Logs around the time the backup runs to see whether there are additional database authentication or credential-related errors. The documentation specifically points to this area for appliance troubleshooting.

At this point I would focus almost entirely on resolving the database authentication issue rather than things like backup targets, storage connectivity, or the 2-hour backup timeout. Those can certainly cause backup failures, but the 1045 Access denied message is occurring earlier in the process and is likely the primary problem.

Can you share:

• Whether this started immediately after the 9.0.2 upgrade, and
• The output of a manual MySQL login test using the morpheus account?

That will tell us very quickly whether we're dealing with a credential mismatch versus a broader database issue.

[jkre]

Hi Calvin,

thank you for your reply.

Yes, the problem was introduced after upgrading from 9.0.1 --> 9.0.2

I have tested the DB connection using passwd from /etc/morpheus/morpheus-secrets.json

root@vmem:~# /opt/morpheus/embedded/mysql/bin/mysql -u morpheus -p -h 127.0.0.1 morpheus

The connection is OK. So where can i change the password used in the backup job?

Regards,

Jirka