IMC stateless failover

[pattap]

Hi All

I'm having a bit of a problem. I've just stood up a new server with 7.3 (E0703) as a secondary. Activated as read only with the intention of using it as a backup. I configured both server as per the document below but something isn't right.

https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-c05162584

Backup is stored on the primary as intended but no db sent to the secondary. I can see that they talk to each other fine on port 2810..

My settings below

some logs dbman_debug on the secondary, it's looking good until it gets errlCode = -1

2021-04-11 17:37:49 [DEBUG] [My_Accept_Handler::handle_input] Connection established 10.x.x.x !IP address of the primary 
2021-04-11 17:37:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive encode length: 24
2021-04-11 17:37:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive encode : gx6yYXShEZNreH38UEv3nw==
2021-04-11 17:37:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] After debase64 length: 16
2021-04-11 17:37:49 [ERROR] [decryptMsgAes] EVP_DecryptFinal() failed !
2021-04-11 17:37:49 [ERROR] [decryptMsg] Failed to decryptMsgAes !
2021-04-11 17:37:49 [ERROR] [response_err_code] errCode = -1
2021-04-11 17:37:49 [ERROR] [CDataConnStreamQueueT::deal_msg] dbman_decode_len() for code failed!

any help is appreciated

EDIT;

I'm not sure but it may have something to do with SSL. I hope someone here can make more sense out of it

[jguse]

Hello,

Considering the error is about decryption, I suspect it's being caused by some security files on the secondary system. Can you please try to copy the following files over from the primary system to the same location on the secondary one and then restart the secondary iMC:

• iMC\client\conf\authserver.conf
• iMC\server\conf\imchw.conf  
• iMC\common\conf\ks.dat

Then you can try the restore again and see if it works.

Note the authserver.conf is only needed if you use an external authentication server (LDAP/RADIUS etc) for operator logins - if it's not present, you will get a NullPointerException when trying to login. 

[pattap]

Hi Justin

things are looking slightly better now. I can see DB being trasnffered to the secondary server as per location indicated in the settings. 

However when I log into web interface of the  secondary IMC there is no sign of the database. Side question, are all the admin, users gruops etc are part of the backup process?

Logs from dbman of the primary

2021-04-13 00:47:15 [DEBUG] [Remove] Remove file: D:\Program Files\iMC\dbman\bin\dbop.sql.log
2021-04-13 00:47:15 [DEBUG] [Remove] Remove file: D:\Program Files\iMC\dbman\bin\dbop.sql
2021-04-13 00:47:15 [INFO] [CDbBackup::BackupOneLocalDBase] Success to backup 'D:\IMC DB Backups\127.0.0.1@unba_master_unba_master_db_20210413_003601_full.db'
2021-04-13 00:47:15 [INFO] [CDbBackup::BackupOneLocalDBase] ifAuto: 1;ifTrans: 1;bakDbIp: [IP of secondary imc],bakDBReceivePath: F:\IMC DB Backups
2021-04-13 00:47:15 [INFO] [Client::connect_to_server] Starting connect to [IP of secondary imc]: 2810
2021-04-13 00:47:15 [INFO] [Client::connect_to_server] Established connection to [IP of secondary imc]: 2810
2021-04-13 00:47:16 [INFO] [Client::send_filetrans_msg] Receive server data ok
2021-04-13 00:47:16 [INFO] [CDbBackup::BackupOneLocalDBase] Backup disk space is OK £¬code = 5 !
2021-04-13 00:47:16 [DEBUG] [Remove] Remove file: D:\IMC DB Backups\127.0.0.1@unba_master_unba_master_db_20210413_003601_full.db
2021-04-13 00:47:16 [INFO] [CDbBackup::BackupAllDataBase] Path: 127.0.0.1@unba_slave_unba_slave_db_20210413_003601_full.db
2021-04-13 00:47:16 [ERROR] [locateDbPosition] Can not find DB[127.0.0.1@unba_slave_unba_slave] information in config !
2021-04-13 00:47:16 [ERROR] [CDbBackup::BackupAllDataBase] Fail to found DB[127.0.0.1@unba_slave_unba_slave] information in backup system config
2021-04-13 00:47:16 [ERROR] [CommonBackup] backup all db failed .
2021-04-13 00:47:49 [DEBUG] [compareVersion] [debug Compare]m_BackHostIp.c_str(): [IP of secondary imc].
2021-04-13 00:47:49 [INFO] [Client::connect_to_server] Starting connect to [IP of secondary imc]: 2810
2021-04-13 00:47:49 [INFO] [Client::connect_to_server] Established connection to [IP of secondary imc]: 2810
2021-04-13 00:47:49 [ERROR] [Client::recv] unknown response code:

2021-04-13 00:47:49 [ERROR] [Client::getVersionMsg] Fail to receive data
2021-04-13 00:57:49 [DEBUG] [compareVersion] [debug Compare]m_BackHostIp.c_str(): [IP of secondary imc].
2021-04-13 00:57:49 [INFO] [Client::connect_to_server] Starting connect to [IP of secondary imc]: 2810
2021-04-13 00:57:49 [INFO] [Client::connect_to_server] Established connection to [IP of secondary imc]: 2810
2021-04-13 00:57:49 [ERROR] [Client::recv] unknown response code:

2021-04-13 00:57:49 [ERROR] [Client::getVersionMsg] Fail to receive data
2021-04-13 01:07:49 [DEBUG] [compareVersion] [debug Compare]m_BackHostIp.c_str(): [IP of secondary imc].
2021-04-13 01:07:49 [INFO] [Client::connect_to_server] Starting connect to [IP of secondary imc]: 2810
2021-04-13 01:07:49 [INFO] [Client::connect_to_server] Established connection to [IP of secondary imc]: 2810
2021-04-13 01:07:49 [ERROR] [Client::recv] unknown response code:

2021-04-13 01:07:49 [ERROR] [Client::getVersionMsg] Fail to receive data

Secondary dbman debug logs 

2021-04-13 00:27:40 [INFO] [Main] Version: 7.3
2021-04-13 00:27:41 [INFO] [Main] Global directory: F:/Program Files/iMC/dbman
2021-04-13 00:27:41 [INFO] [Main] Listenning on port: 2810
2021-04-13 00:27:41 [DEBUG] [Main] arv count 2
2021-04-13 00:27:41 [DEBUG] [Main] arv 1: dbman
2021-04-13 00:27:41 [DEBUG] [Main] arv 2: -n
2021-04-13 00:27:41 [INFO] [DBMAN] dbman exit
2021-04-13 00:27:49 [DEBUG] [My_Accept_Handler::handle_input] Connection established [IP of primary imc]
2021-04-13 00:27:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive encode length: 24
2021-04-13 00:27:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive encode : gx6yYXShEZNreH38UEv3nw==
2021-04-13 00:27:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] After debase64 length: 16
2021-04-13 00:27:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive command code: 10021
2021-04-13 00:36:01 [DEBUG] [My_Accept_Handler::handle_input] Connection established [IP of primary imc]
2021-04-13 00:36:01 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive encode length: 24
2021-04-13 00:36:01 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive encode : anQAHLpJ8OcqjrtNIiB4jw==
2021-04-13 00:36:01 [DEBUG] [CDataConnStreamQueueT::deal_msg] After debase64 length: 16
2021-04-13 00:36:01 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive command code: 10011
2021-04-13 00:36:01 [DEBUG] [CDataConnStreamQueueT::deal_msg] Succeed to create SendBakConfigFile thread
2021-04-13 00:36:01 [INFO] [Client::connect_to_server] Starting connect to [IP of primary imc]: 2810
2021-04-13 00:36:01 [INFO] [Client::connect_to_server] Established connection to [IP of primary imc]: 2810
2021-04-13 00:36:01 [INFO] [Client::send_filetrans_msg] Receive server data ok
2021-04-13 00:36:02 [DEBUG] [My_Accept_Handler::handle_input] Connection established [IP of primary imc]
2021-04-13 00:36:02 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive encode length: 24
2021-04-13 00:36:02 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive encode : oY1rQHA/q1LeqRv0vgRv9w==
2021-04-13 00:36:02 [DEBUG] [CDataConnStreamQueueT::deal_msg] After debase64 length: 16
2021-04-13 00:36:02 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive command code: 10010
2021-04-13 00:36:02 [DEBUG] [CDataConnStreamQueueT::deal_msg] Succeed to create FileTrans thread
2021-04-13 00:37:49 [DEBUG] [My_Accept_Handler::handle_input] Connection established [IP of primary imc]
2021-04-13 00:37:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive encode length: 24
2021-04-13 00:37:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive encode : gx6yYXShEZNreH38UEv3nw==
######### those would carry on for many more lines #########
2021-04-13 00:47:15 [DEBUG] [CDataConnStreamQueueT::deal_msg] After debase64 length: 16
2021-04-13 00:47:15 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive command code: 10010
2021-04-13 00:47:15 [DEBUG] [CDataConnStreamQueueT::deal_msg] Succeed to create FileTrans thread
2021-04-13 00:47:49 [DEBUG] [My_Accept_Handler::handle_input] Connection established [IP of primary imc]
2021-04-13 00:47:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive encode length: 24
2021-04-13 00:47:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive encode : gx6yYXShEZNreH38UEv3nw==
2021-04-13 00:47:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] After debase64 length: 16
2021-04-13 00:47:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive command code: 10021
2021-04-13 00:57:49 [DEBUG] [My_Accept_Handler::handle_input] Connection established [IP of primary imc]
2021-04-13 00:57:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive encode length: 24
2021-04-13 00:57:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive encode : gx6yYXShEZNreH38UEv3nw==
2021-04-13 00:57:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] After debase64 length: 16
2021-04-13 00:57:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive command code: 10021
2021-04-13 01:07:49 [DEBUG] [My_Accept_Handler::handle_input] Connection established [IP of primary imc]
2021-04-13 01:07:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive encode length: 24
2021-04-13 01:07:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive encode : gx6yYXShEZNreH38UEv3nw==
2021-04-13 01:07:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] After debase64 length: 16
2021-04-13 01:07:49 [DEBUG] [CDataConnStreamQueueT::deal_msg] Receive command code: 10021

[jguse]

Hello,

That's good news, at least the initial error is no longer present. You are now running into an issue with unba_slave DB, the one that stores the history data for the NTA module. This is not usually backed up or restored by default, and requires deleting the file unba.skip in the iMC\dbman\etc\ folder so that it will be transferred to the secondary server.

The error: Can not find DB[127.0.0.1@unba_slave_unba_slave] information in config ! - means that while DBMAN is backing up this DB on the primary system, it can't find any information about this DB on the secondary.

Could it be that you have not deleted the unba.skip file on the secondary system? Keep in mind you also need to re-configure the Auto Restore Settings on the system after removing the file, ensuring you check the box for unba_slave to be restored as well.

PS. Yes, the iMC operators are part of the backup and restore process. They are copied over with config_db and the corresponding plat.zip file restore.

[pattap]

Justin

Yes that did the trick. I can see the db being copied and it shows in monitoring agent.

However I'm not able to log in to IMC anymore, not even with default credentials.

Number of nodes exceeds the license size error appears:

To activate the secondary instances I've used the same key we used for primary and selected backup.

Do I need to get/generate a new license for the backup server? 

[jguse]

Hello,

That's odd, you should not be getting this error with an Unlimited license, unless you have some optional modules that have exceeded license limits. It sounds like you had applied the license correctly - it should be entered as Backup on the secondary server, and that will turn it into read-only with the same license limits as the primary system.

You could try to re-apply the license via the Activate page and restart iMC to see if that helps. If you still get this error, please open a support case for our licensing team.

PS. The default 'admin' user login should still work 2 times on the secondary system where you get this error.