Problem with ACL module

[kiml]

Hi!

After an upgrade from 7.3_E0504 to e0506p07 the ACL module stopped working.  We recently upgraded to 7.3_E0605P04 hoping it would magically start working again but sadly it did not :)
On the login page there is a red triangle with the text:

Failed to load components during the system start up:
Component name:iMC Platform - ACL Management
Reason:Database access error.
Remedy:Confirm database is running, and restart 'jserver' process.

We're running iMC on a single (debian) host with mysql on the same host.

From /opt/iMC/deploy/log/dmalog.txt i can se a database connect error but this is the only error i can find in the log files. The password for each module is also displayed in cleartext for some reason so i obfuscated them here:

Spoiler

2018-09-07 12:18:47 [INFO ] [iMC-Database-Connect-Check] [com.h3c.imc.deploy.dma.monitor.DatabaseConnectMonitor::run(84)] DatabaseConfig{type=MySQL, address=127.0.0.1, port=null, name=aclm_db, userName=imc_aclm, password=randomPassword, instanceName=null}
2018-09-07 12:18:47 [INFO ] [iMC-Database-Connect-Check] [com.h3c.imc.deploy.dma.monitor.DatabaseConnectMonitor::run(84)] DatabaseConfig{type=MySQL, address=127.0.0.1, port=null, name=icc_db, userName=imc_icc, password=randomPassword, instanceName=null}
2018-09-07 12:18:47 [INFO ] [iMC-Database-Connect-Check] [com.h3c.imc.deploy.dma.monitor.DatabaseConnectMonitor::run(84)] DatabaseConfig{type=MySQL, address=127.0.0.1, port=null, name=invent_db, userName=imc_inventory, password=randomPassword, instanceName=null}
2018-09-07 12:18:47 [INFO ] [iMC-Database-Connect-Check] [com.h3c.imc.deploy.dma.monitor.DatabaseConnectMonitor::run(84)] DatabaseConfig{type=MySQL, address=127.0.0.1, port=null, name=monitor_db, userName=imc_monitor, password=randomPassword, instanceName=null}
2018-09-07 12:18:47 [INFO ] [iMC-Database-Connect-Check] [com.h3c.imc.deploy.dma.monitor.DatabaseConnectMonitor::run(84)] DatabaseConfig{type=MySQL, address=127.0.0.1, port=null, name=perf_db, userName=imc_perf, password=randomPassword, instanceName=null}
2018-09-07 12:18:47 [INFO ] [iMC-Database-Connect-Check] [com.h3c.imc.deploy.dma.monitor.DatabaseConnectMonitor::run(84)] DatabaseConfig{type=MySQL, address=127.0.0.1, port=null, name=config_db, userName=imc_config, password=randomPassword, instanceName=null}
2018-09-07 12:18:47 [INFO ] [iMC-Database-Connect-Check] [com.h3c.imc.deploy.dma.monitor.DatabaseConnectMonitor::run(84)] DatabaseConfig{type=MySQL, address=127.0.0.1, port=null, name=reportplat_db, userName=reportplat, password=randomPassword, instanceName=null}
2018-09-07 12:18:47 [INFO ] [iMC-Database-Connect-Check] [com.h3c.imc.deploy.dma.monitor.DatabaseConnectMonitor::run(84)] DatabaseConfig{type=MySQL, address=127.0.0.1, port=null, name=report_db, userName=report, password=randomPassword, instanceName=null}
2018-09-07 12:18:47 [ERROR] [iMC-Database-Connect-Check] [com.h3c.imc.deploy.dma.monitor.DatabaseConnectMonitor::run(86)] Connect to database error
java.lang.NullPointerException
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:171)
        at com.h3c.imc.deploy.dma.monitor.DatabaseConnectMonitor.run(DatabaseConnectMonitor.java:86)

 We restarted all services with the stop/start buttons in the monitor tab of the "Intelligent Deployment Monitoring Agent" gui, including the "jserver" process as the web login page suggests.
I can manually connect to the database with "mysql -u imc_aclm -p randomPassword -h 127.0.0.1".
The process is started and deployed aswell with out any error shown.
We also tried a reboot of the host just in case.

Any help on how to move forward with further debugging would be much appreciated.

[NeilR]

I don't think the problem is with your aclm_db, as you say you can connect. Also your log show it connects as well. I looked at my log and i can see that your log shows failure when trying to connect to the syslog_db - it goes in the same order.  I have NTA UAM and WSM besides the base paltform. So next step I'd look at the syslog_db. Maybe use a mysql browser like mysql workbench. See if its there, if you can connect to it.

BTW Here is the full list:

aclm_db
config_db
icc_db
unba_master
unba_slave
invent_db
monitor_db
perf_db
reportplat_db
report_db
syslog_db
ead
vlanm_db
vnm_db
wips_db
wlan_db
EDIT: list pasted twice

[kiml]

Oh that's interesting, does it load or test the modules in that list even if the module is not deployed? As we're not using the syslog module the next in line to be tested should be SCC, if it only tests deployed modules and according to the order of /opt/iMC/common/conf/server-addr.xml that is.

Spoiler

<server-addr>
  <component address="127.0.0.1" id="iMC-ACLM">
    <db-config address="127.0.0.1" dbname="aclm_db" password=randomEncryptedPassword type="MySQL" username="imc_aclm"/>
  </component>
  <component address="127.0.0.1" id="iMC-DM"/>
  <component address="127.0.0.1" id="iMC-ICC">
    <db-config address="127.0.0.1" dbname="icc_db" password=randomEncryptedPassword type="MySQL" username="imc_icc"/>
  </component>
  <component address="127.0.0.1" id="iMC-NETASSET">
    <db-config address="127.0.0.1" dbname="invent_db" password=randomEncryptedPassword type="MySQL" username="imc_inventory"/>
  </component>
  <component address="127.0.0.1" id="iMC-NME-FAULT">
    <db-config address="127.0.0.1" dbname="monitor_db" password=randomEncryptedPassword type="MySQL" username="imc_monitor"/>
  </component>
  <component address="127.0.0.1" id="iMC-NME-PERF">
    <db-config address="127.0.0.1" dbname="perf_db" password=randomEncryptedPassword type="MySQL" username="imc_perf"/>
  </component>
  <component address="127.0.0.1" id="iMC-PLAT">
    <db-config address="127.0.0.1" dbname="config_db" password=randomEncryptedPassword type="MySQL" username="imc_config"/>
  </component>
  <component address="127.0.0.1" id="iMC-REPORT">
    <db-config address="127.0.0.1" dbname="reportplat_db" password=randomEncryptedPassword type="MySQL" username="reportplat"/>
    <db-config address="127.0.0.1" dbname="report_db" password=randomEncryptedPassword type="MySQL" username="report"/>
  </component>
  <component address="127.0.0.1" id="iMC-SCC"/>
  <component address="127.0.0.1" id="iMC-SEPLAT"/>
</server-addr>

As can be seen iMC-SCC lacks some config lines compared to the others, but so does iMC-DM at the top aswell. Comparing with the logfile it seems like it just skips entrys without db config.

The database only contains these dbs, no syslog as it's not deplyed, no scc db nor an "DM" db.

Spoiler

 information_schema
 aclm_db            
 config_db          
 icc_db             
 invent_db          
 monitor_db         
 mysql              
 perf_db            
 performance_schema
 report_db          
 reportplat_db      

[NeilR]

You have modules that I don't have, so your list will be different. I don't think it bothers with things not installed or deployed. Also some dbs have several sub-modules pointing at them

If you are stuck and want to try fixing it, rather than calling support, I'd next try either deploying some of the undeployed modules, Or undeploying then redeploying the suspect modules.  This will only help if this regnerates the configuration at issue.

It will ask if you want to keep the data - if yes then use the backup function to create a backup that you can reload later, but first check and see if the problem is gone.