HP SIM 5.1 lost connection to DB

[tanvir]

I'm using HP SIM 5.1 with a Postgres DB on Linux. Of late, it has stopped working. That is, I cannot access it via https://192.168.23.25:50000/ anymore.

[tanvir]

I restarted postgresql (/etc/init.d/postgresql restart) and HP SIM (mxstart or /etc/init.d/hpsim start) to no avail.

From /var/log/messages I get:

Nov 25 09:58:34 server mxdomainmgr: Lost connection to database, exception: org.postgresql.util.PSQLException: The backend has broken the connection. Possibly the action you have attempted has caused it to close. SQL=select count(*) from trapData where notice_id not in ( select noticeId from notices) To diagnose PostgreSQL problems, please read PostgreSQL log file: /var/log/pgsql or /var/log/messages If the log file is empty or nonexisting, please see your PostgreSQL administrator.

Nov 25 09:58:39 server postgresql: Starting postgresql service: succeeded

Nov 25 09:58:46 server init: Re-reading inittab

/var/log/pgsql was last updated in 2007. So it's not current.

/var/lib/pgsql/data/postgresql.log I can see â FATAL: terminating connection due to administrator commandâ

My netstat output shows Postgresql is listening on 0.0.0.0:5432 (TCP).

Under /var/opt/mx/logs, the only file that's been updated is: mxdomainmgr.log and this file gives the following INFO messages when I start HP SIM and nothing else.

I tried other commands: mxreport, mxlog, mxuser -l t & they all give me the following:

There was a problem connecting to the HP Systems Insight Manager server. Make sure that:

1. Your username has been added to HP Systems Insight Manager.

2. Your username and password, if specified, are correctly spelled.

3. HP Systems Insight Manager is running.

4. You used '--' for any long options and double quotes if your username includes a domain.

Example: --user "mydomain\myusername" --pass mypassword

/var/opt/mx/logs/ mxdomainmgr.log output when I start HP SIM:

09:58:48,025 INFO [Server] Starting JBoss (MX MicroKernel)...

09:58:48,027 INFO [Server] Release ID: JBoss [WonderLand] 3.2.7 (build: CVSTag=JBoss_3_2_7 date=200501280217)

09:58:48,027 INFO [Server] Home Dir: /opt/mx/jboss

09:58:48,028 INFO [Server] Home URL: file:/opt/mx/jboss/

09:58:48,028 INFO [Server] Library URL: file:/opt/mx/jboss/lib/

09:58:48,029 INFO [Server] Patch URL: null

09:58:48,029 INFO [Server] Server Name: hpsim

09:58:48,030 INFO [Server] Server Home Dir: /opt/mx/jboss/server/hpsim

09:58:48,030 INFO [Server] Server Home URL: file:/opt/mx/jboss/server/hpsim/

09:58:48,030 INFO [Server] Server Data Dir: /opt/mx/jboss/server/hpsim/data

09:58:48,031 INFO [Server] Server Temp Dir: /opt/mx/jboss/server/hpsim/tmp

09:58:48,064 INFO [Server] Server Config URL: file:/opt/mx/jboss/server/hpsim/conf/

09:58:48,064 INFO [Server] Server Library URL: file:/opt/mx/jboss/server/hpsim/lib/

09:58:48,065 INFO [Server] Root Deployment Filename: jboss-service.xml

09:58:48,068 INFO [Server] Starting General Purpose Architecture (GPA)...

09:58:48,402 INFO [ServerInfo] Java version: 1.5.0_08,Sun Microsystems Inc.

09:58:48,403 INFO [ServerInfo] Java VM: Java HotSpot(TM) Server VM 1.5.0_08-b03,Sun Microsystems Inc.

09:58:48,403 INFO [ServerInfo] OS-System: Linux 2.6.9-55.0.2.ELsmp,i386

09:58:48,732 INFO [Server] Core system initialized

09:58:52,250 INFO [Log4jService$URLWatchTimerTask] Configuring from URL: resource:log4j.xml

25 Nov 2008 09:58:52,337 INFO [WebService] Using RMI server codebase: http://server.domain.com:50013/

25 Nov 2008 09:58:52,684 INFO [NamingService] Started jndi bootstrap jnpPort=50009, rmiPort=50008, backlog=50, bindAddress=/0.0.0.0, Client SocketFactory=null, Server SocketFactory=org.jboss.net.sockets.DefaultSocketFactory@ad093076

25 Nov 2008 09:59:01,757 INFO [nodeDiscoveryComplete] Bound to JNDI name: topic/nodeDiscoveryComplete

25 Nov 2008 09:59:01,760 INFO [hpsim/nodeDatacollectionComplete] Bound to JNDI name: topic/hpsim/nodeDatacollectionComplete

25 Nov 2008 09:59:01,763 INFO [hpsim/core/users] Bound to JNDI name: topic/hpsim/core/users

25 Nov 2008 09:59:01,765 INFO [hpsim/core/node] Bound to JNDI name: topic/hpsim/core/node

25 Nov 2008 09:59:01,766 INFO [nodeCredentialChange] Bound to JNDI name: topic/nodeCredentialChange

25 Nov 2008 09:59:01,768 INFO [lifecycleindicationhealth] Bound to JNDI name: topic/lifecycleindicationhealth

25 Nov 2008 09:59:01,770 INFO [lifecycleindicationconfig] Bound to JNDI name: topic/lifecycleindicationconfig

25 Nov 2008 09:59:03,605 INFO [HPSIM_DEBUG] Checking for possible database corruption...

Also, /var/opt/mx/output is about 169MB in size with thousands of *.out files (not sure whether that's normal or not)