HPSim 5.2 Linux server, web page not responding

[stephen ashley_1]

I manage a HPsim 5.2 server running on Linux.

We have about 3800 systems known to sim, using an oracle db on another dedicated (for sim) server which is also linux (both 32bit OS’s).

The Sim server hardware is a DL380 G4, with two dual core 3.6Ghx cpus, and 4Gb of memory.
The oracle server hardware is an old DL580 G2, with four dual core 1.6Ghz cpus and 8Gb of memory.

The sim server has been unstable for the past week, we had an ILO port spew out 35000 snmp traps over an hour or so. The oracle server’s recovery log space was consumed and the db went off line. After fixing the db and stopping and restarting the sim server its be ‘broken’ in some way since this event.



I get poor response (slow) from the web browser accessing sim, and after some time the web frames, text (all content) cease to be displayed in the browser window. Closing and restarting the browser will (after the logging time out has expired) re-display the sim login web page, and I can login, but nothing is displayed in the browser (ie is blank) this happen if you user access is still valid too.
I have tried to flush the browser cache, etc but has not fixed this.

The mx audit log file shows the login (root in this case) as successful and is active with SUCCESS, SUMMARY and DISCOVERY etc messages being logged. I don’t see any sim message logged in the system messages file when tailing that when I see the blank browser. I know if I mxstop and mxstart sim will get going again, but will fail some time (1 to 12 hours) later on.

The system message file has had mxdonainmgr error messages, some look like database error, although I can use sql client to access the oracle db (from the sim server) at anytime (ie my dbcheck script displays the system table) and I know that oracle is responding.

These errors appear as sim is re-starting.
(some date and time) systemname Mxdomainmgr: ERROR: TABLE hpmxTargetStatus KEY NodeID may contain 1253 entrie(s) not in table devices KEY MxGUID
(some date and time) systemname Mxdomainmgr: ERROR: TABLE DB_DeviceCpu KEY devicekey may contain 2 entrie(s) not in table devices KEY deviceKey
(some date and time) systemname Mxdomainmgr: Pass 1
(some date and time) systemname Mxdomainmgr: 1253 entries fixed in TABLE hpmxTargetStatus
(some date and time) systemname Mxdomainmgr: 2 entries fixed in TABLE DB_DeviceCpu

I also see this error some time when sim stops working.
(some date and time) systemname Mxdomainmgr: Audit Log queue size exceeded, audit logging paused, verity mxdtf is running.

I would like to know if there is some way to start mxdomainmgr in debug more, or how to get more info on what is failing. I’ve looking in the hpsim tec troubleshooting 5.2 pdf, also the install guides and search on line (inc these forums). But so far not found info to help.

Any suggestions or help accepted.

Regards,
Stephen Ashley,
Alice Springs,
Northing Territory,
Australia.