Constant HP SIM 5.2 Disconnect from External Database

[S.M. Foster]

We have had a very severe ongoing issue for several months now.

We had initially been on a SIM 4X system with a local SQL database.

Since we have transitioned to HP SIM 5.2 we now experience frequent disconnects from our external database.

We had initially thought the issue may be due to using a legacy SQL 2000 database, and have transitioned to a shared SQL 2005 database. Both installations had taken place with full SA rights, and then post install the rights moved down to Database Owner.

Our network traces simply show that ODBC traffic is simply dropped -- no errors. The SQL databases in both instances are shared by many applications, all with consistent connections, except for the HP SIM servers. The SQL servers experience no errors or issues at all.

On theory/item-of-note involves the HP NIC teaming. We are running 32bit Windows 2003 on HP DL380 G4's with a NIC type of HP NC373i Multifunction Gigabit Server Adapter. We team these adapters. It looks like on INITIAL tests that when we have the public network team "Team Type selection" set to "network Fault Tolerance with Preference Order, versus "Automatic" we APPEAR to experiences substantially less database disconnects. To pre-empt the next question, all statistics do not show any instance of failovers or failures. When a database disconnect occurs, more often than not, SIM is not able to recover, and we need to restart all, and do the database fix option, which can and does find table errors to correct.

Is it possible that HP SIM is not resiliant enough to have its database located on an external server?

It is also worth noting that when we transitioned to the new databases, we did install a new instance from the ICE CD rather than attempting to migrate any data, so this is happening for us across the board.

Now for the fun part, log files (with Identifying data X'd out):

Lost connection to database, exception: java.sql.SQLException: I/O Error: Connection reset SQL=SELECT notices.noticeId, devices.overallStatus, associated_device_data.associatedDeviceStatus, case when DeviceStatusValue.StatusValue is null and (devices.ProductType=1 and (DB_DeviceInfoEx.OSType = 'WINNT' or DB_DeviceInfoEx.OSType = 'LINUX')) then 13 else DeviceStatusValue.St

06:45:49,508 INFO [HPSIM_DEBUG] [Panic Logger-0] (Timer-4) Looks like we got a DB disconnect 08S01
06:45:49,508 INFO [HPSIM_DEBUG] [Panic Logger-0] (http-50000-2) Looks like we got a DB disconnect 08S01
06:45:49,508 INFO [HPSIM_DEBUG] [Panic Logger-0] (Timer-4) java.sql.SQLException: I/O Error: Connection reset


1:08:50,561 INFO [HPSIM_DEBUG] [Panic Logger-0] (JBoss Shutdown Hook) shutting down due to SQL error, no db connections returned now
11:08:50,561 INFO [HPSIM_DEBUG] [Panic Logger-0] (JBoss Shutdown Hook) ALERTDB ADDINGALERT 11 Error:java.sql.SQLException: shutting down due to SQL error, no db connections returned now
11:08:50,577 INFO [TomcatDeployer] undeploy, ctxPath=/vse, warUrl=file:/XXXPATHXXX/vse.war/
11:08:52,733 INFO [[/mvcd]] Closing Spring root WebApplicationContext

Lost connection to database, exception: java.sql.SQLException: I/O Error: Connection reset SQL=SELECT notices.noticeId, devices.overallStatus, associated_device_data.associatedDeviceStatus, case when DeviceStatusValue.StatusValue is null and (devices.ProductType=1 and (DB_DeviceInfoEx.OSType = 'WINNT' or DB_DeviceInfoEx.OSType = 'LINUX')) then 13 else DeviceStatusValue.StatusValue end as StatusValue, LockdownStatus.ldStatus, vmmStatus.vmmStatus, ppaStatus.PerfStatus, simAggrStatus.evStatus, devices.Name, devices.ProductTypeStr, ipAddress.ipAddress, devices.productName, DB_DeviceInfoEx.OSName, devices.deviceKey, devices.fullDNSName, associated_device_data.associatedDeviceName, associated_device_data.associatedDeviceType, associated_device_data.associatedDeviceKey, associated_device_data.AssociationTypeNumber, notices.timeStamp, devices.timeStamp FROM devices JOIN notices ON devices.deviceKey = notices.deviceKey and notices.NoticeType = 1 LEFT OUTER JOIN DB_deviceInfoEx ON devices.deviceKey = DB_deviceInfoEx.deviceKey LEFT OUTER JOIN IPAddress ON devices.deviceKey = IPAddress.deviceKey and IPAddress.ipIndex = 0 LEFT OUTER JOIN associated_device_data ON devices.deviceKey = associated_device_data.deviceKey LEFT OUTER JOIN deviceProtocolInfo ON devices.deviceKey = deviceProtocolInfo.deviceKey LEFT OUTER JOIN ppaStatus ON devices.deviceKey = ppaStatus.deviceKey LEFT OUTER JOIN simAggrStatus ON devices.deviceKey = simAggrStatus.deviceKey LEFT OUTER JOIN LockdownStatus ON devices.deviceKey = LockdownStatus.deviceKey LEFT OUTER JOIN vmmStatus ON devices.deviceKey = vmmStatus.deviceKey LEFT OUTER JOIN DeviceStatusValue ON devices.deviceKey = DeviceStatusValue.deviceKey and DeviceStatusValue.StatusSourceId= 6 JOIN consolidatedNodeAuths ON devices.mxGuid = consolidatedNodeAuths.NodeId and consolidatedNodeAuths.userID = 'XXXNUMERICDESIGNATIONXXX' WHERE ( (rtrim(devices.Name) LIKE lower('%XXXDEVICEXXX%') OR rtrim(devices.Name) LIKE lower('% XXXDEVICEXXX %') ) )
@!@,2008-12-03 11:10:19 MST,APPLICATION,FAILURE,START,CMS,ERROR: TABLE hpmxTargetStatus KEY NodeID may contain 49 entrie(s) not in table devices KEY MxGUID,ERROR, XXXSERVERNAMEXXX \mxadmin,,,
ERROR: TABLE hpmxTargetStatus KEY NodeID may contain 49 entrie(s) not in table devices KEY MxGUID
@!@,2008-12-03 11:10:19 MST,APPLICATION,FAILURE,START,CMS,Pass 1,ERROR,XXXSERVERNAMEXXX\mxadmin,,,
Pass 1

------------------------
We know SIM has historically worked using a local database -- is there some conglomeration of issues with NIC teaming, external databases, and SIM 5X stability that is possible causing this issue?

Thank you in advance for any assistance on this issue!

[Rob Buxton]

I can't offer much help, but we do run using an external SQL 2005 database. Bothe HPSIM and the database server are virtual, with just a single NIC. we don't see any disconnect issues.

We also set rules in VMware to try and keep the SIM Server and DB Server on the same host so their NW traffic is over the virtual switch.

Do you need to have two NICs teamed? If possible can you dissolve the team and use a single NIC?
External DBs are supported, I would focus on the NIC / network side first.

[Rancher]

I can't offer much help either, except to say that we have never had any luck with teaming NICs. I agree with Rob that you should start there first.