Association of Onboard Administrators keeps getting removed

[Matt Sebel]

I'm running SIM 7.2

Why does it keep warning me and then automatically removing the association of my c7000 Onboard Administrators to the enclosure they are in?! Anyone have any ideas?

[sterls27]

what firmware revision are you running?  How often are the alerts being generated?  Is there any scheduled task running (i.e. daily system identification) that corresponds to the time of the alerts?

[Matt Sebel]

OA fw is 3.7

No, no scheduled discovery tasks of any kind.

[sterls27]

How often are the alerts being generated?  Once a day, once an hour, every 5 minutes...?

Shall we assume that the chassis firmware is all within the supported matrix as well?  Anything show in the OA logs that indicates it actually had an issue?

[Matt Sebel]

The alerts come only once, but then the removal happens automatically some time later. I don't have the exact times.

Yep, all fw for the OA and VC modules are at 3.7.

[sterls27]

can you check to make sure it's not the daily identification task that is causing the association to be removed (go to tasks and logs, and view all scheduled tasks)?  I'm having a similar issue with some BL465c G7's and their iLO's, when the daily ident task runs it reports the association removed and then corrects it as it finds the iLO again later...of course, the iLO may show up as a BMC and I believe it is related to the FW version of the iLO and it not responding properly when queried.  Another thing you might want to check is to see if the credentials for the OA in SIM are still showing properly, and try running an ident again after entering in the credentials...that's another issue I'm having where SIM "forgets" the credentials even though they haven't changed in months.  I have to either run the ident task manually or re-enter the credentials for it to be identified properly.  If you need steps on where to find the associated credentials let me know.

Definitely keep the logs for the OA's to see if you can correspond any of the deassociations with any scheduled tasks as it sure seems like that is causing it since there is only the one alert per day...

[Matt Sebel]

Ok, I think you might be right on the Daily System Identification job could be causing this.

I'm running all BL465 G7's as well. What fw version is your iLO adapters at. Most of mine are still at 1.28 but I bumped a few to 1.57 because of self-signed cert issues that I needed to test. I thought maybe that could be causing the issue, but I'm pretty sure it's not as this was happening before I upgraded the iLO fw on a few hosts.

[Matt Sebel]

When today's Daily System Identification job ran, it actually removed the association. Why the heck is it doing this?! I like SIM, but goodness can it be a real pita.

[Princes]

Yep - having exactly the same issue since upgrading from HP SIM 7.0 to HP SIM 7.2 (with all current hotfixes). Issue occurs with all c-Class Blade Enclosure OA's that we use here - all are on firmware v3.71. Looks like it is happening during a Daily System Identification task run, however it does not always happen every time that this task runs. In the Events listed for each OA, I receive an Insight.AssociationCounter: MgmtProcToEnclosure event logged, then 4 minutes later an Insight.AssociationRemoved: MgmtProcToEnclosure event, after which the OA is no longer associated with the enclosure. I have only been able to rectify the issue afterwards by deleting all systems associated with a particular enclosure (switches, OA's, Servers - everything), and then re-discovering the enclosure via the OA. The association does not seem to be able to be restored by running System Identification Tasks against the OA's or any other of the enclosures associated systems.

I remember reading something, possibly in one of the User or Administrator guides for HP SIM, or on one of the Support Forums that system associations are removed by a process within HP SIM that uses a counter awarded during the Identification Task to mark any removed systems as potentially disassociated. It would seem possible to me that this process has some issue or other in HP SIM v7.2, where some systems are being incorrectly identified as removed. Oddly, the OA object in HP SIM seems to retain all other associated system data. The c-Class enclosures can also be viewed for information, however due to the lack of an association with the OA, all of the power, fan, temperature, mezzanine card port mappings and other data is now absent.

The issue is repeatable - I have removed the enclosure and all associated hardware and re-discovered. This seems to restore the association for a number of days (sometime for only a few days, then sometimes a week), before the OA is once again orphaned from its enclosure. Have attached a couple of JPG's showing Events and resulting issue in HP SIM.

[Frode Tviberg]

Hello!

Seems like I have the same problem. Im running the 4.01 version of the firmware and HP SIM 7.2 with all the latest patches...

Anyone fixed this? :)

-Frode

[Princes]

Update: still an issue in HP SIM v7.3 with latest Hotfix pack. Also still an issue with OA v4.22.

Is this going to be looked at HP or have we forgotten HP SIM in the rush to rebrand and re-position the whole HP Management Tools division as OneView.....

[LGentile]

SIM 7.4 is due out probably within a month. Support for SIM is difficult to get, for sure, but I am not sure if OneView is the reason or not. At least there are normal hotfixes coming out, even though this issue is very old.

[Evert Goor]

Also here we see this problem on hp sim 7.3 and the latest OA.

HP support could not really give the right sollution till now.

Did any one already solved this issue ?

[sterls27]

I have noticed that it appears to not be occurring as frequently since I split up the identification tasks.  I created one group for the servers and another group for the chassis components (OA, iLO's, etc.), with them running about 4 hours apart and I haven't lost any associations...unless the iLO was actually not responding.  This is with HP SIM 7.3, latest patches and SQL express.  It would be interesting to know if people affected by this issue are running a full SQL version or the express edition that's included with HP SIM.  

Also, when I have lost associations, usually re-running the discovery task with just the OA IP brings everything back into order...I have not had to delete any systems in order for them to be associated properly.

[Princes]

I'll try splitting up the Identification Tasks as you suggest. Running another Identification or Discovery Task against the OA does not make any difference however. Only deleting the whole enclosure and all of its associations (not a small task either hunting down all the switch modules, iLO's, VM's, blades and rack), and then re-discovering the OA seems to return the enclosure to being fully managed and monitored, a situation that sadly only then last around 4-5 days until the problem re-occurs.

[Rodd Masteller]

HP SIM 7.3 running on Linux RedHat varient (Oracle Enterprise Linux) with Oracle 11g DB on same server as SIM application.  The system has been running for a few months without any systems discovered.  I finally got around to it and put a bunch of systems in on Sunday and Monday of last week (8 - 9 days ago).  Everything was fine, until I modified the Hardware Status Polling for Servers job from the default.

Factory default:

5 minute discovery timeout, 1 retry, run every 5 minutes.

I modified to:

5 minute discovery timeout,  2 retries, run every 20 minutes.

I did this around 9:00am today and around 4:00pm got flooded with Association Removed messages.  But, they are not all from our blade chassis / OA systems.  We have stand-alone rack-mount DL380 G6/G7/G8 systems (no shared storage or anything) in remote facilities that also generated these messages.

This is frustrating that this has been a problem for over a year!!

[Rodd Masteller]

Actually, more than a year:

http://h30499.www3.hp.com/t5/ITRC-HP-Systems-Insight-Manager/BL20p-Association-Removed/td-p/4639317#.VFlGtMk005g

Maybe it is not a bug, but a lack of understanding on how these jobs work, what they do, and how they should be configured?  I just mention that because mine was fine until I changed the job definition.  Plus, I never saw this on the SIM 7.2 environment we had running for a long time, and people here say it had the 'bug'.