HPE SimpliVity
1748151 Members
3368 Online
108758 Solutions
New Discussion юеВ

HPE SimpliVity upgrade manager 3.7.10 "object reference not set to an instance object"

 
SOLVED
Go to solution
fahlis
Frequent Advisor

Re: HPE SimpliVity upgrade manager 3.7.10 "object reference not set to an instance object"

@DaveOb and @gustenar 

Thanks for your rapid help, I believe I know what's going on now.
I ran "grep arbiter $SVTLOG" on one of the OVC's
Arbiter is connected and looks fine.
However it also showed "vCenter Version:VMware vCenter Server 6.7.0 build 10244857" which is the one installed pre-uprade,that is for OmniStack 3.7.9

One thing I did different this time was to to take an offline snapshot, uninstalled HPE SimpliVity plugin and upgraded vCenter through VAMI.
VAMI now shows Version: 6.7.0.32000 Build number: 14070457 which is correct and supported version according to the interoperability guide.
I have not yet deleted the offline snapshot so I suppose this is why "grep arbiter $SVTLOG" shows wrong version and thus of course wrong version of Arbiter as I have upgraded Arbiter to 3.7.10.91 already.

Before I go ahead and delete the offline snapshot I just wan't your opinions on my assumption that "grep arbiter $SVTLOG" will show the correct version afterwards, and thus the upgrade should work.
Or do I even need to revert the snapshot and upgrade vCenter the usual way.
Is upgrade through VAMI supported with SimpliVity ?

Br / Tony

gustenar
HPE Pro

Re: HPE SimpliVity upgrade manager 3.7.10 "object reference not set to an instance object"

.

 


I am an HPE employee.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
Accept or Kudo
fahlis
Frequent Advisor

Re: HPE SimpliVity upgrade manager 3.7.10 "object reference not set to an instance object"

@gustenar 
Strange, I got your reply by email but here it shows as empty.

Anyway I checked the service and it shows.

"SimpliVity Arbiter for OmniStack 3.7.10.91 built at 2019-08-17T13:15:06-0400"
And the service is running
What about my question regarding deleting the offline shapshot with hope that it will show the correct running version of vCenter in CLI  with the grep command ?

Br / Tony

gustenar
HPE Pro

Re: HPE SimpliVity upgrade manager 3.7.10 "object reference not set to an instance object"

I think I edited the reply by mistake. Sorry about that!

I am not sure if running built not in compatibility matrix will give this type of problems. It really looks related to the Arbiter. 

Try restarting the Arbiter service and try the upgrade again. If problem continues you might need to open a support call. 


I am an HPE employee.
[Any personal opinions expressed are mine, and not official statements on behalf of Hewlett Packard Enterprise]
Accept or Kudo
DeclanOR
Respected Contributor

Re: HPE SimpliVity upgrade manager 3.7.10 "object reference not set to an instance object"

Hi @fahlis 

Looking at the conversation, and the direction it is going, I suspect you probably need to do one of two things.

Firstly, maybe you need to repoint the cluster to the arbiter again.

On any OVC in the cluster that will be pointing to the arbiter, run the following command: (Note: Only need to run on one OVC in cluster. It will propogate through to all in the cluster)

sudo dsv-arbiter-instance-set --address <ip address arbiter>

If you are already pointed correctly, you will probably seee a message to say the narbiter is already in use or similar. (Can't recall exact syntax)

Once done, run the grep arbiter $SVTLOG command and confirm that the ip address referenced, is the expected one.

Once complete, attempt the commit again.

If it still fails, maybe due to how you upgraded the VC, you need to update the digitalvault. You will definitely need to update the digitalvault if you changed the VC admin password during upgrade. The command for this is:

sudo dsv-digitalvault-update-vcenter --username <administraor@vsphere.local for example> --password <VC admin password>

You will need to do this on all OVC's in the cluster , and possibly all in your fed managed by that VC if the password did change.

Once this cluster at least is done, attempt the commit again.

If this fails, a full VC repoint may be necessary. Let us know if this helps.

Thanks,

DeclanOR #I am a HPE Employee

Accept or Kudo



fahlis
Frequent Advisor

Re: HPE SimpliVity upgrade manager 3.7.10 "object reference not set to an instance object"

Hi @DeclanOR 

Thanks for your suggestions.

I'll try repointing tomorrow. 

I did not change the VC password during upgrade.

My task progress after upgrading SPP on both hosts without problem.

1..Offline snapshot of the vCenter

2. Uninstalled HPE SimpliVity Plugin 

3. Upgraded vCenter through VAMI

4.Installed the new HPE SimpliVity Plugin

5.Validated and everything good thus far.

6.Upgraded Arbiter and validated the service was running

7.Started upgrade manager from admin cmd

8.Logged in with vCenter IP address

9. Choosed to upgrade only one of the OmniStacks - All VMs apart from OVC migrated to host2 at this stage

10. Validated OK and uploaded the file without problems.

11.Ran commit and it showed the popup for wrong Arbiter version

 

Output from "grep arbiter $SVTLOG" from the OVC I ran commit on.

 

Welcome to SimpliVity OmniCube 3.7.9.279

administrator@vsphere@omnicube-ip5-42:~$ grep arbiter $SVTLOG
2019-10-07T06:51:21.962Z WARN 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is not responding
2019-10-07T06:54:54.472Z INFO 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:401 Discovered an arbiter at 172.18.9.6 with id cbbd0d7e-4 91e-4e74-a5f7-161d7f2fc07b
2019-10-07T06:54:54.472Z INFO 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is responding
2019-10-09T10:50:57.891Z WARN 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is not responding
2019-10-09T10:51:07.999Z ERROR 0x7f8208ffd700 [:] [control.phonehome.phonehomemo nitor] phoneHomeMonitor.cpp:2316 Phonehome-Lost contact with the SimpliVity Arbi ter at 172.18.9.6.. Severity Level:PHONE_YELLOW; System Serial Number:CZ3839DJMZ ; Source IP:172.18.5.42; vCenter Version:VMware vCenter Server 6.7.0 build 10244 857; Virtual Controller SW Version:Release 3.7.9.279; Arbiter Version Release Un able to read arbiter version info; Model:HPE SimpliVity 380 Series 4000
2019-10-09T10:54:23.138Z INFO 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:401 Discovered an arbiter at 172.18.9.6 with id cbbd0d7e-4 91e-4e74-a5f7-161d7f2fc07b
2019-10-09T10:54:23.138Z INFO 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is responding
2019-10-09T15:35:43.228Z WARN 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is not responding
2019-10-09T15:45:18.283Z INFO 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:401 Discovered an arbiter at 172.18.9.6 with id cbbd0d7e-4 91e-4e74-a5f7-161d7f2fc07b
2019-10-09T15:45:18.283Z INFO 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is responding
2019-10-10T09:33:08.804Z WARN 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is not responding
2019-10-10T09:36:16.102Z INFO 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:401 Discovered an arbiter at 172.18.9.6 with id cbbd0d7e-4 91e-4e74-a5f7-161d7f2fc07b
2019-10-10T09:36:16.102Z INFO 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is responding
2019-10-19T10:30:13.316Z WARN 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is not responding
2019-10-19T10:30:13.316Z ERROR 0x7f8208ffd700 [:] [control.phonehome.phonehomemo nitor] phoneHomeMonitor.cpp:2316 Phonehome-Lost contact with the SimpliVity Arbi ter at 172.18.9.6.. Severity Level:PHONE_YELLOW; System Serial Number:CZ3839DJMZ ; Source IP:172.18.5.42; vCenter Version:VMware vCenter Server 6.7.0 build 10244 857; Virtual Controller SW Version:Release 3.7.9.279; Arbiter Version Release Un able to read arbiter version info; Model:HPE SimpliVity 380 Series 4000
2019-10-19T10:35:48.369Z INFO 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:401 Discovered an arbiter at 172.18.9.6 with id cbbd0d7e-4 91e-4e74-a5f7-161d7f2fc07b
2019-10-19T10:35:48.369Z INFO 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is responding
2019-10-22T12:01:28.759Z INFO 0x7fd45816b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:401 Discovered an arbiter at 172.18.9.6 with id cbbd0d7e-4 91e-4e74-a5f7-161d7f2fc07b
2019-10-22T12:01:28.784Z INFO 0x7fd45816b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is responding
2019-10-22T12:02:32.784Z WARN 0x7fd45816b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is not responding
2019-10-22T12:02:47.821Z INFO 0x7fd45816b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:401 Discovered an arbiter at 172.18.9.6 with id cbbd0d7e-4 91e-4e74-a5f7-161d7f2fc07b
2019-10-22T12:02:47.826Z INFO 0x7fd45816b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is responding
2019-10-22T13:16:11.847Z WARN 0x7fd45816b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is not responding
2019-10-22T13:20:59.328Z INFO 0x7fd45816b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:401 Discovered an arbiter at 172.18.9.6 with id cbbd0d7e-4 91e-4e74-a5f7-161d7f2fc07b
2019-10-22T13:20:59.331Z INFO 0x7fd45816b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is responding
2019-10-22T13:23:46.031Z ERROR 0x7fe04e672700 [:] [control.phonehome.phonehomemo nitor] phoneHomeMonitor.cpp:2316 Phonehome-Software update has failed for versio n 3.7.10.200: This node has an uncommitted upgrade. Severity Level:PHONE_RED; Sy stem Serial Number:CZ3839DJMZ; Source IP:172.18.5.42; vCenter Version:VMware vCe nter Server 6.7.0 build 10244857; Virtual Controller SW Version:Release 3.7.9.27 9; Arbiter Version Release Unable to read arbiter version info; Model:HPE Simpli Vity 380 Series 4000
administrator@vsphere@omnicube-ip5-42:~$

VAMI is showing version 6.7.0.3200 build number 14070457 which I as mentioned find odd due to the output above ":VMware vCenter Server 6.7.0 build 10244857"

Arbiter IP is 172.18.9.6 and judging by above log arbiter does not seems to be in a healthy state ?

As i mentioned it's still running on the offline snapshot.

Do you find anything wrong with my task progress ?

Perhaps I should try to reinstall the arbiter even though it installed-upgraded without issues.

 

Br / Tony

 

DeclanOR
Respected Contributor

Re: HPE SimpliVity upgrade manager 3.7.10 "object reference not set to an instance object"

Hi Tony,

 

Yes...I think uninstalling the arbiter and re-installing would be a good step. Be sure to remove all old arbiter files also that may not necessarily get removed by uninstalling.

Once new arbiter installed, use the sudo dsv-arbiter-instance-set mentioned previously to repoint to it from any ovc in the cluster. Make sure cluster is healthy before unistalling/reinstalling.

Question: The arbiter IS installed off of the cluster it arbitrates against right?

One other point, in relation to the stepsyou mention:

The commit for an upgrade should only be done when all nodes in a cluster are already upgraded. Are you trying to commit a node whilst others are not yet upgraded? If so, others should be upgraded firstly also.

Thanks,

DeclanOR

Accept or Kudo



DeclanOR
Respected Contributor

Re: HPE SimpliVity upgrade manager 3.7.10 "object reference not set to an instance object"

Hi Tony,

Only see your comments about the VC snaphot. Are you saying that the VC is still running on a snaphot?

Thanks,

declanOR

 

Accept or Kudo



fahlis
Frequent Advisor

Re: HPE SimpliVity upgrade manager 3.7.10 "object reference not set to an instance object"

Hi @DeclanOR 

Answers to your latest inputs below.

Regarding the commit before upgrading OmniStack on both hosts, I believe I might by mistake have tried to commit before both where update. I usually as my best practice upgrade them one at a time while all VMs are moved and running on the other host, and as you state run commit once both are upgraded. I will try again just by marking one of the hosts and choose next instead of commit.

And yes VC is still runing on the offline snapshot, I thought best would be to delete-merge it once everyting in done and upgraded , but now I'm pussled that might be the reason for the GREP command shows the previous version and thus it comes in a not supported mode with VC vs. arbiter ?

Arbiter is running outside of the federation, it has been on the same server ever senice initially deployed.. WS2012 R2

The cluster seems healthy according to "svt-federation-show"

Edit: I wen't ahead and tried to upgrade the first host in upgrade manager.

It seems stucked in the state "Upgrade in Progress" "Waiting for upgrade to start". Also in vCenter I can see this alert

Upgrade Datacenter "There is an uncommited upgrade blocking this action"

SimpliVity Upgrade  "There is an uncommited upgrade blocking this action"

After a refresh I got these critical triggered alarms.

"SimpliVity software upgrade failed"

SimpliVity software node upgrade failed"

Cleared those alarms and tried again with the same host,

It showed "Validating upload package" and the "FAILED cluster is in commit state" or something  like that just flashed by.

Tried a third time with same result as the first "stucked in the state "Upgrade in Progress" "Waiting for upgrade to start".

I should also mention that in the "Pre-Upgrade Validation Tests" I get only three blue marked "Skipped" rows. 

None of the green ones whatsoever.

Perhaps I can safly choose "Cancel Upgrade" at this stage and try again?

Wondering if that is what's causing the "There is an uncommited upgrade blocking this action" alerts in vCenter.

Not sure what can be wrong now, so 'll await a reply/suggestions before I do anything more.

Br / Tony

DeclanOR
Respected Contributor

Re: HPE SimpliVity upgrade manager 3.7.10 "object reference not set to an instance object"

Hi Tony,

I suspect the snapshot may have been the original cause of the issues. It looks as though there may be a stuck task also based on your last input. At this stage, I would recommend opening a support case to allow us to clear any stuck tasks.

Once thats done, i suspect consolidating the snapshot on VC, and attempting upgrade again.

Possibly an uninstall/re-install of arbiter and repoint to the new one also.....but for now, due to need for a clearer view of what's going on, i recommend a support case where we can look at this more clearly.

Thanks for the updates.

DeclanOR  # I am  HPE Employee

Accept or Kudo