HPE SimpliVity upgrade manager 3.7.10 "object reference not set to an instance object"

[fahlis]

Hello!
I'm in the middle of upgrading OmniStack 3.7.9->3.7.10. When I try to login with upgrade manager using IP address of vCenter I get this error message.

"object reference not set to an instance object"

Need to solve this a.s.a.p

Anyone who has the solution ?

Br / Tony

[gustenar]

Hello @fahlis , 

Have you tried uninstalling Deployment manager and re-install it? Also, make sure your are opening the software as administrator. What version of Windows are you running Deployment Manager on?

Gus

[fahlis]

Hello @gustenar 

I am upgrading so no use of Deployment manager am I right ?
Or do I need to install the latest Deployment manager prior to ruinning the Upgrade manager ?
Never had to do that previously.
Just noticed some on-prem admin had uninstalled Java 1..8 so I'll try again after reinstalling Java.

Br / Tony

[DaveOb]

Try restaring VC.

If you have an older version of upgrade manager this should be uninstalled.

[gustenar]

@fahlis my bad. I meant to say "Upgrade Manager". 

[fahlis]

@gustenar

No problem, but now I have this other issue. When I choose commit for upgrading OmniStack I get this popup.

"The arbiter version is incompatible with this software"

I started "simplivity-upgrade-manager-3.7.10.249.exe" from admin cmd ,  it resides in same folder as all others from the extracted ISO as it should.
Is there a newer separate version of upgrade manager available separately or what causes this error ?
How can I mitigate it ?
OS on the arbiter server is WS2012 R2 full updated. Java 1.8 , .NET 4.0

Br / TOny

[gustenar]

@fahlis 

What version of Simplivity Arbiter are you running? It must be upgraded to the supported version of the software you are trying to intall prior to installing the software.

[fahlis]

@gustenar
"simplivity-arbiter-3.7.10.91"
And 
"simplivity-upgrade-manager-3.7.10.249"
Which both are included in the OmniStack 3.7.10 ISO
I noticed some leftover files under appdata\local\temp reffering to last upgrade. So i clerared that folder.
Tried again and same thing happened. 
I find this very strange. I suppose it won't work if I try CLI upgrade either ?
If so, could you please provide the correct command to do the CLI upgrade ?

Br/Tony

[DaveOb]

Have you upgraded the arbiter on the machine where the arbiter is installed to 3.7.10 ?

grep arbiter $SVTLOG this should show where the arbiter is installed.

[gustenar]

@fahlis As DaveOb mentioned, you need to make sure the new Arbiter version is installed on the Arbiter server. 

[fahlis]

@DaveOb and @gustenar 

Thanks for your rapid help, I believe I know what's going on now.
I ran "grep arbiter $SVTLOG" on one of the OVC's
Arbiter is connected and looks fine.
However it also showed "vCenter Version:VMware vCenter Server 6.7.0 build 10244857" which is the one installed pre-uprade,that is for OmniStack 3.7.9

One thing I did different this time was to to take an offline snapshot, uninstalled HPE SimpliVity plugin and upgraded vCenter through VAMI.
VAMI now shows Version: 6.7.0.32000 Build number: 14070457 which is correct and supported version according to the interoperability guide.
I have not yet deleted the offline snapshot so I suppose this is why "grep arbiter $SVTLOG" shows wrong version and thus of course wrong version of Arbiter as I have upgraded Arbiter to 3.7.10.91 already.

Before I go ahead and delete the offline snapshot I just wan't your opinions on my assumption that "grep arbiter $SVTLOG" will show the correct version afterwards, and thus the upgrade should work.
Or do I even need to revert the snapshot and upgrade vCenter the usual way.
Is upgrade through VAMI supported with SimpliVity ?

Br / Tony

[gustenar]

.

[fahlis]

@gustenar 
Strange, I got your reply by email but here it shows as empty.

Anyway I checked the service and it shows.

"SimpliVity Arbiter for OmniStack 3.7.10.91 built at 2019-08-17T13:15:06-0400"
And the service is running
What about my question regarding deleting the offline shapshot with hope that it will show the correct running version of vCenter in CLI  with the grep command ?

Br / Tony

[gustenar]

I think I edited the reply by mistake. Sorry about that!

I am not sure if running built not in compatibility matrix will give this type of problems. It really looks related to the Arbiter. 

Try restarting the Arbiter service and try the upgrade again. If problem continues you might need to open a support call. 

[DeclanOR]

Hi @fahlis 

Looking at the conversation, and the direction it is going, I suspect you probably need to do one of two things.

Firstly, maybe you need to repoint the cluster to the arbiter again.

On any OVC in the cluster that will be pointing to the arbiter, run the following command: (Note: Only need to run on one OVC in cluster. It will propogate through to all in the cluster)

sudo dsv-arbiter-instance-set --address <ip address arbiter>

If you are already pointed correctly, you will probably seee a message to say the narbiter is already in use or similar. (Can't recall exact syntax)

Once done, run the grep arbiter $SVTLOG command and confirm that the ip address referenced, is the expected one.

Once complete, attempt the commit again.

If it still fails, maybe due to how you upgraded the VC, you need to update the digitalvault. You will definitely need to update the digitalvault if you changed the VC admin password during upgrade. The command for this is:

sudo dsv-digitalvault-update-vcenter --username <administraor@vsphere.local for example> --password <VC admin password>

You will need to do this on all OVC's in the cluster , and possibly all in your fed managed by that VC if the password did change.

Once this cluster at least is done, attempt the commit again.

If this fails, a full VC repoint may be necessary. Let us know if this helps.

Thanks,

DeclanOR #I am a HPE Employee

[fahlis]

Hi @DeclanOR 

Thanks for your suggestions.

I'll try repointing tomorrow. 

I did not change the VC password during upgrade.

My task progress after upgrading SPP on both hosts without problem.

1..Offline snapshot of the vCenter

2. Uninstalled HPE SimpliVity Plugin 

3. Upgraded vCenter through VAMI

4.Installed the new HPE SimpliVity Plugin

5.Validated and everything good thus far.

6.Upgraded Arbiter and validated the service was running

7.Started upgrade manager from admin cmd

8.Logged in with vCenter IP address

9. Choosed to upgrade only one of the OmniStacks - All VMs apart from OVC migrated to host2 at this stage

10. Validated OK and uploaded the file without problems.

11.Ran commit and it showed the popup for wrong Arbiter version

Output from "grep arbiter $SVTLOG" from the OVC I ran commit on.

Welcome to SimpliVity OmniCube 3.7.9.279

administrator@vsphere@omnicube-ip5-42:~$ grep arbiter $SVTLOG
2019-10-07T06:51:21.962Z WARN 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is not responding
2019-10-07T06:54:54.472Z INFO 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:401 Discovered an arbiter at 172.18.9.6 with id cbbd0d7e-4 91e-4e74-a5f7-161d7f2fc07b
2019-10-07T06:54:54.472Z INFO 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is responding
2019-10-09T10:50:57.891Z WARN 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is not responding
2019-10-09T10:51:07.999Z ERROR 0x7f8208ffd700 [:] [control.phonehome.phonehomemo nitor] phoneHomeMonitor.cpp:2316 Phonehome-Lost contact with the SimpliVity Arbi ter at 172.18.9.6.. Severity Level:PHONE_YELLOW; System Serial Number:CZ3839DJMZ ; Source IP:172.18.5.42; vCenter Version:VMware vCenter Server 6.7.0 build 10244 857; Virtual Controller SW Version:Release 3.7.9.279; Arbiter Version Release Un able to read arbiter version info; Model:HPE SimpliVity 380 Series 4000
2019-10-09T10:54:23.138Z INFO 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:401 Discovered an arbiter at 172.18.9.6 with id cbbd0d7e-4 91e-4e74-a5f7-161d7f2fc07b
2019-10-09T10:54:23.138Z INFO 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is responding
2019-10-09T15:35:43.228Z WARN 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is not responding
2019-10-09T15:45:18.283Z INFO 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:401 Discovered an arbiter at 172.18.9.6 with id cbbd0d7e-4 91e-4e74-a5f7-161d7f2fc07b
2019-10-09T15:45:18.283Z INFO 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is responding
2019-10-10T09:33:08.804Z WARN 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is not responding
2019-10-10T09:36:16.102Z INFO 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:401 Discovered an arbiter at 172.18.9.6 with id cbbd0d7e-4 91e-4e74-a5f7-161d7f2fc07b
2019-10-10T09:36:16.102Z INFO 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is responding
2019-10-19T10:30:13.316Z WARN 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is not responding
2019-10-19T10:30:13.316Z ERROR 0x7f8208ffd700 [:] [control.phonehome.phonehomemo nitor] phoneHomeMonitor.cpp:2316 Phonehome-Lost contact with the SimpliVity Arbi ter at 172.18.9.6.. Severity Level:PHONE_YELLOW; System Serial Number:CZ3839DJMZ ; Source IP:172.18.5.42; vCenter Version:VMware vCenter Server 6.7.0 build 10244 857; Virtual Controller SW Version:Release 3.7.9.279; Arbiter Version Release Un able to read arbiter version info; Model:HPE SimpliVity 380 Series 4000
2019-10-19T10:35:48.369Z INFO 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:401 Discovered an arbiter at 172.18.9.6 with id cbbd0d7e-4 91e-4e74-a5f7-161d7f2fc07b
2019-10-19T10:35:48.369Z INFO 0x7f760eb6b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is responding
2019-10-22T12:01:28.759Z INFO 0x7fd45816b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:401 Discovered an arbiter at 172.18.9.6 with id cbbd0d7e-4 91e-4e74-a5f7-161d7f2fc07b
2019-10-22T12:01:28.784Z INFO 0x7fd45816b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is responding
2019-10-22T12:02:32.784Z WARN 0x7fd45816b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is not responding
2019-10-22T12:02:47.821Z INFO 0x7fd45816b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:401 Discovered an arbiter at 172.18.9.6 with id cbbd0d7e-4 91e-4e74-a5f7-161d7f2fc07b
2019-10-22T12:02:47.826Z INFO 0x7fd45816b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is responding
2019-10-22T13:16:11.847Z WARN 0x7fd45816b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is not responding
2019-10-22T13:20:59.328Z INFO 0x7fd45816b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:401 Discovered an arbiter at 172.18.9.6 with id cbbd0d7e-4 91e-4e74-a5f7-161d7f2fc07b
2019-10-22T13:20:59.331Z INFO 0x7fd45816b700 [:] [control.arb.arbiterdiscovery] arbiterdiscovery.cpp:447 Arbiter is responding
2019-10-22T13:23:46.031Z ERROR 0x7fe04e672700 [:] [control.phonehome.phonehomemo nitor] phoneHomeMonitor.cpp:2316 Phonehome-Software update has failed for versio n 3.7.10.200: This node has an uncommitted upgrade. Severity Level:PHONE_RED; Sy stem Serial Number:CZ3839DJMZ; Source IP:172.18.5.42; vCenter Version:VMware vCe nter Server 6.7.0 build 10244857; Virtual Controller SW Version:Release 3.7.9.27 9; Arbiter Version Release Unable to read arbiter version info; Model:HPE Simpli Vity 380 Series 4000
administrator@vsphere@omnicube-ip5-42:~$

VAMI is showing version 6.7.0.3200 build number 14070457 which I as mentioned find odd due to the output above ":VMware vCenter Server 6.7.0 build 10244857"

Arbiter IP is 172.18.9.6 and judging by above log arbiter does not seems to be in a healthy state ?

As i mentioned it's still running on the offline snapshot.

Do you find anything wrong with my task progress ?

Perhaps I should try to reinstall the arbiter even though it installed-upgraded without issues.

Br / Tony

[DeclanOR]

Hi Tony,

Yes...I think uninstalling the arbiter and re-installing would be a good step. Be sure to remove all old arbiter files also that may not necessarily get removed by uninstalling.

Once new arbiter installed, use the sudo dsv-arbiter-instance-set mentioned previously to repoint to it from any ovc in the cluster. Make sure cluster is healthy before unistalling/reinstalling.

Question: The arbiter IS installed off of the cluster it arbitrates against right?

One other point, in relation to the stepsyou mention:

The commit for an upgrade should only be done when all nodes in a cluster are already upgraded. Are you trying to commit a node whilst others are not yet upgraded? If so, others should be upgraded firstly also.

Thanks,

DeclanOR

[DeclanOR]

Hi Tony,

Only see your comments about the VC snaphot. Are you saying that the VC is still running on a snaphot?

Thanks,

declanOR

[fahlis]

Hi @DeclanOR 

Answers to your latest inputs below.

Regarding the commit before upgrading OmniStack on both hosts, I believe I might by mistake have tried to commit before both where update. I usually as my best practice upgrade them one at a time while all VMs are moved and running on the other host, and as you state run commit once both are upgraded. I will try again just by marking one of the hosts and choose next instead of commit.

And yes VC is still runing on the offline snapshot, I thought best would be to delete-merge it once everyting in done and upgraded , but now I'm pussled that might be the reason for the GREP command shows the previous version and thus it comes in a not supported mode with VC vs. arbiter ?

Arbiter is running outside of the federation, it has been on the same server ever senice initially deployed.. WS2012 R2

The cluster seems healthy according to "svt-federation-show"

Edit: I wen't ahead and tried to upgrade the first host in upgrade manager.

It seems stucked in the state "Upgrade in Progress" "Waiting for upgrade to start". Also in vCenter I can see this alert

Upgrade Datacenter "There is an uncommited upgrade blocking this action"

SimpliVity Upgrade  "There is an uncommited upgrade blocking this action"

After a refresh I got these critical triggered alarms.

"SimpliVity software upgrade failed"

SimpliVity software node upgrade failed"

Cleared those alarms and tried again with the same host,

It showed "Validating upload package" and the "FAILED cluster is in commit state" or something  like that just flashed by.

Tried a third time with same result as the first "stucked in the state "Upgrade in Progress" "Waiting for upgrade to start".

I should also mention that in the "Pre-Upgrade Validation Tests" I get only three blue marked "Skipped" rows. 

None of the green ones whatsoever.

Perhaps I can safly choose "Cancel Upgrade" at this stage and try again?

Wondering if that is what's causing the "There is an uncommited upgrade blocking this action" alerts in vCenter.

Not sure what can be wrong now, so 'll await a reply/suggestions before I do anything more.

Br / Tony

[DeclanOR]

Hi Tony,

I suspect the snapshot may have been the original cause of the issues. It looks as though there may be a stuck task also based on your last input. At this stage, I would recommend opening a support case to allow us to clear any stuck tasks.

Once thats done, i suspect consolidating the snapshot on VC, and attempting upgrade again.

Possibly an uninstall/re-install of arbiter and repoint to the new one also.....but for now, due to need for a clearer view of what's going on, i recommend a support case where we can look at this more clearly.

Thanks for the updates.

DeclanOR  # I am  HPE Employee

[fahlis]

Hi @DeclanOR 

Thanks for your reply.

I will open a case straight away.

Br / Tony

[DeclanOR]

Hi Tony,

Thanks! Also, commit should be the final step...so upgrade all nodes firstly (at least on a per cluster basis), and only then should you commit.

I'm sure support will help resolve this with you.

Thanks,

DeclanOR

[DeclanOR · Accepted solution]

Hi Tony,

Just wanted to follow up and see if you managed to get this resolved today?

Thanks,

DeclanOR

[fahlis]

Hi @DeclanOR 

It has not been resolved yet, I uninstalled-reinstalled arbiter yesterday and ran the sudo command.

All is good and ALIVE on the OVCs

However no one from HPE support has replied today.

I just sent a mail asking if there's any progress yet. See below.

I got a feeling we will need a rollback plan and then try upgrade again.

Can you provide the steps to rollback the upgrade manager current state ?

My idea of tasks if we can accomplish that.

1. Revert vCenter snapshot
2. Take a new offline snapshot
3. Uninstall HPE SimpliVity Plugin
4. Upgrade vCenter through VAMI
5. Install new HPE SimpliVity plugin
6. Try upgrade manager again

Case details FYI

HPE Support Case 5342502284 for HPE OmniStack for DL380 System Components

Maybe you can speed it up

Your reflections ?

Br / Tony

[fahlis]

Hi @DeclanOR 

I have finally managed to upgrade successfully.

I had a remote session with HPE SimpliVity support today.

In short what he did.

svt-federation-show -OK

svt-software-status-show

Then we saw this.

# 3.7.9 - Can Rollback or Commit

# There is an uncommitted upgrade blocking this action

Solution:

# Uninstall arbiter 3.7.10

# Install arbiter 3.7.9

# Run commit

# Exit Upgrade Mgr

# Uppgrade arbiter to 3.7.10

#Run upgrade Mgr again.

It failed the first time with a message in OVC CLI to try again and if it fails again contact HPE support.. So I tried again and this time it worked. Upgraded one host at a time.

# Lastly I upgraded ESXi thorugh CLI

Thx for all your assistant and also to @gustenar 

Next customer upgrade to follow next week, hopefully it will go smoother.

Br / Tony