Transforming IT
cancel
Showing results for 
Search instead for 
Did you mean: 

DPTIPS: Getting DP GRE to work with VMware 6 and vCenter Server Appliance

Jim Turner

Booth.jpgHewlett Packard Enterprise is chock full of sharp-eyed techies who work tirelessly to ferret-out answers to vexations caused by an ever-shifting landscape of different vendor solutions.  VMware's move to an appliance-based vCenter Server presents one such challenge, and I know just the man for the job.

It is my distinct pleasure to introduce Mr. David Booth of Adelaide, Australia.  David is an HPE Account Support Manager who works heavily with Data Protector and was particularly tenacious in his recent DP VMware GRE investigation .  If you are trying to configure DP's VMware GRE with a vCenter 6 Server Appliance, his information will be immensely helpful.

DP-GRE
DP’s ‘GRE’ is a licensed add-on feature that allows individual files to be recovered from ESX-hosted VMDK ‘image’ backups, for both Windows and Linux guest servers.

GRE consists of two component add-ons that must be installed like any DP client:

  • The GRE ‘agent’ that run’s on a windows/linux ‘proxy’ server:

‘proxy’ in this sense meaning a server with enough disc space to contain the recovered VMDK image and ‘mount’ it to inspect the internal contents

VMwareGRE01.jpg

  • The GRE ‘web-plugin’ for the vcenter-UI that adds the necessary web-pages to initiate, monitor and recover the files from the vcenter UI

VMwareGRE02.jpg

Some Background to the DP-TIP note:
IN line with an approach of many vendors, VMWARE have ‘applianced’ the v-center utility.

Previous generation vcenter was ‘another windows application’ to install onto a suitable windows host. DP had to be install into windows as a standard client, then the GRE layer added.

AS of VMWARE 6.x, the vcenter is now a self-contained virtual application, accessed via a browser, that run’s as ‘just another VM’, in a similar way to some HPE pre-canned VM’s, eg VSA, OneView, RSM etc.

The advantage of this approach is the utility run’s as a ‘black-box’, built and maintained by the vendor. The only typical user configuration is hostname, ip_address, DNS, and NTP.

By ‘locking out’ access to the innards, it can keep a lot of the ‘das-gefingerpoken und mittengrabben’ potential side-effects at bay.

The Problem for GRE
As it comes out of the wrapper, DP-GRE, though it will install the v-center web-plugin appliance, it won’t function, till some ‘gefingerpoken und mittengrabben’ is applied to the v-center underlying Linux.

This is because the v-center appliance engine – is running a ‘VMWARE’ internal firewall that blocks most – INCLUDING GRE requests from the DP-GRE-proxy.

Fortunately, as will be covered later, this is not difficult to adjust, as will be shown shortly. Some modest Linux editing skills are needed.

The visible symptom of an un-modified v-center is when a request is made in the UI, the UI hour-wait-circle will spin forever, possibly with a box stating ‘Loading’ and the requests list never populates.

VMwareGRE03.jpg

The flow of events behind the scenes leading up to the wait-spin is, using the v-center GUI:

  • Select target VM needing restore
  • V-center contacts DP-GRE-proxy to ask for list of backups – which in turn contact the DP cell-manager – then ‘pause-waits’ on the response
  • The proxy having collected the backup history for the VM then contacts the v-center server, but the tcp ports are blocked, so the request never makes it back into the GRE-web-plug in, which then wait’s forever

Log analysis of the so-called ‘virgo’ logs in the v-center show it waiting, waiting, waiting for the DP session data.

Log analysis from the DP-GRE-proxy server show the connections into the v-center are denied.

The ‘Fix’
Open the internal v-center firewall ports used by the GRE-proxy-agent that has to connect back to the v-center to send the session information data and status reports.

This open-firewall is achieved though editing the file /etc/vmware/appliance/services.conf and a tweak edit in the /etc/hosts file.

The Tech Details
The v-center appliance VM is internally a base of SuSE 12 Linux, with all the necessary v-center smarts loaded into a web-server engine.

The appliance is ‘protected’ to a degree, using a block-unless-permitted firewall ruleset, loaded into the Linux iptables().

Usual v-center activity is through a web interface to manage the VM’s and server/storage resources, but, like any Linux server, it is possible to ssh() into the underlying Linux and ‘gefingerpoken und mittengrabben’ – which is what must be done – edit two text files in the vcenter.

NOTE that ssh() access must be enabled – accomplished on the CLI console post install and boot.

Once ssh() root login to the v-center Linux, inspecting the output from iptables –l will show the firewall rules, which by default include blocking the ports needed by GRE.

NOTE that VMWARE for probably excellent reasons do not use the ‘native’ SuSE Linux firewall control files, so don’t edit those files; that will have no effect on the active ruleset. VMWARE have a ‘parallel’ set of files for the firewall and many other vital components in the /etc/vmware directory.

The two files to edit, then reboot the v-center linux to activate, are:

File 1:  /etc/hosts

  • Swap the two lines so the network ip address comes before the localhost entry.
  • IF there is no DNS, add in FQDN’s for the CM and the DP-GRE-proxy.

File 2:  /etc/vmware/appliance/services.conf

  • Add rules to permit the GRE ports, per the example in the appendix.
  • Ports 7116 and 15000-15999 should be sufficient, but if this is still does not fully resolve, for testing purposes, open all ports.
  • Not a lot of ‘experimenting’ was done, but from behaviour noted in response to the editing it appears that formatting, including using spaces rather than tabs, seems to be significant. IF there appears to be issues, check the formatting carefully.

Review of DP-GRE Install Process
Presuming all the core components are in place: CM, Web-GRE-Proxy (could be CM), ESX farm, then a brief summary of the steps to get the GRE operational:

1. Use DP to install the GRE-Web-Plug in on the GRE proxy as a standard ‘add component’.

2. Ensure the GRE-proxy server has a large chunk of disc space for VMDK image restores. NOTE the directory path of the space.

3. Install the vcenter appliance, ensure the DNS lookups, forwards and backwards are consistent across the CM, Proxy and the vcenter itself. Check the v-center functionality through a browser.

4. Log into the vcenter linux as root, and edit the two files as noted above. Restart v-center and allow 15 minutes. It takes a while for vcenter to fully bring on line all its background tasks.

5. In the DP CM, ‘import’ the v-center as a vcenter-host. Include the credentials.

6. Verify the credentials by creating a test backup and backup one or several of the VM’s managed by the v-center. [ Aside, if the transport is not using SAN, potential performance gains are possible – outside the scope of this TIPS note. Ask your friendly HPE ASM for help. ].

7. In the DP-CM client screen, check the box for the ‘GRE plug in’ – click the OK and note the ‘install successful’ dialog box.

8. In the v-center web interface, select a VM.

9, Select ‘Manage’.

10. Select “HPE GRE Extension”

11. Select the ‘cog-wheel’ and edit GRE settings, ensuring the proxy hostname is right, and set the directory restore path identified in step 2.

VMwareGRE04.jpg

12. Select a new restore session, wait for the list to populate (about 10 seconds), nominate a SCSI disc, and ‘submit’.

13. In DP GUI, ‘monitor’ the restore session.

14. When finished, in the GRE, expand the list of directories and files and restore as needed.

VMwareGRE05.jpg

Final Point
This note does NOT cover any additional works that might be required on actual in-situ network firewalls. The presumption in all the above is that the ‘raw’ network interconnecting infrastructure used to transport packets bouncing between CM, gre-proxy, v-center, D2Ds (if cat-stores are used), file-stores, DNS, AD’s, LDAP’s etc etc is indeed ‘transparent’ in a network passing sense. IE a straight-forward switched/routed network, without their own firewalls and vlans.

IF the network infrastructure is not transparent, this will be need its own remediation steps.

Perhaps encourage the network team to add a “src=any, dest=any, port=any” rule, so that the lan network functions in a manner like the transparent network as it was originally designed.

Acknowledgement
Thank you to tireless assistance from DP L2 and L3 support, identifying a DP config error, providing crucial clues of DP and ‘virgo’ log analysis.

Also to the Mighty Mr T for assisting with authoring a DP-TIPS note and most of all to Mark at our Customer with infinite patience as we unraveled the symptoms and developed a fix/workaround.

Hopefully, in the background, the DP Lab and the VMWARE Engineers are discussing a more permanent and less intrusive long-term-fix.

Appendix
NOTES:

The first few entries have been removed from the below for clarity on this printout. DO NOT remove them from the actual file.

The black text (sans initial ***comment***) is the content ‘as shipped’ in the conf file. DO not change.

The blue text is the added rules that _should_ suffice to permit GRE functionality. Use this in preference.

The red text is the ‘emergency break glass’ open everything, if it’s still not playing ball with GRE.

There are no tabs used (unlike ‘normal’ Linux text files) – spaces only

/etc/vmware/appliance/services.conf

{
*********START OF FILE OMITTED FOR BREVITY – DO NOT CHANGE  ***********
  "vsphere-client": {
    "firewall": {
      "enable": true,
      "rules": [
        {
           "direction": "inbound",
           "protocol": "tcp",
           "porttype": "dst",
           "port": "9443",
           "portoffset": 0
        },
        {
           "direction": "inbound",
           "name": "Appserver",
           "port": "7116",
           "porttype": "dst",
           "protocol": "tcp"
        },
        {
           "direction": "inbound",
           "name": "GRE",
           "port": "15000:15999",
           "porttype": "dst",
           "protocol": "tcp"
        },
        {
           "direction": "inbound",
           "name": "TEST-DP-TCP",
           "port": "0:65535",
           "porttype": "dst",
           "protocol": "tcp"
        },
        {
           "direction": "inbound",
           "name": "TEST-DP-UDP",
           "port": "0:65535",
           "porttype": "dst",
           "protocol": "udp"
        }
      ]
    }
  }
}

Mr_T here again.  My thanks to David for his dilligence in getting this sorted as well as documenting everything.  We hope the information here proves useful to many.

About the Author

Jim Turner

Jim is a multi-disciplined engineering professional with 32 years of electronic and systems experience. For the past 19 years, Jim's primary focus has been enterprise backup, recovery, and archiving (BURA). As an HPE Master Technologist and Veeam Certified Engineer, he is recognized as a global authority on data availability and protection. Jim's consulting has stretched over 425k miles and 156 unique locations in North America during his 10 years with HP(E). When not traveling, Jim resides in Edmond, OK with his wife and three dogs.

Comments
DrFriday

Aloha Jim,

  A good article to resolve the HPE GRE problems running on Linux and using Vmware Vcenter appliance.  Unfortunately this article does not solve a similar problem I've come up against running the HPE GRE on a Vmware Vcenter 6.02 running on Microsoft Server 2012 R2.  

We are utilizing HP DP 9.0 update bundle 906.  Our problem began when we updated both the Vmware environment from 5.1 U2 to 6.0 U2 and created all new VM MS server 2012 R2.  Deleted the original Vcenter from Cell Manager, imported the new Vcenter MS 2012 R2 checked off the Advanced HPE GRE and that all worked fine.  Ran all my VMDK full backups, no worries.  Cell Manager pushed out the HPE GRE plug in to the new Vcenter.  OK, try to do a restore and get an error message :

Log on to virtual (OLD DELETED VCENTER NAME) could not be peformed. Details: Connection information missing. Check Vmware Vcenter host has been imported to Data Protector Cell.

So this error message is telling me the HPE GRE plug-in is still locking for the deleted OLD Vcenter, as we already know the new Vcenter was imported and works just fine to perform the VMDK backups.

We do not use host files, only DNS of which we have two servers.

Any ideas?  It has been 3 days waiting on HP Software support for HP DP out of Costa Rica!

Thank you,  Respectfully,

 

Wee Kiong Tan

Would editing the vCSA conf file void VMware support as understand support does not cover modification of vApp.

 

Is this a certified method by VMware?

T. Cooper

FYI - I just stood up a VCSA 6.5 vm and I was able to add the appliance to Data Protector and Install GRE recovery out of the box.

Labels
Events
Nov 27 - 29
Madrid, Spain
HPE Discover 2018 Madrid
Learn about all things HPE Discover 2018 in Madrid, Spain, 27 - 29 November, 2018.
Read more
See posts for
dates/locations
HPE at 2018 Technology Events
Learn about the technology events where Hewlett Packard Enterprise will have a presence in 2018.
Read more
View all