EVA and Continuous access

[Dips123]

Hello all

I am new to this forum so apologies if I make any errors. I am at my wits end with issues that I am having with trying to replicate between two EVAs using continuous access. The scenario is as follows:

The customer has two sites, one production and one on DR. In each site there is an EVA. Each site has two switches, switches 1of the two sites are connected with each other using CWDM. The same applies for swithces 2 . I am trying to get the EVAs to replicate from the production to DR using continuous access over the CWDM link. My issues are as follows:

• I have zoned the controllers of the production EVA to the other EVA. However when I use the discover storage from the command view I cannot find the second EVA. The continuous access license has been applied already. Why is this happening?
• Do I need Replications Solutions Manager (RSM) to configure and start the replication or will the Command View software suffice?
• Lastly, do I need one RSM installation each on each site, i.e. the production and the DR or will installing RSM on one server which has got access to both the EVAs will suffice?

I found in the Continuous Access implementation guide the following:

• NOTE: The following configuration must be used in an HP P6000 Continuous Access environment for all Fibre Channel switches in the path from the source array to the destination array. This includes FCIP routers that are connected to the fabric.

Execute the following commands to establish the required switch settings:

1. switchdisable

2. aptpolicy 1 (enable port-based routing)

3. dlsreset (disable DLS)

4. iodset (enable in-order delivery)

5. switchenable

I have done the following but still no success. 

Any help will be greatly appreciated.

Thanks

[Johan Guldmyr]

Hi, about the routing policy. 

http://h20000.www2.hp.com/bizsupport/TechSupport/DocumentIndex.jsp?lang=en&cc=us&contentType=SupportManual&prodTypeId=18964&prodSeriesId=471572

What you read is only for "HP-FC". If you instead use "HP SCSI-FCP" you can use exchange-based routing policy.

See output of "aptpolicy" command on the switches.

Personally I like exchange-based routing as it gives a better utilization of links, but perhaps your CWDM solution wants port-based.

[giladzzz]

Hi

To be able to create CA the two EVA's have to see each other (controllers in the same zone) also Command View has to manage both EVA's this means the command view server has to be zoned with both EVA's. RSM is not needed this is just a scripting software but it's not a "must".

After you see both EVA's in command view you can create CA . Check the data replication properties are HP-FC (preferred) or  "HP SCSI-FCP"

look at the CA implantation guide

http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c02838340/c02838340.pdf

Regards

Press the kudos star if you liked

[Dips123]

Hi Johan

Thanks for your response. However I don't quite understand the difference. I ran the commands because that's what the document said to do. I ran the command you suggested and please find the output below.

admin> aptpolicy
Current Policy: 1 0(ap)

3 0(ap) : Default Policy
1: Port Based Routing Policy
3: Exchange Based Routing Policy
0: AP Shared Link Policy
1: AP Dedicated Link Policy

The most disappointing thing about the HP documentation is that it says that to start the Continuous Access, it has to be made sure that the controllers see each other. But it doesn't say anything about how to achieve this.
One other thing I wanted to mention is that the existing EVA in the production site has already got data being written to it by the hosts. So in this scenario, does the property of controller port on the EVA of the production site have to be changed from a target to initiator for the replication? I don't have the EVA controller zoned with the management server. Is that required for the replication to start?
As you may understand I didn't realise that this will be so complicated and the amount of information out there will be so limited. So any help is much appreciated.

Regards

[Johan Guldmyr]

Hi Dips :)

Replication is a bit complicated.

So currently (on one switch, you need to set this on each switch) you are running port based exchange policy.

Then you need to set to use the "HP-FC" protocol in CV.

In the CA implementation guide it says that "you must use.." - but this is if you have "HP-FC" protocol configured.

It says further down in the CA implementation guide that you get better performance with aptpolicy 3 and iod disabled. And if you use "HP SCSI-FCP".

But, it also requires 09520xxxx or highe rfirmware (so not for EVA8100 or older).

To get controllers to see eachother you need to zone the host ports. It is common/advised to dedicate  the host port(s) in each EVA for replication. 

You need to zone the CV servers with the EVAs or you will not be able to manage them. You haven't written which type of EVA you have but you cannot do CA from the management module. For redundancy you probably also want to have a CV server on the remote site. Or if the primary site goes down how will you manage the backup site?

Also from the CA implem guide:

"Management servers must be in the same zones as the local and remote array host ports that are
used for the intersite links."

And no, it's not enough to "just" change a port into a different mode. You also need to set up DR groups, chose replication mode (sync, async) and other things.

I would like to suggest that you set up a test DR group, vdisk and servers - before you start doing replication for production. This way you can see how servers react to failover, ho wto failover, how to fail back, etc.

You also want to do performance monitoring. Do learn how to use evaperf or HP performance analyzer. Also set up monitoring of the CWDM equipment, so you can see if they are being maxed out, high latency or things like that. 

---

I don't think information is limited, it may be hard to find it though.

For additional reading I would advice you to read the "HP SAN Design Reference Guide". There is a section dedicated to Continuous Access.

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00403562/c00403562.pdf

Also the P6xxx user guide has some information. http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c02864791/c02864791.pdf

CV User guide has info too: 

http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c03375092/c03375092.pdf

Also the Replication Manager document has some information about CA: 

http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c02838342/c02838342.pdf

VMWare and CA best practices:

http://h20195.www2.hp.com/V2/GetPDF.aspx/4AA2-8848ENW.pdf

(There are more best practices around).

[Dips123]

Hi Johan

Thanks so much once again for your response and your suggestions.

I understand the concept of the CA at a high level but the issue that I am facing is that I can't seem to find the correct guidelines for the configuration. At this point of time,  the zoning is like this. The management server is connected to the SAN switches 1 and 2 on the production site only. Controller 1 port 1 of production EVA, controller 1 port 1 of DR EVA and port 1 of the HBA on the management server are in one zone on switch 1 at the production site. Similarly on switch 2, Controller 2 port 2 of the production EVA, controller 2 port 2 of the DR EVA and port 2 of the HBA on the management server is on another zone. The controller and the management server HBA ports that have been zoned are also being used by other devices and are members of other zones. For example, the controller port on the production EVA that has been zoned in the replication zone, is also used by the exchange server. The HBA port on the management server that has been zoned is also used to write to the tape library. There is no reference in the HP documentation on this however I do think that this is in line with the best practice. Is this something that can be a cause for the issue? I have been told by a friend of mine who is a storage expert that, if an EVA port is the target for a fibre connection then that same port cannot be an initiator for another connection.  Logically, to me, that doesn't sound right. Even if it was, I can't find any option in the CV to change the property of the port from initiator to target. Is that something that needs to be looked into?

Please bear in mind that only the management server in the production server has HBAs connected to the switches in the production site. The management server in the DR site does not have a fibre connection to the switch.

The most disappointing thing is when I run a discovery from the CV it says that the discovery is a success, however no new EVA is discovered. I know that there is both IP and FC connectivity between the two sites. Everything seems to be correct but it still isn't working. You can understand my frustration.

I have inherited this project and I wasn't involved in the architecture of this. As a result, I am trying to plug the holes. 

Thanks once again for your continuous support on this. I thank you from the bottom of my heart.

Regards,

[Johan Guldmyr]

No problem dips :)

Your zoning sounds OK as the CV server's ports are zoned to both controllers in both EVAs (which is good because CV only speaks with the master controller and this can change). And the EVAs are zoned.

From CA implementation guide:

"For low-bandwidth intersite links, HP recommends using separate array ports for host I/O and
replication I/O." 

Which DR protocol do you have configured now and which routing policy is configured on the switches?

Which EVAs models do you have, which firmware is installed on each of them and which CV versions on the management server and possible management modules?

Are these models and firmwares compatible?

"I know that there is both IP and FC connectivity between the two sites." :  I'm interested in how do you know that you have FC connectivity?

Also from CA implementation guide:

"HP P6000 Command View 9.0 or later enables the monitoring and setting of controller host ports

preferences for data replication traffic. This functionality is supported on XCS 6.xxx or later and
XCS 09xxxxxx or later. Host port replication settings are managed from the Data Replication Folder
Properties page within HP P6000 Command View. See Figure 20 (page 57)."

"Source and destination arrays must have remote replication licenses." - Do you have this?

[Dips123]

Hi Johan

Thanks for your response once again. Please find the answers to your questions below:

Which DR protocol do you have configured now and which routing policy is configured on the switches?I don't know to be honest. I didn't change anything on the switch other than the commands that I shared on my previous post. If you tell me I can find that for you.

Which EVAs models do you have, which firmware is installed on each of them and which CV versions on the management server and possible management modules? P6300 and P6500. The CV versions are 10.1.0.120411 and 10.0.0.110909

Are these models and firmwares compatible? I can check the documentation again, but I am pretty sure that they are. 

"I know that there is both IP and FC connectivity between the two sites." :  I'm interested in how do you know that you have FC connectivity? I know that the IP connectivity exists is because the management server on the production site is on the same subnet on the production equipment. From the management server I can access the switches, CV web console etc. From a FC point of view, the CWDM links are working fine and as a result the ISL has been created. That is what I meant by the connectivity is fine between the two sites.

Once again thank you so much for your help. I really appreciate it. Please let me know what you think.

Regards,

[Johan Guldmyr]

Hi,

on all switches, is it output of aptpolicy 0?
If so, you need to run HP-FC., you set this under the replication settings in CV on the CV server.
What is the firmware XCS on the two systems?
You gave two CV numbers is it 10.1 on both management modules and 10.0 on the CV server?

OK. You also should confirm FC connectivity between the management server and the EVA that it cannot see.
Another way to check connectivity is to use the tool "cvutil". It should come with the CV installation.
You can also use "nscamshow" and "nsshow" on the switches.
Make sure zoning is accurate.

[Dips123]

Hi Johan

Many many thanks again and please find the answers to your questions below:

• on all switches, is it output of aptpolicy 0? The following is the output of aptpolicy. 

  Current Policy: 1 0(ap)

  3 0(ap) : Default Policy
  1: Port Based Routing Policy
  3: Exchange Based Routing Policy
  0: AP Shared Link Policy 
  1: AP Dedicated Link Policy

• If so, you need to run HP-FC., you set this under the replication settings in CV on the CV server. From the above output, isn't the aptpolicy 1 and not 0? If it is and please pardon my ignorance, should the replication setting be still HP-FC?
• What is the firmware XCS on the two systems? 

DR

Version:

10001000 

Build:

CR1F1Blesp-10001000 

Prod

Version:

11001000 

Build:

CR21A4lep-11001000 

• You gave two CV numbers is it 10.1 on both management modules and 10.0 on the CV server? I am not quite sure as to what is the CV server. I go to the CV gui by directing the browser to <IP address>:2374. The CV version that I provided was the one that was on the login page that is presented when I go to that URL.

• OK. You also should confirm FC connectivity between the management server and the EVA that it cannot see. The management server is connected to the switch in the production site. From the same server I can go to the two different CV URLs and can manage the two EVAs from two different browser windows. The HBA of the mgmt server is connected to the switches in the Prod site, the WWPN can be seen on the switches and the ports are zoned to make use of the tape library at the moment. So I think the physical connectivity is fine. 
• Another way to check connectivity is to use the tool "cvutil". It should come with the CV installation.You can also use "nscamshow" and "nsshow" on the switches. Make sure zoning is accurate. I ran the command, but wasn't sure as to what I was looking for. I have attached the file. I'd be really grateful if you can find some time out to point me towards any obvious mistake that I might be making.

I am running out of good things to say to you. I am really grateful for your support. Thanks again.

Regards,

Dipan

[Johan Guldmyr · Accepted solution]

Hi!

"If so, you need to run HP-FC., you set this under the replication settings in CV on the CV server. From the above output, isn't the aptpolicy 1 and not 0? If it is and please pardon my ignorance, should the replication setting be still HP-FC?"

Sorry, That was a typo from me. If it's "0" or port-based routing you need to use HP-FC.

"You gave two CV numbers is it 10.1 on both management modules and 10.0 on the CV server? I am not quite sure as to what is the CV server. I go to the CV gui by directing the browser to <IP address>:2374. The CV version that I provided was the one that was on the login page that is presented when I go to that URL."

1. It sounds to me like you actually do not have Command View installed on the management server, but that you instead from the management server use a web browser to connect to the management modules on the two EVAs.

The management modules have an IP address and they are actually a module in the controller enclosure of the P6000. One definite problem and another potential one:

a) Command View on the management module cannot be used to set up CA - you need to have Command View installed on a server that has FC connectivity to more than one EVA.

b) When you go to <IP>:2374 you do this over TCP/IP - this is not how you confirm FC connectivity between a server and a target.

The files you attached were the supportshow, not the output of "cvutil" (probably cvutil paths is a good place to start confirming connectivity). You could also use the HBA's management tool BCU or HCM).

Before you start installing Command View on a server, it would be a good idea to streamline things, so that you have the same XCS and Command View on all the devices. Generally it's bad to manage the same EVA/P6000 with different versions of Command View.

[Dips123]

Hi Johan

Thanks once again for your valued contribution. 

I think that the server I am logging on to is the CV server. The reason why I say this is because I am RDPing into the server. After logging in I point my browser to an IP address which is the same as the server and to port 2374. I have also seen that the CV is running on the server as a service. Is it sufficient confirmation that this means this is the CV server? This should also confirm that I am not going to the management module of the EVA.

When you go to <IP>:2374 you do this over TCP/IP - this is not how you confirm FC connectivity between a server and a target.

I apologise if I was not clear in my description before. The server which I use to connect to the port 2374 is also connected to the switches using HBA and fibre and on the switches it is zoned to the EVAs. I can see the WWPNs of the HBA of the management server on the switches. This is how I confirmed fibre connectivity.

The output that I attached previously, has the output of supportshow as well as nscamshow and nsshow. Apologies again if I was not clear in description. Do I need to run cvutil from cmd prompt?

Thanks for your suggestion about the XCS version, I am in the process of bringing the version to same in both the arrays.

One other new development that has happened since we last touched bases, 

I checked the switches again and saw that although I was pressing the Save Config on the switches it was not saving the config. I had to manually add the newly created zones in the zone config of the switch and then save. That saved the zones. After that I ran the discover storage again and it found something this time. 

Under the unmanaged storage systems on the prod EVA, it found something which has the same node WWN as the DR EVA. However the screen tells me that the system requires a password. I have tried entering the password that I use to login to the DR EVA CV interface but it has failed. I believe the password needs to be retrieved from the OCP. If this is the case, is there a way to retrieve the OCP password remotely without actually being present near the EVA?

Also it gives me a License alert that the system does not have the required license. However it does have the necessary licenses as per the EVA CV interface. 

I thank you again for your generous support. I can't thank you enough.

Regards,

[Johan Guldmyr]

Hi,

OK yes if you point your browser on the CV-server to <localhost:2374> then yes you have CV installed on it.

"I can see the WWPNs of the HBA of the management server on the switches." This is not how you confirm connectivity between a server and a target either. The above is perhaps how you confirm it between a server and a switch.

Yes, cvutil you need to run from the command prompt. But this should be fine now as you can see the DR EVA in CV.

Perhaps CV cannot see the licensing properly until you've entered the password for the remote EVA.

Do you have access to the WOCP (nowadays called "HP P6000 Control Panel GUI")?

You access it with https://IP.to.management.module:2373

[Dips123]

Hi Johan

Thanks so much for your contribution. I can't thank you enough. Yes going to port 2373, I could change the password and get the replication started. But I have faced another issue. I have created 8 replication groups and they have synced and are working fine. However when I want to create any more I get the error

"Operation failed

Not enough data storage available in the disk groups, or no compatible disk groups available for the requested raid type. If this error occurred during a data replication operation please check if the requested capacity/raid level can be supported in the disk groups on both source and destination."

The error doesn’t make any sense simply because, the destination EVA has not got any disks provisioned. I was always under the impression that once the replication starts the source disks and disk groups are created in the destination EVA. I haven’t created any disk groups in the destination EVA because of that. Any ideas why is it failing? The destination EVA has space, I was under the impression that as soon as the replication is started the destination EVA will have the disks from the source EVA recreated for presentation during a disaster. Why is it then failing?

I would appreciate, once again, any feedback. Thank you so much for your support thus far, I can't thank you enough.

Regards,

[Johan Guldmyr]

Hello Dips!

I'm glad to help :) But I must confess that now we enter subjects where I have not so much experience.

P6000 compatibility reference say that you can have 256 single-member DR groups. 

http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c02523047/c02523047.pdf

Some DR methods use a transaction log (wire history log?). Perhaps you do not have enough space for that?

[Dips123]

Hi Johan

I just wanted to inform you that I have successfully managed to start the replication. Thanks to you and other's help and support. You are right, it was a log issue. The default log size is 100 GB and as a result at the source the log file space was used up after creating 8 replication sets. I found this and configured the logs to be 5 GB instead and that fixed it. The logs can be also configured to be in tier 1 or tier 2 disks. Thanks a lot again.

One thing I wanted to ask was, after the initial sync is complete, it shows me that the log is not in use and there is no way to know that the files that are written to the source disk are replicated in the production site. Please find below one test replication set that I've created. It shows that the logs are not in use. But if the replication has started shouldn't the logs be in use? Also if the source disks are thick provisioned, then do the destination disks automatically become thick provisioned or is there a way to create the destination disk as thin provisioned to save space? Lastly, if I have multiple disk groups at the source, for eg: OS, Database, VMs, and I replicate them, is there a way to create separate folders to categorize them at the destination EVA replication set?

Name

Role

Write Mode

Log State

Test

Source

Synchronous

Not in use

Thanks a lot again. I can't thank you enough for the help and support that you have provided already.

Regards,

[Johan Guldmyr]

Hello! Glad you got it working.

It  would be have been good if you would have posted these questions in a new post.

Some comments nonetheless: 

• WHL (Write History Log) is for example used when synchronous or basic asynch replication is stopped or failed. 
• See page 42 on http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c02838340/c02838340.pdf