Remote snap shot ( copy )

[larry-cgb]

I have 2 SAN's ( 4 p4000) Connected with 100meg WAN. I set the bandwidth to remote to 100meg. The remote copies are at only 4 – 8k sec. If I move the local bandwidth to 40 mb the nightly copies jump to 40,000k to 90,000k. Now the 197 Gigs take 5 hours. I thought the local bandwidth was for SAN back ground tasks ? Is it ok just to leave the local set to 40, the book says it may effect application servers ? Is there a way to have local bandwidth cranked up at night only ?

[Bryan McMullan]

4 to 8k a second is VERY low, do you mean 4 to 8 mbps? If it's 4 to 8k, you may have something very wrong.

The issue you'll have is when you move the local bandwidth up, you can starve access to your volumes as management tasks can take more priority than data access which could cause disk errors on your servers (if you have volumes connected).

I'm not aware of any method that can be used to switch the local bandwidth at night automatically. Perhaps you could have a scheduled task on one of your server that uses CLIQ to change it?

I'd highly recommend against 40 though. I'd suggest inching it up a little at a time to see what the impact is first.

[Bryan McMullan]

Just one aside, here's someone else who was having the same type of issue with the slow remote snaps. Apparently he let the system create the remote volumes and it sped things up.

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1415381

[larry-cgb]

switching both ends back to for tonight to see. I tried this before and by speed dropped 1000 times slower. Been working for 10 days with both ends set to 40. During the remote copies the SAN has no other work to do. Did try setting both to 20 meg and I still got or remote copy rates.

[Bryan McMullan]

If there's no volume access to speak of at the time you do the remote snaps, I guess there's no issue with pushing it to 40. We don't have any downtime like that, so pulling something like that caused ISCSI PRT errors on our machines and quite angry SQL installations (and we were nowhere near 40).

[larry-cgb]

I attach the screen shoots. You can clearly see the remote copy rate the last two days took a big hit. I move the slider rate back to 4 meg two days ago, now for two days the rate dropped. How can I fix this without moving the slider and slowing the apps during the day ?

[Bryan McMullan]

Well, I can see in either case you are nowhere near 4-8kbps. You are running at minimum 12Mbps (12,000kbps). Looks like out of the 7 volumes, 4 continued in roughly the same speed and 3 cut down by about 50%.

Are you running all of these snapshots concurrently? Are you staggering them throughout the day? Meaning, do you start them all at the same time?

[larry-cgb]

these are logs of two remote copies sets for the last week. Once a day, one each day. During this new test it didn't drop down compleatly but still dropped from 57000k to 14000k and the other set dropped by 50 percent. This is a production network and what happens is the remote copies don't finish in the window then a new one kicks off and the repeat of this snow balls. There should be no way a remote copy over a 100 meg wan can slow the SAN's lan side. Just the fact that the setting effects the remote copy so much seems to show there is an issue with the settings. remote copies need there own settings, which they have, but are not used to determin bandwidth it is using. If I move the slider to max I use my whole 100 meg bandwidth which shows me the WAN is working 100 percent just the remote copy is not. The way it is now if I need to do a restripe or a rebuild my remote copies wont work unless this slider is cranked.

[Bryan McMullan]

It could be possible that there is a bad node. But think about it this way, when you slide it all the way to 40, the SAN is using most of it's processing time and disk IO to do management tasks and assigning very little to actual application/machine access.

I have a 50Mb pipe between sites and can usually only pull about 20-25Mb with the slider on 6. Which is still well above the 4-8k. I know you gave an example of your results in the file you attached, but are you still seeing the actual 4-8k transfers? What I saw wasn't maxing out your line, but it was still healthy compared to what you were seeing before.

I've got to talk to my HP contacts about the slider's relationship to remote snapshot speed. If I hear back before you contact support, I'll let you know what I find.

[larry-cgb]

I think it drops way down only when I have just created a volume or done a resize. As I setup the boxes I would do my admin tasks after hours right before the remote copies start. I still need to test again to be sure but it appears if I do the admin task then the remote copy starts it just pushes off the remote copy unless the slider is pushed up, when the admin job finishes then the remote copy speeds back up.

[Bryan McMullan]

May be a worthless question, but were the volumes restriping/resyncing during the remote snapshot?

If so, then you are probably correct in that your admin tasks are impacting performance. I've never seen it cause a problem here, but I'm running 50 nodes, so it may be different if you only have a couple.

[larry-cgb]

left it set a 4 for a week them moved both ends to 16 ( heard it was the new setting in 8.5 ) but as you see it speeds the remote copy up a lot. I attached the screen print. I remote copy once both ways every night. The new setting is the top one. I was hinted that this is fixed in 8.5 but not sure.

[Bryan McMullan]

I played around with the settings in my environment and I see what you are saying. My speeds never went down to 4/8k sec, but there was quite a difference.

I'm still waiting on a response from my SE on this as remote copy speed used to be controled by remote bandwidth, local bandwidth only controlled resync/restripe/rebuild speed. The horror that would come from the mouths of LHN support when it came to talking about the local bandwidth setting made it sound quite taboo to change the default (though we still did when needed).

[Bryan McMullan · Accepted solution]

I finally received notification back from my SE to confirm your findings.

Cutting and pasting the response*

Local Bandwidth of the target group will limit the remote copies between the two groups if Remote_Bandwidth > Local_Bandwidth. We do have plans to de-couple this behavior in a future release, however, for 8.5 this behavior remains. In 8.5 we did increase default Local Bandwidth to 16MB/sec. Key info is local bandwidth of the target group is the key so they could throttle up their DR siteâ s local bandwidth and not hurt anything.

End cut and paste*

So it looks like the speed can be adjusted by setting only the remote side local bandwidth rather than needing it on both sides. (Not sure if you had set only the remote side, or both sides.) I can confirm in testing it does work with remote side only.

This must have been snuck into the 8.x releases.

[larry-cgb]

I DR both site to each other. I will look for the script the changes the local bandwidth at night.