LVM Mirroring Performance

[Steven Hargus_3]

We are in the process of migrating from one EMC DMX frame to another. Because of the amount of data involved and the business impact of an extended outage, we are doing the migration using mirroring.

When we mirror one volume group, the mirroring takes place about 20 extents / minute. On a second volume group with many more PV's, we are only getting 10 extents / minute.

Although syncing up a mirrored LV does not appear to have a major impact on server performance (you can see it on GPM, but you have to look for it), the DBA's and end-users report a 50% drop in performance while the extents are being synced-up. This observation has also been made during past migrations.

Currently, we are facing 72 hours to fully sync up the larger volume group. If we have to break up the process (so that mirror syncing is done during off-peak periods), it may take over a week.

What we would like to be able to do is control the rate that the LV's are synced up - making it sync up faster during off-peak periods and slower during peak times. Is there a way to do this?

[Patrick Wallek]

Nope. Sorry.

Once you issue the 'lvextend -m 1' command, everything is out of your control.

[Mel Burslan]

Unfortunately, Mirror-UX is not a good method for data migration. I am not sure about the nature of the business conducted on these servers, but you are better of taking the database(s) off line over a weekend and doing a vxdump/vxrestore for a much faster transfer. My experience, using vxdump was at least 3 to 4 times faster than doing a vgsync. And this was when the database was down and nothing significant was running on the server other than either process. You can shave off the 72 hours time frame to 24 hrs or less.

Something to consider as the long weekend is on horizon.

[Pedro Cirne]

Hi Steven,

I had a similar migration from a Symmetrix 8530 to a DMX-800.

We used a mixed solution for migration because you can't use mirror in all cenários:

-You can't use it if you have VG in Shared Mode.

-You can't use if lvol is LVM striped.

-You can't use it if the MAX_PV in your VG is near the limit and you're not able to add the disk from the new storage.

On the servers I used mirror, performance was not a problem, be carefull when adding the new PV to VG to use a diferent HBA, do it in a way that one HBA is used to READ and the other is used to WRITE.

On servers that mirroring was not possible, we used SRDF(from EMC) to replicate volumes from one storage to the other, this has no impact at all on performance, you'll need some downtime to enable the replicated volumes on your servers, but carefully planned, you just need some minutes.

If you have Service Guard, don't forget the Lock Disks issue!

Enjoy :)

Pedro

[UX LX Support]

If you can afford the down time you could use dd from the raw lv
e.g
dd if=/dev/vga/rlvol1 of=/dev/vgb/rlvol1 \
bs=64k
You could perform this on multiple LVs in parallel depending how your PVs are laid out on the array.

[TwoProc]

Greetings Steven,
I'm pretty sure I know you from years and years back, so - good to hear that you're about.

For speed, I think you're going have to give up on mirroring. I'm thinking that if you use vxdump you will have the best luck from a speed standpoint. But, that will have to be during down periods as your databases will have to be cold for the duration of the copy - which I'm guessing is what you're trying to avoid.

Another possible (if you're using Oracle), is have your dba's clone your database from a weekend cold backup from your tape storage system, and without "open"ing it, apply archive logs against it from your live environment. When you're ready for your cutover, just switch out the last archive logs from your database to force out the last changes, and then shutdown the original db. Then apply these last changes to your still unopened database from the archive logs( as they come out of the production DB ), then "open" it. This would be for your dbas much like a recovery of their live system. I've used this technique before to minimize downtime in cutting over to new servers, it should work equally well to cut over to new storage.

The beauty of it is you can build it and prepare it all week long after the weekend cold backups run without affecting production, and the total cut-over time for your go-live transition is only about 15 minutes or so (without considering issues affecting your applications' need to actually connect to the new database, which could possibly require lots of planning for that aspect - or it could be as easy as changing one tnsnames.ora file, depending).

[Steven E. Protter]

How much data is it?

lvextend -m 1

is one of the few commands that warns you it is going to take some time. That is because it does.

You might want to measure transfer rates on your SAN to make sure there is no bottleneck making things worse.

SEP

[Bill Hassell]

Mirroring is done at the atomic level, that is, via the driver is such a way that other activity will not corrupt the mirroring task. Essentaially, the extent being mirrored is locked on both the source and the destination to prevent changes, then the extent is copied and verified, and then the extent is unlocaked. 10-20 extents/min sounds like the extents are larger than the default 4k, but it will be slow *and* it will impact other processes using the same extents. This is good news for reliability (the mirrored extents will be accurate) but the disk is going to appear to be quite slow.

Given a slow replication and impact on applications, or a complete database shutdown and disk copies (cpio -p, fbackup|frecover, dd, etc) in a much shorter time, you'll have to decide.

[TwoProc]

Whoops, I'm sorry - that would be from a hot backup, not from a cold. And, if you can take the load, it can be run while actually running the database, otherwise, you'd pick to run it during a slow period.

[Steven Hargus_3]

The reason we are doing this migration is because EMC wants to reformat the DMX frame the data is currently running on.

EMC wants to stripe the extents as they are being moved over to the new frame, so we will end up with the data striped across metavolumes that are themselves striped within the DMX frame (so we will end up with double-striped filesystems). EMC wants to do the striping in pairs of metavolumes, so we set up the lvmpvg file so that the extents are arranged in pairs on the destination frame.