StoreEasy Storage
1748179 Members
3881 Online
108758 Solutions
New Discussion юеВ

Re: Storageworks StorageMirroring AIO 1200

 
Jonathan Haase
Advisor

Storageworks StorageMirroring AIO 1200

I'm running into some issues currently with the Storagemirroring product. I have latest available version instlaled on both of my AIO 1200 devices (5.1.1.867).

Right now I have a total of about 445GB of data that I want to mirror between the two servers. The data consists of a small Exchange Store ( < 16GB ), a variety of shared folders, and the replicated volumes from Microsoft Data Protection Manager 2007 for 8 other servers. Both AIO 1200's have 4GB of RAM installed. At the moment for the initial mirroring of the data, they are both connected via a Gigabit backbone. Once the initial mirror is done we will be moving one of them to the other side of a T1 link for offsite data protection.

The problem I'm experiencing is that even over the Gigabit backbone the Mirroring just never quite get's caught up. I create the initial replication set and configure the connection to start the mirror and replication and and first things seem to go well. An example of the most recent attempt, I started the process at 2:21pm. over the next 30 minutes, the mirroring computed the total size of the mirror, and started sending the data. Then over the then after 30 minutes or so and after having sent around 4GB of the data ( I really would think it would get better throughput than that...)
through the mirror process, and maybe 1GB or so through replication. The replication queue starts incrementing, and quickly within 10 minutes or so has grown to having 10-20GB in the replication queue, and the Replication Bytes sent continues incrementing, however the mirrored bytes sent freezes. another 10 minutes or so after that happens, then one of two things will happen.

1.) The process will decide that it's used too much memory in the queue, and a re-mirror is needed, so the process starts all over.

2.) The event that has happened the most however is that I'll get disconnected and see an error in the event log
The Storage Mirroring service terminated unexpectedly. It has done this 1 time(s). The following corrective action will be taken in 60000 milliseconds: Restart the service.

This process all *used* to work even across the T1, until recently when we added a few additional backup targets for Microsoft DPM, and the total mirrored data ended up growing from around 275-300GB of data to 445GB of data. Do we just have too much for this software to even work for us anymore? I would think it should be able to keep up over a 1Gb Link.

25 REPLIES 25
peyrache
Respected Contributor

Re: Storageworks StorageMirroring AIO 1200

hello
please post event system and application
from both source and target servers
thanks
JYP
Jonathan Haase
Advisor

Re: Storageworks StorageMirroring AIO 1200

The eventlogs, even zipped are too large to fit here... since this limits the maximum size to 1MB.

I just had this happen again. Last night just to ensure that I wasn't dealing with some type of data corruption issue on the target side of things I went ahead and triggered a "full" mirror with it sending all files to the target, but did *not* enable replication. That process completed successfully, taking about 7-8 hours or so to complete the full mirror of 445GB of data (remember this is over gigabit). So this morning when I come in, I went ahead and told it to start the mirror again to replicate any changes that were made through the night, and this time I also allowed it to start the replication. I started the process at 8:08am this morning. It spent the first 8 minutes or so calculating the size of the mirror and then started churning through the data. It made it until about 8:35 at which point the service terminated unexpectedly on the source. The last time that I looked at the console before the service terminated It was about 5% done with the mirror, and also had about 5GB in the replication queue. The replication queue didn't start filling up until 5 minutes or so before the service terminated, prior to that the replication queue had stayed pretty well empty. This particular time the service did not create a .DMP file in the program files directory, however in the pas it has done so, as recently as yesterday I have a "Doubletake_1000_1376_2009_11_12_15_30_155.dmp" file sized at 100MB.

I will note that at 8:30, the DPM service did do a backup job for one of the servers so that is likely the cause of the replication queue growing suddenly. The only other event of note in the System event log is "The Storage Mirroring service terminated unexpectedly. It has done this 1 time(s). The following corrective action will be taken in 60000 milliseconds: Restart the service."


The last entry from the Storage Mirroring prior to the service crashing was when it started the replication and mirror after calculating the replication set size at 8:16am.

THe last entries in the Driverlog.dtl prior to the crash are as follows:
05/12/2009 08:30:26.0128 WARN:FileAccessZwError. ZwOpenFile failed for \DosDevices\C:\Program Files\Microsoft DPM\DPM\Volumes\Replica\mvreport01.CBOPC.local\File System\H-3d3d6a19-6446-11dd-8c0a-002264c29406, status C000009D (IoStatus 0)

05/12/2009 08:30:26.0144 WARN:FileAccessZwError. ZwOpenFile failed for \DosDevices\C:\Program Files\Microsoft DPM\DPM\Volumes\Replica\mvreport01.CBOPC.local\File System\H-3d3d6a19-6446-11dd-8c0a-002264c29406, status C000009D (IoStatus 0)

05/12/2009 08:30:26.0144 WARN:FileAccessZwError. ZwOpenFile failed for \DosDevices\C:\Program Files\Microsoft DPM\DPM\Volumes\Replica\mvreport01.CBOPC.local\File System\H-3d3d6a19-6446-11dd-8c0a-002264c29406, status C000009D (IoStatus 0)


THe dtlog.dtl file shows nothing from 8?18 when the mirror started, until 8:37 when it shows the application starting again.
peyrache
Respected Contributor

Re: Storageworks StorageMirroring AIO 1200

attached best pratices with DPM and storage mirroring , since they use same part of Server Kernel Memory , you need to avoid to run them concurrently
let me know if you need more help i'll open an FTP account in order to push events
JYP
Jonathan Haase
Advisor

Re: Storageworks StorageMirroring AIO 1200

The document you've attached doesn't really address anything at all related to running DPM and the storage mirroring software at the same time. In point of fact I don't really see how it would be possible to avoid running them both at the same time. Since DPM by default is taking snapshots of the servers once per hou, and the whole point of the StorageMirroring/Doubletake software is to replicate the changes "as they occur", how can you avoid running them both at the same time?

It also does not address the fact that the actual StorageMirroring service keeps dying with the service "terminating unexpectedly".
peyrache
Respected Contributor

Re: Storageworks StorageMirroring AIO 1200

first of all stop dpm do a full replication
between source and target then restart dpm
JYP
Jonathan Haase
Advisor

Re: Storageworks StorageMirroring AIO 1200

I'm starting that process right now... I'll let you know.
Jonathan Haase
Advisor

Re: Storageworks StorageMirroring AIO 1200

Ok, that made absolultely no difference. I disabled all of the DPM services on the Source storage server, and restarted the mirror process. I started the process at around 10:15am CST It ran for less than 15 minutes, and this time the replication queue didn't even start to build up when at approximately 10:30am the Storage Mirroring service once again "Terminated Unexpectedly".
peyrache
Respected Contributor

Re: Storageworks StorageMirroring AIO 1200

drop event system and application on this ftp
pointer please
JYP
TEMPORARY ACCOUNT INFORMATION --

FTP System: hprc.external.hp.com (15.192.32.69)
Login: haase
Password: Fireman6 (NOTE: CASE-sensitive)

Created: 5/12/2009 at 3:39:08 PM (UTC)
Expire: 6/11/2009 (in 30 days)
Expire Reminder: none

Owner: ******
Owner E-Mail: ******@hp.com

--------------------------------------------------------------------------------

BROWSER/FTP ACCESS --

ftp://haase:Fireman6@hprc.external.hp.com/
ftp://haase:Fireman6@15.192.32.69/
Jonathan Haase
Advisor

Re: Storageworks StorageMirroring AIO 1200

The files are finishing uploading now. I also uploaded the most recent Doubletake_....dmp file from the storagemirorring directory. That is the one from the crash where it only ran for 15 minutes or so...

I'm in the process of retrying the mirror right now.

These eventlogs hadn't been cleared in a long time, so there is a lot in those logs. The basic parts that you are interested in would be beginning Friday May 8th through now.

The Dump file is from 10:31am CST today.