D2D 4112 performance?

[maarten.vanachte]

Hi everybody.

I've got some questions about the 4112. Currently we're implementing this device in our serverroom but the first test doesn't seem that prommising.
Our first tought was at night backing up to the D2D (with deduplication because of the amount of data and the retention) and staging it at daytime to our 8096 liberary. (everything is connected to a redundant 4Gb fibre brocade SAN)
Our test now show that the D2D is A LOT slower then the physical liberary. D2D gives about 40MB/s and the liberary between 50 and 90MB/s.

Is this normal? Are there any ways to speed up the D2D?
Even when we switch the both and do the staging to D2D on daytime, the windows seems to small. A night window to liberary seems to work.

Thanks for the help.

[Dave Dewar]

Hi,

You should be achieving better performance than that, however without knowing more details on what else the appliance is doing at the time etc its difficult to comment with any certainty what the issue would be in your case.

A good starting point would be to go through the performance best practices white paper at the following link

http://h20195.www2.hp.com/v2/GetPDF.aspx/4AA2-7710ENW.pdf

and make sure you have the latest software for your 4112

[maarten.vanachte]

we're running at the latest firmware with the expansion shelve, so 20TB of storage.

Software is dataprotector and the tests where run as a single backup. No other backups where running.

The first backup seem to go reather well. The seconds onen the same data is slow (deduplicating). But I can't believe that a quadcore machine in raid is that slow.

I'll take a look at the document, maybe something rings a bell.

[andre80]

hi maarten

Just can speak about the 4004fc device. i don't know the 4112 system, but if it is about the same in hw as the 4004 device, you won't get a much better backup performance then about 40MB/s. one big problem is the housekeeping task which is running on those systems. if a cartridge gets unloaded during/after a backup job, housekeeping does start which use system and i/o performance. depending on the data which is written to the box, it can take severl hours until housekeeping is complete. if during this time offload or backup operations take in place, the performance does decrease (housekeeping and backup/offload operation). unforunately there is no monitoring availalbe about the status of house keeping for us customers...)

do you also make offloads with dp to tape?

i don't want to make the box bad, i just want tell you that performance about 40MB/s for backup is normal...

regards
andré

[marcus1234]

ok marten with library and tape tools run the dev perf test

and see what results you get


on this system what you are getting is not far from normal,
hint



This is how we thank each other in the forum

http://forums11.itrc.hp.com/service/forums/helptips.do?#33

Enjoy:)

[maarten.vanachte]

Hi all,

I've got a little time to check some things about the D2D but I didn't get far yet. I've noticed that the performance droped again. I think our ofloadingspeed has droped to about 10MB/s (I'll have to doublecheck).
Take notice that our backups happen at night and thake place on tape (MSL8096) using Dataprotector.
In the day we do offloading to D2D and that is verry slow. (tape 2 tape offloading is verry quick)

I tried using the tape tools (new v4.9) but it doesn't recognize the D2D. The MSL is ok.
Does anybody know what I'm doening wrong?

All drivers are installed on server except HP D2DBS diagnostic SCSI. Does anybody know where I can find this.

Thanks

[Dave Dewar]

Maarten

Just reading your latest post regarding tape offload poor performance.

Could you describe in more detail what your backup strategy is please. From the earlier posts right now I am assuming you are attempting to run a nightly backup to the D2D followed by a tape ofload the next day?
How big are the backup jobs and how do you have your appliance organised in terms of virtual libraries?

From your original post I got the impression you were sending a single stream (1 backup job) to the D2D and expecting it to outperform a physical tape library? These class of D2D appliances work best when multiple backup jobs are sent in parallel to get aggregrate performance that would match or exceed a tape library. Single stream performance will never match an equivalent physical tape product. This is explained in the performance best practices whitepaper on pages 7 and 8.

Tape offload on a large complex dedup store can be slow since the virtual cartridge has to be reconstructed from the store. This uses up both disk io and cpu overhead and if other tasks are also occurring then thats an additional overhead.

If tape offload is going to be a very frequent aspect of the overall backup policy for large job sizes then HP would recommend that a separate job from backup server to physical tape library is used either in parallel or after the D2D backup has finished rather than offloading direct from the D2D.

[maarten.vanachte]

Hi Dave,

we first tried the following setup:

at nigth: backup to D2D with duplication
at day: offloading to MSL


We soon noticed that this didn't work out becuase of the time window and the amount of data, so we changed the setup.
Know it is like this.

every night: backup to MSL
every day: offloading to D2D

The offloading is programmed so that it copies everything that backed up at night from 2 phys drive to 2 virtual drives on the D2D. Concurrency is set to 4.

this is the schedule:

mon-fri (at night): inc
thu-fri (at day): offloading to D2D
sat: full
sun: offloading to Tape for external storage


The offloading is about 3,2Tb of data. Our D2D deduplication ratio is about 5,2:1

If I work the speed out, I get around 12MB/s Also if I thake a look at the task, I see that it happens in bursts of 8000Kb

Hopefully you understand everything now and I hope you can shed a light.

Thanks

[Dave Dewar]

Maarten,

Apologies for the delay in replying back to you.

Thanks for providing more details of your new setup.

If I understand correctly you are now doing a DP copy from your MSL to your D2D each day and the total job size is about 3.2TB on a daily basis?

My understanding from your emails is that you have a requirement to have backups on both D2D and physical tape on a daily basis correct?

My guess right now with the model you have described is that you are using multiplexing for your MSL backup jobs - concurrency greater than 1 in DP which is right for the MSL but one days physical tape could look quite different to the next days tape from a D2D viewpoint due to how multiplexing works.

Sounds like you have a concurrency greater than 1 for the phsycial tape copy to the D2D as well. Not quite sure how DP would handle this. You appear to be backing up from two physical tape drives to two virtual tape drives each night, I am assuming you have LTO4 physical drives and are using 800GB virtual cartridges so the 3.2TB represents about 2 tapes worth per drive?

This is certainly a different usage model from the standard but the physical tape drives should be able to send data fast enough for the D2D and so the performance figures you are seeing sound like they are due to the D2D ingest rate no being sufficicent. This might well be due to other tasks active on the appliance such as houskeeping activity - depending how you do media pooling you could be generating unnecessary amounts of overwrites and possibly the data coming into the appliance has a high effective change rate and so the dedup process has to work harder to find matches.

My best suggestion is for you to raise a support call via HP support with the subject of poor performance. Ths will be the best way to get the 4112 you have set up to meet your backup goals. If the first and 2nd level support engineers are unable to help you the case would get escalated up to level 3 and level 4. I'm at level 4.

The appliance you have should be capable of dealing with a 3.2TB daily backup to D2D with correct setup of virtual libraries, ensuring you are sending the backups in a multistream fashion with at least 4 streams active and using media pooling etc to control overwrites as per the recommendations of the performance best practices whitepaper.

A daily tape offload you want to acheive in parallel together with the 3.2TB daily backup job size would be difficult to acheive in a 24 hour window and so I would recommend that you setup separate physical tape backup jobs after the D2D jobs have completed.

[David Ware_2]

We have the D2D 4004i version that we backup to at night and then copy to a HP MSL2024 during the day using Data Protector 6.11 and the performance during the day IS very slow. 12hrs to copy 460GB. Since our fulls are only about 460GB, I'm going to try wiping it and creating a VTL with no dedup. I suspect HP dedup engine is not up to the task of housing keeping and reading out to physical tape. I'm using the VTL more for staging, idea being to keep thruput to the tape drive high to protect from shoe shining. We also have a Data Domain DD530 at our other site which blows the D2D out of the water 80MB/s or up to 260GB/hr, but they are about 2-3 times more expensive.

[Dave Dewar]

David,

The basic 4004i has 6 disks in RAID 6 array and it's performance spec is 70MB/s aggregate assuming best practices as followed. I don't know whether you have an additional storage kit applied to take that up to 12 disks or what configuration you are following.
The aggregate word is key here because if you are performing a single backup to a single virtual tape drive you will not reach 70MB/s. The product performs best with multi stream operation.

You mentioned removing a VTL and creating a new one? So my guess is you only have 1 VTL set up and hence are only using up to 4 virtual drives and hence only operating a maximum of 4 streams in parallel? I guess it is possible you are only doing a single stream backup.

You mentioned you are using DP 6.11 for your backups? Best practice for DP with D2D would recommend a block size of 512KB for best performance and dedupe ratio.
I wonder also whether you are using multiplexing in you DP backups? The D2D prefers to be used in non multiplexed mode as multipexing can end up randomising the data stream coming in and so may result in lower dedupe ratio and lower performance.

We recently released an updated best practices document for D2D that can be found at the following link:

http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c02511912/c02511912.pdf

Which contains lot of information on how to use the D2D and acheive the best performance and dedupe ratio. There are also sections on how to get the best out of the recent NAS backup mode we introduced.

If your plan was to use the D2D as a very short term staging area or disk buffer to ensure good performance to your physical tape drive then probably a deduplication product was not the best fit since you are basically running dedupe to store to the D2D and then running reverse dedupe straight afterwards to store to the physical tape. A standard disk array might have been a better choice for this application since deduplication is pointless for that.