Restore is faster then backup ?

[Leif Halvarsson_2]

Hi,

I get significant better throughput when restoring a filesystem then when backing up the same filesystem

Test scenario:
- Host is a rp3410, HP-UX 11.23
- File system: 20GB, 350 000 files.
- Disk system is a EVA3000, 56 72GB 10k disks.
- Raid level: Vraid5
- Library is a MSL6060 (Ultrium 2).
- Backup software Data Protector 5.1.

- Backup throughput: 25MB/s
- Restore throughput: 33MB/s

I have always thought that reading data (backup) from a RAID volume is faster then writing data (as writing needs more disk I/O).

Is it normal that restore performs better then backup (with EVA disks) or is there something strange in our enviroment.

[Uwe Zessin]

I have never seen any timings, but you have a lot of files with an average file size of 57 KBytes.

For reading, you can assume that the data is not in cache so that the disks need to do a physical read, perhaps with a head seek.

For a restore, it is possible that some meta data is still in the writeback cache and can be replaced immediately while the next data can already be fetched from tape.

[C. Beerse_1]

Are you sure the raid-box is your bottleneck? Most times tape-devices cannot keep up with the disks. Then, yes, you are right, reading is faster than writing but is are the tapes to look for.

There are some more factors to look at: do the drives write raw to tape or is hardware-compression enabled? what is the scsi-configuration? are disk and tape over the same bus or separate each? Are the busses in the same controller or in separate controllers?

Peek at the activity lights while performing backup or restore: you will notice the tape will go in streaming mode and the disk only flash once in a while.

[Uwe Zessin]

C.,
we're talking about an Ultrium 2 tape drive, not a DAT ;-)

Modern tape drives are no longer 'slow'!


My guess is that there are 'Ultrium 460' tape drives in the library, which can do between 10 and 30 MegaBytes native on tape.


'disk and tape over the same bus'? The disks are on a Fibre Channel storage array, most likely connected through a 2 GigaBit infrastructure.

[Leif Halvarsson_2]

Hi,

The filesystem is part of a much bigger filesystem on a production server. I restored a subdirectory from that server to a test server. This directory was selected because it had the largest number of files/smallest average filesize. The reason for the tests is poor backup performance on the production server (I get better throughput on the test server but this is a different question). The problem here is why I bet better throughput for restore.

I doubt if caching can have any impact in this case. The Vdisk and filesystem on the test server was fresly created for this test. The data was restored direct from the tape to the test server (Restore to different client in DP).

[TwoProc]

I find that DataProtector increases its throughput to tape as the number of filesystems that it is backing up increases. The reason is that a single mountpoint has trouble keeping the tape running at maximum speed - more than one filesystem on different disks has a better chance of accomplishing this. Althought the HP Ultrium2 drives have technology that tries to make the tape run at a speed that keeps the tape from having to resync - it's still not as a good as keeping the data coming at full speed.

Ironically, the bottleneck here is your hard drives, and how fast they can pull data and get them to the agents (and the number of agents, and the number of agents per mount point), not how fast the tape drives are.

Compare the same idea with 2, 3, or 4 different mountpoints being backed up from across different controllers and disks and you'll see what I mean. You'll see then that your saves are going to be more like your restores. You'll see your MB/sec throughput go up.

I don't find that DataProtector is all that fast for single mount points/file systems - but where it really shines is when you've got multiple file systems at play at the same time.

[krusty]

A good website for troubleshoot backup problems is here:

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=lpg50460&locale=en_US

At a cursory glance of the issue, my best guess is that many of the files are very small and small files do not "feed" well to current tape drive technologies.

There is a FREE program on that webpage that may help you. It is called hpReadData, and it can help you analyze the mountpoint you are trying to back up. By running the hpReadData program, it will tell you MB/s the filesystem can source to the tape. As Uwe stated, an Ultruim 460 (LTO2) can sink 10 to 30 MB/s, but if hardware compression is being used, you need to multiply those numbers by 1.5 - 2.0 (typical actual compression ratios). Therefore you need to source a MINIMUM of 15 MB/s to the tape to keep it streaming. Theoretically, the tape could sink up to 60 MB/s, so you need to feed it fast! You may need to use multiple agents/hosts to get these numbers

Cheers,

Curt

[Leif Halvarsson_2]

Hi,
Yes, I am aware of the performance problem when backing up a filesystem with a large number of small files but that is not the main reason for my question, it is why restore performs better then backup (on a EVA disk system).

I tried the same backup/restore on a "ordinary" RAID system (9 disks in a RAID5 disk group). This system behaviours exact as expected, restores (writing) is about 25% slower then backup (reading).

So, I still wonder if it is something strange with out EVA or, higher write performance is normal on EVA arrays ?

[Alzhy]

Leif,

How many times have you done your test? Is this EVA shared?
It is possible when you're doing your backups the EVA wasbusy servicing other connected servers. Also, what size is your backup/restore set?

[Leif Halvarsson_2]

Hi,

A number of times (10-12). Always when low activity on the EVA. Data size between 10 and 20GB.

In my latest test I created 2 10GB volumes on the HP-UX server, one located on the production EVA described in my original post, the other on a different EVA3000 with only 8 300GB 10k drives. Tried a number of backup/restores.

I consequently got better backup performance with the 8 disk system then the with system 56 disks. Restore performance was about equal.