Slow Backup / Copy

[Ray Pemberton]

I have 2 800GB LUNs presented from an EVA500 vraid 5 to 2 HPUX Servers.

Each LUN is in a volume group with 1 logical volume and holds dbf files for an oracle database.

backup on Server ssv3000d using legato:-
average speed of 19MB per second

Copy of a 10GB file in ssv3000d:/u65/oradata/TSPP (EVA5000 DG5) date ; cp applsysx_TSPP03.dbf copyofpplsysx_TSPP03 ; date Tue May 27 13:59:11 BST 2008 Tue May 27 14:08:45 BST 2008 9mins 34secs



backup on server orafre03 using legato:-
average speed of 65MB per second

Copy of a 10GB file in orafre03:/u35/oradata/DBA2 (EVA5000 DG5) date ; cp applsysx_DBA2_03.dbf copyofapplsysx_DBA2_03 ; date Tue May 27 14:25:43 BST 2008 Tue May 27 14:30:07 BST 2008 4mins 24secs



mount the filesystem from ssv3000d onto orafre03 read only
backup average speed 20MB per sec

mount the filesystem from orafre03 onto ssv3000d read only
backup average speed 66MB per sec

This I think tells be I have an issue with the data not the server.

From the output of fsadm and mkfs the only difference I can see is that the Average Extents for the filesystem backing up / copying slowly is extreemly large?

{ssv3000d} root> # fsadm -F vxfs -ED /u65/oradata

Directory Fragmentation Report
Dirs Total Immed Immeds Dirs to Blocks to
Searched Blocks Dirs to Add Reduce Reduce
total 3 1 2 0 0 0

Extent Fragmentation Report
Total Average Average Total
Files File Blks # Extents Free Blks
69 1179118 54231 23478554
blocks used for indirects: 7383
% Free blocks in extents smaller than 64 blks: 0.00
% Free blocks in extents smaller than 8 blks: 0.00
% blks allocated to extents 64 blks or larger: 99.99
Free Extents By Size
1: 0 2: 1 4: 0
8: 1 16: 1 32: 0
64: 0 128: 0 256: 1
512: 0 1024: 0 2048: 0
4096: 0 8192: 0 16384: 1
32768: 4 65536: 2 131072: 1
262144: 2 524288: 1 1048576: 1
2097152: 2 4194304: 0 8388608: 0
16777216: 1 33554432: 0 67108864: 0
134217728: 0 268435456: 0 536870912: 0
1073741824: 0 2147483648: 0

{ssv3000d} root> # mkfs -m /dev/evg65_oradata/lvol1
mkfs -F vxfs -o ninode=unlimited,bsize=8192,version=4,inosize=256,logsize=256,largefiles /dev/evg65_oradata/lvol1 838729728


{orafre03} root> # fsadm -F vxfs -ED /u35/oradata

Directory Fragmentation Report
Dirs Total Immed Immeds Dirs to Blocks to
Searched Blocks Dirs to Add Reduce Reduce
total 3 1 2 0 0 0

Extent Fragmentation Report
Total Average Average Total
Files File Blks # Extents Free Blks
71 1135468 65 24219465
blocks used for indirects: 56
% Free blocks in extents smaller than 64 blks: 0.00
% Free blocks in extents smaller than 8 blks: 0.00
% blks allocated to extents 64 blks or larger: 100.00
Free Extents By Size
1: 1 2: 4 4: 6
8: 1 16: 0 32: 1
64: 0 128: 0 256: 1
512: 1 1024: 1 2048: 1
4096: 0 8192: 0 16384: 0
32768: 1 65536: 13 131072: 14
262144: 14 524288: 14 1048576: 6
2097152: 2 4194304: 0 8388608: 0
16777216: 0 33554432: 0 67108864: 0
134217728: 0 268435456: 0 536870912: 0
1073741824: 0 2147483648: 0

{orafre03} root> # mkfs -m /dev/evg35_oradata/lv_u35_oradata
mkfs -F vxfs -o ninode=unlimited,bsize=8192,version=4,inosize=256,logsize=256,largefiles /dev/evg35_oradata/lv_u35_oradata 838729728

Help!

[TTr]

You have to check all the other paths and factors that the backup traffic follows. Any one of them can affect your backups. Some of the things to look at are

- 800 GB LUNs
- LUN spindles (number, size, speed)
- SAN path (zoning, switches, port speed)
- server HBA (speed, multiport, other ports active)
- server I/O bus (PCI slot speed)
- server (model, backplane, server load)
- server LAN (LAN based backups?)
- LAN (route from DB server to backup server)
- backup server (I/O, load, interfaces, tape resources)

[Ray Pemberton]

TTr - Both LUNs are created within the same disc group on the same EVA.

With mounting the filesystems to the alternate systems and performing the test backups, do you other points baer any relevance? If so could you explain please.

During the tests I observed no system or SAN performance issues.

Both backup and data servers have 2 fibre connections to 2 director switches.

Is an average file extents of 54231 an issue?

[OFC_EDM]

I'd review the stuff TTr provided.

I'd compare the following too:
Version of the Legato agent on the two servers.
Driver versions on everything (HBA's, NICs etc.)

Are you certain Legato is backing up over the SAN for the problem server?...and not over the network?

How is everything zoned together? (tape library, disk, servers). Do they all go through the same switch?

Did you check the Legato logs for any messages for either server?

[Ray Pemberton]

The filesystem on Server 1 which was backing up slowly was mounted on the eother server, and it still backed up slowly. The filesystem on server 2 which backed up fast was mounted on the eother server, and it still backed up fast.

[TTr]

The VXFS fragmentation could definitely be an issue. What is the nature of the DB data in these servers? usually with DBF files you don't have that problem because the # of files is small and they don't change a lot. Well they do but oracle itself minimizes that effect.

Back to the backups, are both of these servers media servers or clients? And where is the mediaserver? Is the backup LAN or SAN based? And is the route of the backup traffic for each server the same?

[OFC_EDM]

Now that makes it more interesting. Because Server 1 does seem capable of backing up quickly.

Without more information on each server though can't help you much. RE: TTr's comments