backup symbolic link very slow.

[Panuwat]

Hi all,
I has the symbolic link files about 400,000 files on HP-UX 11.11. The total space is about 25GB. Now I use "tar" command for backup and restore. And the tape machine is Ultium2. It take time for backup and restore over 2 days. I think that very slow.

Please help me. Who have the better backup solution?

Thank you.

[Vibhor Kumar Agarwal]

Are the links referring to different file systems or machines.

Then this will take time.

[Himanshu_3]

Panuwat,

There could be many factors which could decrease your speed for the backup and restore among them mainly...

- The disks could be attached to a SAN.
- Check the speed of your LAN network card in that case.
- Kind of connectivity to storage ie FC etc

All these factors may influence the backup and restore to disks.

Regards,
Himanshu

[DCE]

Have you thought about using HPUX fbackup,or any of the commercial packages? They all offer the ability to perform incremental backups. You can perform a full backup every so often (i.e. once a week or once a month) and just backup those files that changed since the last full back up in between (i.e. daily)

Y

[Matti_Kurkela]

Fast tape drives like Ultrium2 are a bit like CD-ROM burners (without BurnProof or similar technology): they want to have a continuous stream of data as fast as they can write.

If your data stream is not fast enough or gets interrupted, bad things happen: with CD-ROM burner, you get "buffer underrun error" and an unusable disc.
With Ultrium2 tape drive, the drive needs to rewind the tape a bit, find the exact spot where the data stopped, then wait for more data before starting to write again. If this happens repeatedly, this makes backup *very* slow.

Check all the components between the data disk(s) and the tape drive. Any component which cannot handle the tape drive's speed is a potential cause for your problems.

[Panuwat]

Thank all,

The links are referred to the same filesystem. All data is in the internal disks. I just only have problem about backup the 400,000 links files is very slow. Otherwise is fast.
The Ultium2 tape drive is connected ot the system machine by scsi cable. No LAN. No Fibre.
I'm not sure the backup very slow because the size of symbolic link is vary small. It cannot use the full performance transfer rate of Ultium2. That right?

Best regard.

[Matti_Kurkela]

Are the 400,000 symbolic links all in one directory?

Any directory with 400,000 files (of any kind) in it is going to be rather tricky to access. If you use "ls", the computer will sort the listing before showing it to you, and this will take a noticeable amount of time and memory.

The same is usually true about wildcard use: when the wildcards are expanded by the shell, the expanded list of filenames is sorted alphabetically. The wildcard expansion might also stop with a "command line too long" error message.

If you need a list of filenames from a directory with 400,000 files for some processing, use the "find" command. It will list the files in the order they are stored in the filesystem, which will be much faster in cases like this.

If some of the 400,000 files get deleted and new ones added on a regular basis, the directory structure in the filesystem will get fragmented. You can check this with "fsadm -D". You can reorganize the directories to remove the fragmentation with "fsadm -d". If you have not done this before, the fragmentation might be so severe you need to increase the number of reorganization passes using the "-p" option.

The reorganization will probably not remove the problem (unless the fragmentation is extreme now), but it might give some improvement.

[Bill Hassell]

If you are truly backing up just the symbolic links, then it should take just a few minutes. However, tar has the option to backup the item that the symbolic link represents (the -h option). Now it's important to define the scope of your bakcup: are you backing up the symbolic links *and* the 400,000 files at the same time? If so, then the symlinks are not the problem. You can prove this by creating a /var/tmp directory, then create the 400,000 symlinks in that directory and backup just that directory--it should take a few minutes.

Now all of this assumes that you are using the standard tar command and that tar has not been replaced with some freeware program (where the options and defaults may be quite different)

If you are backing up both the symlinks and the data files, then your system is too slow to keep the Ultrium tape busy and it is constantly re-positioning (a bad thing for tape and head wear, and a terrible thing for performance). You must use a program that can read multiple files at the same time (tar, cpio, dump, pax, etc cannot so this). The only program that can do this is fbackup (or any number of commerical backup tools like HP's Data Protector).

If you use the command:

tar cvf /dev/rmt/0m /mybigdir

then the fbackup command will be:

fbackup -v -c /root/fbackup.config -f /dev/rm,t/0m -i /mybigdir

Note that you need a config file to adjust fbackup's performance. The file should contain:

blocksperrecord 512
records 64
checkpointfreq 1024
readerprocesses 6
maxretries 5
retrylimit 5000000
maxvoluses 200
filesperfsm 2000

Store this in root's HOME directory (/root in this case, never /). Then compare the backup time. Note also that a busy system can still slowdown the backup speed. Today's tape drives are not just bigger, they are extremely fast and quite intolerant of long system delays.