For similar tasks I use a perl script similar to the one attached and included below.
It may serve you as a starting point.
I starts a controllable number of concurrent dd commands, piping output into a compress tool.
I feed it a file of files.
The file of files is generally organized, sorted by descending size so that you do not have to wait for the longest file at the end.
I also try to rotate through vg's, trying to avoid starting all jobs from a single vg.
I gave the output numbers as name dirived from the entry in the file of files, but you could ofcourse mung the raw device name into a useable filename (vgxx_lvyy.gz).
The file should probably capture the dd 'records input' output from STDERR but it does not for now.
This version prints the start time, stop time, elapsed time, stream used, pid used, file number and file name for each file.
(Oops, did not include final streams, oh well, left as exercise to reader!)
This version is for tests only and thus limits the data volume with count=40 on the dd line.
In my own versions I derive max-streams from the cpu count and also use mpsched to force cpu affinity as I 'own' the box.
waddayathink?
Hein.
$file = shift or die "Must provide file of files";
$max_streams = shift (@ARGV);
$max = ($max_streams)? int ($max_streams) : 4;
open (FILE, $file) || die "Error open file of files: $file";
while () {
$num++;
chop;
if ($par++ >= $max) {
$pid = wait();
$par--;
$s = 1;
$s++ while (($stream[$s] != $pid) && ($s <= $max) );
printf("%02d:%02d:%02d %04X %5d %s\n",
$hour,$min,$sec,$pid,time() - $time[$s], $file[$s]);
die "Lost child process ?" if ($s > $max);
} else {
$s = $par;
}
$pid = fork();
if ($pid) {
$stream[$s] = $pid;
$file[$s] = sprintf ("%02d %3d %s",$s,$num,$_);
$time[$s] = time();
} else {
($sec,$min,$hour) = localtime(time);
printf("%02d:%02d:%02d %04X BEGIN %02d %3d %s\n",
$hour,$min,$sec,$$,$s,$num,$_);
exec ("dd bs=1024k count=40 if=$_ | gzip > /backup/hein/$num.gz")
}
}
close (FILE);
print "Last job started\n";
$pid = wait() while (--$par > 0);
print "Last job (pid=$pid) ended\n";
;
