Parallel cpio in one shell-script or acceleration of cpio

[Peinelt]

Hi,

i ve to copy with cpio an amount of data, so my full-backup needs 3h.
cpio is my choice..but how can i copy parallel with cpio in one thread?
IO has enough bandwith left and the system idles with 95%.
So my questions: how can i copy parallel with cpio or are there other possibilities to accelerate my cpio-job?

Thanks & Regards!

[Ravi_8]

Hi,

# & (put backup process in background)

# & (put cpio process also on back ground)

now both backup and cpio will be running in parallel in background

[Peinelt]

hi,

thx 4 answer. this is the first step.
but after the backup i ve to restart my whole applications so i need to know when background-processes ve finished.

greetz,
peinelt

[Steve Lewis]

I think you need to stop considering cpio and use fbackup instead.
Fbackup uses a shared memory segment and multiple reader processes to greatly accelerate the throughput of data to the tape drive.
Unless you write your own software to parallel stream and double-buffer in this manner, I don't think you will be able to do this using the older UNIX utilities.
Another option is to use a commercial backup package, such as Veritas Netbackup, Legato or HP's own (whatever Omniback is called these days).
Of course, if you have several drives, you can simply run multiple backups at once of different directories to the different drives.
Another necessity is to ensure that your disk storage system can stream the data to the tape drive at the full rate that allows the drive to write continuously. If your tape drive constantly has to stop then start, you will get very bad performance indeed and the tape drive will wear out far quicker.

[Steve Lewis]

Sorry, didn't realise you were copying to disk. To answer your question about checking on background processes:

wait

[Ralph Grothe]

wait is the correct call,
but wait would block until the child(ren) you are waiting for finished.
So you may want to put this call after your asynchronous processing has been done.
Alternatively you could set a trap for a SIGCHLD.
Remember that the PID of your last backgrounded process is stored in the special shell variable $!.
You could also use the jobs command to fetch the PIDs of your background jobs.
Then with those PIDs you could look up the process table or send a SIGNULL via kill to see if the processes are still alive.

[Florian Heigl (new acc)]

Ralph:

$! is the best thing I've ever learned, You just accelarated all those remote-config-check scripts by 100's of times. Now I only need to think of a nice load monitoring mechanism :)

florian is a happy florian now.