May 16, 2015

How To: Speed Up File Transfers in Linux using RSync with GNU Parallel

    In order to rsync a huge chunk of data (containing considerably large number of smaller files), the best option one can have, is to run multiple instances of rsyncs in parallel. This seems to be pretty effective, but at the cost of high load average, more I/O oparations and network bandwidth utilization.
 
    So as to parallelize multiple rsync commands, one might use xargs or a series of rsync commands run in the background using &. But, over all of those alternatives, I would prefer GNU Parallel, a utility used to execute jobs in parallel. It is a single command that can replace certain loops in your code or a sequence of commands run in background.
 
Read more at YourOwnLinux
Click Here!