I’ve found the rsync protocol to be of an immense help when transferring a large number of files, especially for site migrations or pulling down a local copy of a site that isn’t already in a repository like Github.
It’s faster and more efficient than FTP and most online file managers, the latter which carries the overhead of the browser.
Keeping it fast and efficient depends on some key configuration options, and there are a lot of them. You can connect directly to an rsync daemon, but this article assumes the use of a remote shell connection (SSH), which is more likely to be available on a wide number of hosts.
Per example:
-z - compress files during transfer, to really speed things up
Ideally, you’ll want a prompt free SSH connection as long as this doesn’t go against your security policy, with a proper setup of public / private key credentials between the local and remote hosts.
Here’s a simple command to recursively copy a directory, and lets use something many of us are familiar with, a WordPress site hierarchy:
rsync -rvlz --copy-unsafe-links --size-only --ipv4 --progress -e 'ssh {-p PORT, if other than 22}' {username}@{hostname.com}:public_html/wp-content/plugins ./wp-content
From your WP web root (above wp-content), issuing the above command will transfer the plugins directory in its entirety inside the wp-content (local destination) directory. If you issued this command while in wp-content, you’d suddenly have /wp-content/wp-content/plugins.
Run a man rsync to get more details on each option, such as pattern matching:
rsync -t *.c foo:src/
This would transfer all files matching the pattern *.c from the current directory to the directory src on the machine foo.
Or “archive” mode, which ensures that all attributes such as file ownership, group, etc. are preserved:
rsync -avz foo:src/bar /data/tmp
But in general:
-r: Recurse into subdirectories
-v: Verbose output
-l: Copies symlinks as symlinks
-z: Compress during transfer
--copy-unsafe-links: Transforms symlinks into files when the symlink target is outside of the tree being copied
-e: Specify remote shell to use (notice we provided ssh for this)
--size-only: Skip files that match in size (part of the remote-update protocol that skips files that already exist on the destination machine)
--ipv4: Prefer IPv4 IP's