Synchronize folders between two servers using rclone and ssh-agent with ssh based authorisation

README.md

Introduction

This tutorial will help you transfer your folders / files between two server using rclone and ssh-agent.
It was tested between two Ubuntu 16.04 servers.

Installation

First things first. We need to install the neccessary packages:

apt-get update -y
apt-get install unzip screen -y
curl https://rclone.org/install.sh | sudo bash

Then we need to setup a 'remote' with a 'SFTP' conntection. Read more here about this process.
As a 'name' use IP of your server where you want your files to land. Use blank for ssh port if it's 22 or set your custom port.
Use blank for 'username' and 'password' if it's a newly created server and you are working as a root user. Don't type in path to the PEM file, we will use ssh-agent.
Use false and false accordingly for the last two options.

Then we will need to initialize a screen shell that will work independetly of our connection to the server.
Type the following command to the console and press 'Enter':

screen

If you are using PuTTYgen to generate your private key you need to convert it to the OpenSSH format.
Select 'Conversions' and then 'Export OpenSSH Key' to do this in PuTTYgen.
Now upload your private key to the server.
Let's use ~/.ssh/ directory to store it and let's name your private key id_rsa
Restrict access to this file:

chmod 0600 ~/.ssh/id_rsa

Initialise a remote connection using ssh-agent:

eval "$(ssh-agent -s)" && ssh-add ~/.ssh/id_rsa

If you did everything right you will see the following line:

Identity added: /root/.ssh/id_rsa (/root/.ssh/id_rsa)

Put list of the folders that you want to copy from the original server to the new one in following format in the file like 'copy.txt':

/var/wwww/folder_1 remote:/var/www/folder_1
/var/wwww/folder_2 remote:/var/www/folder_2
...
/var/wwww/folder_n remote:/var/www/folder_n

Change 'remote' to the name of your second server specified in the rclone config.
I suggested earlier to use IP of the second server in the name of its remote config.
Then run the following command:

cat copy.txt | xargs --max-args=2 --max-procs=4 rclone copy

This will start the copying process in 4 threads.
After it ends your can close the remote connection:

eval "$(ssh-agent -k)"

Then close the screen shell:

exit

Troubleshooting

If you are using Windows to create your 'copy.txt' it's most likely that it will use Windows EOL Conversion (CR LF).
That can result in appending \r to the name of each folder on your second server.
Use the command ln -lb to investigate your folders on the second server.

• If you haven't transfer any folder use your text editor to set Unix EOL Conversion (LF) instead of Windows in the 'copy.txt'.
• If you already sent somefiles and noticed that you can't access your files because of the exessive symbol that they got in their path use the following command to replace that symbol:

find . -type d -name $'\r' -exec mv "{}" "$(echo {} | sed -e 's/\\r//g')" \;

Or you can use this command. But beware that it's more strict because it basically uses whitelist of characters that are allowed in the folder name:

for f in *; do mv "$f" "$(sed 's/[^0-9A-Za-z.-._]//g' <<< "$f")"; done