How to utilize cloud storage like Google Drive, to copy local or NAS data with rclone, for out of region disaster recovery

Published in

Google Cloud - Community

9 min readJan 23, 2019

tl;dr: If you want to copy data from local or network attached storage to your cloud storage — encrypted or not — without relying on any applications other than rclone, keep on reading.

Destroyed hard disks. Hopefully you have a copy of those… (copyright applies to this photo)

Background

A couple of weeks ago I decided to lift my private NAS backup strategy to the next level by adding out-of-region disaster recovery. In other words, I wanted to copy my most important data not only to local disks and NAS, but also store them in the cloud on Google Drive.

Of course I already have a lot of files in the cloud, e.g. Google Docs and Sheets. But there are others, like some music and especially RAW photos, which have not been so far.

This guide will outline, how to use rclone for encrypted copies from a QNAP NAS to Google Drive, while keeping your current storage setup and workflows as is. It is also applicable to other NAS, Linux or even Windows environments, as well as other Cloud Storage (e.g. Google Cloud Storage, OneDrive, Dropbox, S3), requiring only minor adjustments.

Additionally, it describes how to make use of Google Cloud Service Accounts, to limit rclone’s access rights and the chance of your account credentials getting in the wrong hands. I will provide an option to use Google Drive’s folder sharing mechanics, to make sure you can access your encrypted files without the service account, and even access those files in the Google Drive web application.

I focused on Google Drive as part of the Google Cloud and G Suite family. It is a no-cost, consumer ready solution for low data volumes. It also offers an easy to understand pricing model for consumers if you need more storage.

Important to understand: this approach doesn’t support snapshots or versioning of files (see section “Difference between copy vs. sync and why versioning is not available” below)!

Alternatives

Alternatives on QNAP

There are plenty of similar use cases utilizing QNAP built-in features which allow for encryption out of the box, like backing up to Google Cloud Storage or AWS S3, (e.g. QNAP Hybrid Backup Sync App).

There is even a Google Drive connector available (e.g. QNAP Cloud Drive Sync). But for reasons only known to QNAP, it does not support encryption.

Alternatives for general purpose encryption

You could work around the missing encryption feature by encrypting your data before putting it on your local storage. Or use software like Cryptomator (now also available as server edition running in a container), or Boxcryptor in desktop environments. But these options require you to make adjustments to your existing storage / backup concept or make you dependent on specific vendors.

rclone.conf example file we are going to build

[GDrive_User]
type = drive
scope = drive.file
service_account_file = /share/homes/username/BackUp/rclone/my-service-account-file-12345.json
auth_owner_only = true/* Below is optional and only required if you want to encrypt your files */
[GDrive_User_Encrypted]
type = crypt
remote = GDrive_User:NAS_Documents/20181117_Baseline/
filename_encryption = standard
directory_name_encryption = true
password = someHash
password2 = otherHash

Install rclone on QNAP NAS

Download rclone according to your CPU architecture. For QNAP NAS devices you find unofficial pre-built packages in the QNAP Club (that’s the one I used).
Connect to your NAS using SSH.
Check if rclone is installed by invoking the rclone command.

[/] # rclone
//Output
Usage:
 rclone [flags]
 rclone [command]…

Create a Google Drive remote link and add encryption

This guide assumes your are using Google Drive, but rclone supports most other cloud storage options, too (see configuration options).

Create a Google service account

I propose to connect using a Google service account, which needs to be created first (see below). There is also an option using standard OAuth2 tokens, but I don’t think this would work in a NAS context, as it requires a web browser and (periodic) log-in to your Google account.

Edit October 22, 2019: Just learned from a fellow reader that OAuth2 should work. You'd receive a URL you open in the browser once, then copy the OAuth2 token to the console and take it from there. One additional benefit is that you can restrict rclone's access only to files created by it.

Creating a service account not only works via the Google Cloud Console, but also via a lightweight interface, the Google APIs IAM site. I suggest the latter if you don’t use any other Google Cloud Platform services so far.

Go to the Google APIs IAM site
Klick on “Service Accounts”
Create a new project, if you haven’t done so far. Project name could be “rclone copy”. Then you get your service account ID / email, e.g.

Click on the newly created service account and create a key in JSON format

Download the key, as it will be required for your rclone config file

Keep the key in a safe location as it will grant everybody who gets hold of it, read and write access to the files stored in (or shared with) the service account.

Create rclone “remote”

Please follow the official documentation and create a standard remote. A remote is like a connector to a specific folder in a particular cloud storage.

[Optional] Add encryption

Now you add encryption on top of the remote link created above. The official documentation also describes this in detail.

[Optional] Make copied files visible in your Google Drive

During setup you might have checked back to your browser and your Google Drive account, realizing that the files you just copied are not there. That’s because the service account doesn’t have access to your existing Google Drive files! Picture the service account as kind of a virtual, new Google Drive account, but tied to your quota. You can only access it’s content via the Google Drive API, like rclone does.

You can make your (encrypted) files available in drive.google.com or the Google Drive app, but be aware that this also means that you, or somebody / an app having access to your Google Drive could (accidentally) delete your copy created with rclone.

If you decide to do so, share a specific folder (e.g. shared_folder) in your Google Drive with the service account. As your service account name looks like an email address (e.g. name-of-your-sa-account@your-project-name-123456.iam.gserviceaccount.com) you can share the copy folder in the Google Drive web UI. Just add the service account “email” as an additional user to the copy folder. You may receive an automated email stating that the service account email is not reachable, but it will work nevertheless.

From now on you can advise rclone to access this shared folder using the drive-shared-with-me flag. This works for all rclone commands (see official documentation). Here’s an example:

rclone copy --drive-shared-with-me /source/files your_remote:shared_folder/

Make rclone.conf durable and back it up

Configuration durability on QNAP devices

On QNAP devices the rclone configuration is stored at /root/.config/rclone/rclone.conf. As this location will be wiped after NAS restart or latest after a NAS upgrade, you need to manually copy the rclone.conf file at a different location, e.g. at /share/homes/your-user/rclone/. Please note: anytime you change the configuration or create a new one using rclone config, you need to save the configuration again at a durable location.

Configuration backup

Backing up rclone.conf to a save remote location is strongly advised as it contains the (hashed and salted) passwords to restore your encrypted data.

As you only need to recover your files in case of a major event, I propose to encrypt it and put it to a different cloud storage or maybe even print it and store in a save, remote location. If all your local copies and backups are destroyed or not accessible, your password store, notes or however you try to remember the password and salt might also be gone!

You could recreate the file using your password and salt in a new rclone config run, but better be safe than sorry. The service account key file (*.json) could be downloaded any time from the Google Cloud Console, so backing it up isn’t required.

Difference between copy vs. sync and why versioning is not available

Rclone supports several file manipulation commands. I’d like to quickly highlight the difference between copy and sync and make clear that currently there is no snapshot feature!

Copy

Well, this command copies files, ignoring unchanged ones, utilizing file size and modification time. Important for our use case: this command does never ever delete a file at the target location. Even if the file was deleted at the source. But it does overwrite changed files! I use copy in this guide. See also the official documentation.

Sync

This command synchronizes the source to the destination. Especially it would also delete files at the destination if deleted at the source. But never the other way around. See also the official documentation.

Snapshots / versioning

Rclone doesn’t support out of the box versioning or snapshot creation!. Changed files will be overwritten on copy instead of versioned! So I recommend not to rely on rclone “backups” only. Make sure your actual backup (e.g. on USB disks, local servers) allows for versioning and use rclone only, if all your backup sources are not / no longer available. Usually this would imply a disaster scenario.

To give a quick example: if you have a file like important.txt, save it on Monday, change it on Tuesday and want to revert to the Monday version on Wednesday, you cannot do this with rclone. Except you use this work-around which implements a versioning-like behavior, but I don’t use it here (see official documentation on backup-dir).

[Optional] Create your specific run script and schedule for periodic execution

I propose to create a script summarizing all required options and schedule periodic execution using crontab.

In my copy script below, I use two additional options:

--config and --bwlimit.

The first one is required to point to the rclone.conf file. Remember the standard location for this file gets wiped after reboot on QNAP NAS devices! The second one should limit bandwidth usage to 500 kilobyte. Adjust to your specific uplink environment.

Please note, bandwidth limitation doesn’t work for me. A quick research showed this might be a common bug. Therefore I use my router to control bandwidth usage by utilizing Quality of Service settings. It is important to keep plenty of spare upload. Otherwise your download might get affected, including e.g. video streaming.

My rclone_copy.sh script

#!/bin/sh
rclone --config /share/homes/username/rclone/rclone.conf --bwlimit=500k copy --drive-shared-with-me /share/homes/username/documents GDrive_Username_Encrypted:
echo -e $(date -u) “ rclone_copy.sh ran” >> /share/homes/username/rclone/rclone.log
exit 0

Make script executable

Don’t forget to make your rclone_copy.sh script executable on *NIX environments. Usually you use

chmod +x /path/to/rclone_copy.sh

Schedule regular execution

Last but not least, I propose scheduling using crontab. Usually you would create the crontab entry via crontab -e. But if you do that on a QNAP NAS device your command gets lost! After some confusion I found this description by Daniel Hansson on his Tech and Me blog. Thank you very much! Please follow Daniel’s workaround if you are on QNAP.

In short you need to:

crontab -e
// insert your entry (vi-like editor)…
// make persistent
crontab /etc/config/crontab
// reload
/etc/init.d/crond.sh restart

My crontab entry (running rclone_copy.sh every hour):

0 0–23 * * * /share/homes/username/rclone/rclone_copy.sh

As my copy script writes to a log file after every execution, you can use this log as a cross check. The alternative is looking at cryptic filenames or counting encrypted files in your Google Drive copy folder :D.

Avoid parallel execution during initial upload

Depending on the amount of data and your upload capacity, initial copying to Google Drive might take days or weeks. I recommend not to setup a cron job during this time. Multiple rclone instances running in parallel will usually not increase upload efficiency but lead to heavy seeking on your NAS hard disks. You might want to check from time to time if your initial upload is completed, or for potential early script terminations.

Whats next?

Open a terminal and start building your out of region disaster recovery setup for your most precious local data. Remember: data is only valuable if you have access to it. So please test a restore.

You could also have a look on further reading recommendations:

I appreciate your feedback and improvements, like implementing a snapshot feature.