Rclone - CLI cloud backup via rsync

Submit command line tools that you find here.
Post Reply
Message
Author
shnbwmn
Posts: 265
Joined: Sat Jul 11, 2015 12:59 am

Rclone - CLI cloud backup via rsync

#1 Post by shnbwmn » Thu Aug 18, 2016 10:37 am

Homepage: http://rclone.org/
Github: https://github.com/ncw/rclone

Downloads: http://rclone.org/downloads/
Rclone is a command line program to sync files and directories to and from:
  • Google Drive
  • Amazon S3
  • Openstack Swift / Rackspace cloud files / Memset Memstore
  • Dropbox
  • Google Cloud Storage
  • Amazon Drive
  • Microsoft One Drive
  • Hubic
  • Backblaze B2
  • Yandex Disk
  • The local filesystem
Features:
  • MD5/SHA1 hashes checked at all times for file integrity
  • Timestamps preserved on files
  • Partial syncs supported on a whole file basis
  • Copy mode to just copy new/changed files
  • Sync (one way) mode to make a directory identical
  • Check mode to check for file hash equality
  • Can sync to and from network, eg two different cloud accounts

User avatar
webfork
Posts: 9250
Joined: Wed Apr 11, 2007 8:06 pm
Location: US, Texas
Contact:

Re: Rclone - CLI cloud backup via rsync

#2 Post by webfork » Sun Aug 21, 2016 1:35 pm

Limitations: only supports one way sync, which is fine for backup purposes.

License: MIT (http://rclone.org/licence/)

The number of platforms this covers is impressive. Cross-platform (BSD, Linux, Mac, Solaris).

I've long held that RSYNC on Windows this is an important problem to solve.

Unfortunately, it's not clear to me that this does delta copying (only what's changed) as the RSYNC reference suggests. The FAQ is a little unclear on this fact.

User avatar
Midas
Posts: 5331
Joined: Mon Dec 07, 2009 7:09 am
Location: Sol3

Re: Rclone - CLI cloud backup via rsync

#3 Post by Midas » Tue Jun 19, 2018 9:17 am

Rclone v1.42 released (changelog at http://rclone.org/changelog/).

User avatar
Midas
Posts: 5331
Joined: Mon Dec 07, 2009 7:09 am
Location: Sol3

Re: Rclone - CLI cloud backup via rsync

#4 Post by Midas » Mon Feb 10, 2020 6:25 am

Rclone v1.51.0 released 2020-02-01 (changelog at https://rclone.org/changelog/).

About the delta copying aspect, the relevant FAQ entry appears to be the following:

rclone.org /faq/#why-doesn-t-rclone-support-partial-transfers-binary-diffs-like-rsync

rfithen
Posts: 14
Joined: Wed Apr 01, 2020 5:44 am

Re: Rclone - CLI cloud backup via rsync

#5 Post by rfithen » Wed Apr 01, 2020 6:13 am

I love rclone because: robocopy lets me down with regards to timestamps, inevitability it recopies large files that are the same after some time of use. Even if using /dst /fft. Switches /xo /xn are not a usable solution. Instead rclone compares checksums of files, not just timestamps and file sizes. rclone proceeded where robocopy left off. (imho)

User avatar
Midas
Posts: 5331
Joined: Mon Dec 07, 2009 7:09 am
Location: Sol3

Re: Rclone - CLI cloud backup via rsync

#6 Post by Midas » Thu May 28, 2020 5:21 am

Rclone v1.52.0 released 2020-05-27 (changelog at https://rclone.org/changelog/).

Since posted here back in 2016 a lot has changed, not least the amount of compatible file hosts -- here's the full list as copied verbatim from the ReadMe file (highlights are mine):

1Fichier, Alibaba Cloud (Aliyun) Object Storage System (OSS), Amazon Drive, Amazon S3, Backblaze B2, Box, Ceph, Citrix ShareFile, C14, DigitalOcean Spaces, Dreamhost, Dropbox, FTP, Google Cloud Storage, Google Drive, Google Photos, HTTP, Hubic, Jottacloud, IBM COS S3, Koofr, Mail.ru Cloud, Memset Memstore, Mega, Memory, Microsoft Azure Blob Storage, Microsoft OneDrive, Minio, Nextcloud, OVH, OpenDrive, OpenStack Swift, Oracle Cloud Storage, ownCloud, pCloud, premiumize.me, put.io, QingStor, Rackspace Cloud Files, rsync.net, Scaleway, Seafile, SFTP, StackPath, SugarSync, Tardigrade, Wasabi, WebDAV, Yandex Disk, and the local filesystem.


The main rclone highlight has to be its builtin functionality to encrypt cloud storage; for a brief intro, check https://rclone.org/crypt/. But, alas, as I noted above there's no delta sync.

Rclone is a command line program to manage files on cloud storage. It is a feature rich alternative to cloud vendors' web storage interfaces. Over 40 cloud storage products support rclone including S3 object stores, business & consumer file storage services, as well as standard transfer protocols.

Rclone has powerful cloud equivalents to the unix commands rsync, cp, mv, mount, ls, ncdu, tree, rm, and cat. Rclone's familiar syntax includes shell pipeline support, and '--dry-run' protection. It can be used at the command line, in scripts or via its API.


Rclone also sports an experimental web-based GUI by default -- for more info, check https://rclone.org/gui/.

FYI, there's also independently developed RcloneBrowser (https://github.com/kapitainsky/RcloneBrowser) with native portability (just create an INI file with same name as the EXE alongside, otherwise settings will go to registry and '%APPDATA%').

Rclone 32 and 64-bit Windows versions can be downloaded from https://rclone.org/downloads/ (releases repo is at https://downloads.rclone.org/).

Post Reply