Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Fixing Duplicates in Google Drive using Rclone to Dedupe
<#
RCLONE HELP
https://rclone.org/commands/rclone_dedupe/
Useful options I choose
--max-depth int
--dry-run
--log-file=PATH
--tpslimit 1 # can help prevent rate limiting errors you might see if you run verbose x2, ie `-vv`
--checkers 1
#>
# Install Rclone and a log viewer to make it nice to review results
choco upgrade rclone -y
choco upgrade tailblazer -y # nice tail log view for streaming rclone activity
# Alias for usage
new-alias rclone -value "C:\Program Files\rclone\rclone-v1.42-windows-amd64\rclone.exe" -force
# logging to this directory
New-Item 'C:\temp' -ItemType Directory -force
$LogFile = "C:\temp\rclone-$(Get-Date -format 'yyyy-MM-dd').log"
# Tested Against Folder With Dups With Dry Run
rclone dedupe newest googleappsdrive:Test --log-file=$LogFile --dry-run
# Ran against folder and it removed 3 of the 4, leaving only one file, now deduplicated
rclone dedupe newest googleappsdrive:Test --log-file=$LogFile
# another folder with dups but larger. This ran into issues when I didn't limit depth
rclone dedupe newest --dry-run googleappsdrive:"Amazon Drive\Development" --log-file=$LogFile --max-depth 2
# merge the root folder level only of duplicate folders
rclone dedupe newest googleappsdrive:"" --drive-skip-gdocs --log-file=$LogFile -vv --tpslimit 4 --transfers 1 --fast-list --max-depth 1 --stats=30s
# now dig into my Lightroom Library and deduplicate a bit more. I did this to confirm it was working before I did everything in my folders.
rclone dedupe newest googleappsdrive:"Lightroom" --drive-skip-gdocs --log-file=$LogFile -vv --tpslimit 4 --transfers 1 --fast-list --max-depth 1 --stats=30s
rclone dedupe newest googleappsdrive:"Lightroom" --drive-skip-gdocs --log-file=$LogFile -vv --tpslimit 4 --transfers 1 --fast-list --max-depth 2 --stats=30s
rclone dedupe newest googleappsdrive:"Lightroom" --drive-skip-gdocs --log-file=$LogFile -vv --tpslimit 4 --transfers 1 --fast-list --max-depth 3 --stats=30s
# now that I've got everything figured out, I ran against the entire drive.
rclone dedupe newest googleappsdrive:"" --drive-skip-gdocs --log-file=$LogFile -vv --tpslimit 4 --transfers 1 --fast-list --stats=30s
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment