Fixing Duplicates in Google Drive using Rclone to Dedupe

RClone-Fixing-Duplicates-GoogleDrive.ps1

<#
RCLONE HELP
    https://rclone.org/commands/rclone_dedupe/

Useful options I choose
    --max-depth int
    --dry-run
    --log-file=PATH
    --tpslimit 1 # can help prevent rate limiting errors you might see if you run verbose x2, ie `-vv`
    --checkers 1
#>

# Install Rclone and a log viewer to make it nice to review results
choco upgrade rclone -y
choco upgrade tailblazer -y # nice tail log view for streaming rclone activity

# Alias for usage
new-alias rclone -value "C:\Program Files\rclone\rclone-v1.42-windows-amd64\rclone.exe" -force

# logging to this directory
New-Item 'C:\temp' -ItemType Directory -force
$LogFile = "C:\temp\rclone-$(Get-Date -format 'yyyy-MM-dd').log"



# Tested Against Folder With Dups With Dry Run
rclone dedupe newest googleappsdrive:Test --log-file=$LogFile --dry-run

# Ran against folder and it removed 3 of the 4, leaving only one file, now deduplicated
rclone dedupe newest googleappsdrive:Test --log-file=$LogFile

# another folder with dups but larger. This ran into issues when I didn't limit depth
rclone dedupe newest --dry-run googleappsdrive:"Amazon Drive\Development" --log-file=$LogFile --max-depth 2

# merge the root folder level only of duplicate folders
rclone dedupe newest googleappsdrive:"" --drive-skip-gdocs --log-file=$LogFile -vv --tpslimit 4 --transfers 1 --fast-list --max-depth 1 --stats=30s

# now dig into my Lightroom Library and deduplicate a bit more. I did this to confirm it was working before I did everything in my folders.
rclone dedupe newest googleappsdrive:"Lightroom" --drive-skip-gdocs --log-file=$LogFile -vv --tpslimit 4 --transfers 1 --fast-list --max-depth 1 --stats=30s
rclone dedupe newest googleappsdrive:"Lightroom" --drive-skip-gdocs --log-file=$LogFile -vv --tpslimit 4 --transfers 1 --fast-list --max-depth 2 --stats=30s
rclone dedupe newest googleappsdrive:"Lightroom" --drive-skip-gdocs --log-file=$LogFile -vv --tpslimit 4 --transfers 1 --fast-list --max-depth 3 --stats=30s

# now that I've got everything figured out, I ran against the entire drive.
rclone dedupe newest googleappsdrive:"" --drive-skip-gdocs --log-file=$LogFile -vv --tpslimit 4 --transfers 1 --fast-list --stats=30s