Skip to content

Instantly share code, notes, and snippets.

@indiesquidge
Last active March 12, 2024 12:55
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save indiesquidge/b0c1af6fa838013feac0cd0d2b0b6875 to your computer and use it in GitHub Desktop.
Save indiesquidge/b0c1af6fa838013feac0cd0d2b0b6875 to your computer and use it in GitHub Desktop.
Node binary to delete duplicate files in a directory
#!/usr/local/bin/node
/*
* Usage: in-place deletion of duplicate files within a folder.
* ```
* ./deleteDuplicates.js relative/folder/path/of/duplicate/files
* ```
*
* You may need to change the access control to run this as an executable.
* ```
* chmod 755 deleteDuplicates.js
* ```
*
* Keep in mind this will delete *in place*, so make a backup of your folder
* if you think you may need to keep the duplicates for whatever reason.
*
*/
const crypto = require("crypto");
const fs = require("fs");
const folder = process.argv[2];
const absoluteFolder = `${__dirname}/${folder}`;
fs.readdir(absoluteFolder, (err, files) => {
if (err) {
console.log(err);
return;
}
const map = {};
files.forEach(filename => {
const absoluteFilePath = `${absoluteFolder}/${filename}`;
if (fs.existsSync(absoluteFilePath)) {
const input = fs.createReadStream(absoluteFilePath);
const hash = crypto.createHash("sha256");
hash.setEncoding("hex");
input.on("end", () => {
hash.end();
const hashValue = hash.read();
if (map[hashValue]) {
fs.unlinkSync(absoluteFilePath);
} else {
map[hashValue] = true;
}
});
input.pipe(hash);
}
});
});
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment