MichaelSasser

#!/usr/bin/env bash
# dotfiles
# Copyright (c) 2021  Michael Sasser <Michael@MichaelSasser.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,

MichaelSasser

#!/usr/bin/env bash
# find-pacnew - find and optionally delete pacnew and pacsave files
# Copyright (c) 2020  Michael Sasser <Michael@MichaelSasser.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,

MichaelSasser

I used these scripts with wordlist-dedup to deduplicate a whole collection of wordlists safely. I used a tempdir variable in the dedup script to temporary dir located on another harddrive to speed up the sorting process. I changed it to /tmp in the file below. You can choose something else, if you like.

• dedup does the whole thing. Let's assume, you have a file: filename.ext. The script first sorts the output in a file: filename_sorted.ext and then deduplicates the file to a third one: filename_sorted_dedup.ext.
• The script to_txt converts non txt files to txt files.
• The cleanup script deletes non deduped files. Move the three files in the folder you want to dedup filewise.

MichaelSasser

; Input	MD5   :	8473EE6AE0AE1EDB104B48997C081E17
; Input	CRC32 :	95D320A2

; ---------------------------------------------------------------------------
; File Name   :	PIC18LF4525_ESAM4500_SW1_0_DelonghiCode5213214971.hex
; Format      :	Intel Hex Object Format

; Processor	  : PIC18Cxx
; Target assembler: Microchip's MPALC
include	"P18F4525.INC"

MichaelSasser

; Input	MD5   :	ADC48838255379FC0933AC3FB11DEE55
; Input	CRC32 :	407BF79D

; File Name   :	PIC18F452_EAM3500_SW19_DelonghiCode5232109900.hex
; Format      :	Intel Hex Object Format

; Processor	  : PIC18Cxx
; Target assembler: Microchip's MPALC
include	"P18F452.INC"