The following bash one-liners are useful commands for manipulating wordlists (or any text file). For instance, if you need to remove all blank lines from a file, a one-liner will do the trick. Similarly, if you need to remove duplicate passwords (or text), you can do that too. If you have multiple wordlists, you can also combine them into one large file.
- Remove duplicates
awk '!(count[$0]++)' old.txt > new.txt