So, for validation, here is my suggestion. Works for however people want to submit (separate hashes.txt, password.txt or combined "hash:pass" file).
grab a copy of mdxfind/mdsplit from http://hashes.org . Ensure you have the contest-orginal pcrack.master.hashed.txt file (it will be read-only).
If separate files:
mdxfind -f pcrack.master.hashed.txt -h ^sha1$ plaintext.txt >result mdsplit -f result hashes.txt
You will be left with hashes.txt and hashes.SHA1x01. The hashes.SHA1x01 are the validated number of cracks (wc -l to get number), and any invalid hash submissions will be in hashes.txt. If this file is empty, all hashes validated.
- If combined hash:pass format:
cut -c 42- combined.txt | mdxfind -f pcrack.master.hashed.txt -h ^sha1$ stdin >result mdsplit -f result combined.txt
As before, correct solutions will be in combined.SHA1x01, and invalid submissions will be in combined.txt
It will literally take seconds to do this, and will need a machine with 3g of memory. On my machine (rotating rust, no SSD), ~30 seconds to read hash list, and 120 seconds to run 57M passwords through. Because you have almost 1,000,000 duplicate hashes in your original list, it's important to use mdsplit (which will count the finds for you, too)