Deduplicating a result set using shingleprints

dedup_results.py

seen_shingleprints = set()
for doc in search_results:
    if any(shingleprint in seen_shingleprints for shingleprint in doc.shingleprints):
        continue  # doc has at least 1 already-seen shingleprint, so skip it
    
    final_results.append(doc)
    seen_shingleprints.update(doc.shingleprints)

the continue only affect the inner for-loop, maybe should be modified like this

seen_shingleprints = set()
for doc in search_results:
    skipped = False
    for shingleprint in doc.shingleprints:
        if shingleprint in seen_shingleprints:
            # doc is a near-duplicate, skip it
            skipped = True
            break
    if not skipped:
        final_results.append(doc)
    seen_shingleprints.update(doc.shingleprints)

Good catch - thanks for letting me know! I switched to using any(), which keeps the code more succinct.

Also note that if the document is skipped, its shingleprints shouldn't be added to the seen_shingleprints. (Your last line should be indented to be inside the if: ).