Skip to content

Instantly share code, notes, and snippets.

@avangardistic
avangardistic / pyDupRem.py
Created August 6, 2020 12:37
Remove duplicate urls of a text file
# Require python 3.8
import sys
import pandas as pd
def remove_dup(path):
f = open(path, "r")
arr = pd.Series([line for line in f])
f.close()
return arr.drop_duplicates()