Skip to content

Instantly share code, notes, and snippets.

@bhavbhavyaa

bhavbhavyaa/gist:ce00417975ae4ef8e3324f4d05f20151

Last active June 24, 2021 02:29
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save bhavbhavyaa/ce00417975ae4ef8e3324f4d05f20151 to your computer and use it in GitHub Desktop.
Save bhavbhavyaa/ce00417975ae4ef8e3324f4d05f20151 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
def clean(doc):
tokens = doc.split()
table = str.maketrans("","",string.punctuation)
tokens = [w.translate(table) for w in tokens]
tokens = [word for word in tokens if word.isalpha()]
tokens = [word.lower() for word in tokens]
return tokens
tokens = clean(data)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment