Skip to content

Instantly share code, notes, and snippets.

@ayushoriginal
Created June 24, 2019 05:46
Show Gist options
  • Save ayushoriginal/16ccbb23697c15c7f23ea4ca242e0bd5 to your computer and use it in GitHub Desktop.
Save ayushoriginal/16ccbb23697c15c7f23ea4ca242e0bd5 to your computer and use it in GitHub Desktop.
tokenize and lowercase
def tokenize(self):
from nltk import word_tokenize
for i,tweet in tqdm(enumerate(self.data),'Tokenization'):
self.data[i] = word_tokenize(tweet.lower())
return self.data
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment