Skip to content

Instantly share code, notes, and snippets.

@prateekjoshi565

prateekjoshi565/text_preprocessing_elmo.py

Created March 6, 2019 18:33
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save prateekjoshi565/fbe2aa38e16de4314563d4612fb4cfd6 to your computer and use it in GitHub Desktop.
Save prateekjoshi565/fbe2aa38e16de4314563d4612fb4cfd6 to your computer and use it in GitHub Desktop.
Download ZIP
text preprocessing elmo
Raw
text_preprocessing_elmo.py
# remove punctuation marks
punctuation = '!"#$%&()*+-/:;<=>?@[\\]^_`{|}~'
train['clean_tweet'] = train['clean_tweet'].apply(lambda x: ''.join(ch for ch in x if ch not in set(punctuation)))
test['clean_tweet'] = test['clean_tweet'].apply(lambda x: ''.join(ch for ch in x if ch not in set(punctuation)))
# convert text to lowercase
train['clean_tweet'] = train['clean_tweet'].str.lower()
test['clean_tweet'] = test['clean_tweet'].str.lower()
# remove numbers
train['clean_tweet'] = train['clean_tweet'].str.replace("[0-9]", " ")
test['clean_tweet'] = test['clean_tweet'].str.replace("[0-9]", " ")
# remove whitespaces
train['clean_tweet'] = train['clean_tweet'].apply(lambda x:' '.join(x.split()))
test['clean_tweet'] = test['clean_tweet'].apply(lambda x: ' '.join(x.split()))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment