Skip to content

Instantly share code, notes, and snippets.

@AyishaR

AyishaR/sarcasm_tokenization.py

Created February 19, 2021 07:11
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save AyishaR/fbe0c8883a7460dba410f253ecf830f1 to your computer and use it in GitHub Desktop.
Save AyishaR/fbe0c8883a7460dba410f253ecf830f1 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
sarcasm_tokenization.py
# Tokenization
vocab = 1500
mlen = 200
tokenizer = Tokenizer(num_words = vocab, oov_token = '<UNK>')
tokenizer.fit_on_texts(Xtrain)
Xtrain = tokenizer.texts_to_sequences(Xtrain)
Xtrain = pad_sequences(Xtrain, maxlen=mlen)
Xval = tokenizer.texts_to_sequences(Xval)
Xval = pad_sequences(Xval, maxlen=mlen)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment