Skip to content

Instantly share code, notes, and snippets.

@AyishaR

AyishaR/question_tokenization.py

Created January 30, 2021 17:46
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save AyishaR/d3ed51c699116218cfd28ba638fb4daa to your computer and use it in GitHub Desktop.
Save AyishaR/d3ed51c699116218cfd28ba638fb4daa to your computer and use it in GitHub Desktop.
Download ZIP
Raw
question_tokenization.py
# Tokenization
vocab = 1500
mlen = 200
tokenizer = Tokenizer(num_words = vocab, oov_token = '<UNK>')
tokenizer.fit_on_texts(Xtrain)
Xtrain = tokenizer.texts_to_sequences(Xtrain)
Xtrain = pad_sequences(Xtrain, maxlen=mlen)
Xval = tokenizer.texts_to_sequences(Xval)
Xval = pad_sequences(Xval, maxlen=mlen)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment