Skip to content

Instantly share code, notes, and snippets.

@aravindpai

aravindpai/texttokenizer.py

Last active May 31, 2019 13:03
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save aravindpai/f4fbfe97ff7c9907f45f4db8a361d384 to your computer and use it in GitHub Desktop.
Save aravindpai/f4fbfe97ff7c9907f45f4db8a361d384 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
texttokenizer.py
#prepare a tokenizer for reviews on training data
x_tokenizer = Tokenizer()
x_tokenizer.fit_on_texts(list(x_tr))
#convert text sequences into integer sequences
x_tr = x_tokenizer.texts_to_sequences(x_tr)
x_val = x_tokenizer.texts_to_sequences(x_val)
#padding zero upto maximum length
x_tr = pad_sequences(x_tr, maxlen=max_len_text, padding='post')
x_val = pad_sequences(x_val, maxlen=max_len_text, padding='post')
x_voc_size = len(x_tokenizer.word_index) +1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment