Skip to content

Instantly share code, notes, and snippets.

@aravindpai

aravindpai/summarytokenizer.py

Last active May 31, 2019 13:04
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save aravindpai/892ac5df032c3167482e8a36d0ff16ec to your computer and use it in GitHub Desktop.
Save aravindpai/892ac5df032c3167482e8a36d0ff16ec to your computer and use it in GitHub Desktop.
Download ZIP
Raw
summarytokenizer.py
#preparing a tokenizer for summary on training data
y_tokenizer = Tokenizer()
y_tokenizer.fit_on_texts(list(y_tr))
#convert summary sequences into integer sequences
y_tr = y_tokenizer.texts_to_sequences(y_tr)
y_val = y_tokenizer.texts_to_sequences(y_val)
#padding zero upto maximum length
y_tr = pad_sequences(y_tr, maxlen=max_len_summary, padding='post')
y_val = pad_sequences(y_val, maxlen=max_len_summary, padding='post')
y_voc_size = len(y_tokenizer.word_index) +1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment