eustin/letor-part-1-model-tokeniser.py

Created May 25, 2020 22:35

Star () You must be signed in to star a gist
Fork () You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/eustin/d3388e80cca81bfe36f7a31076b1bf5a.js"></script>
Save eustin/d3388e80cca81bfe36f7a31076b1bf5a to your computer and use it in GitHub Desktop.

Download ZIP

Raw

letor-part-1-model-tokeniser.py

	tokeniser = tf.keras.preprocessing.text.Tokenizer()
	tokeniser.fit_on_texts(sentences)

	print(tokeniser.word_index)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment