HarshSingh16/Tokens.py

Created February 8, 2019 18:55

Star () You must be signed in to star a gist
Fork () You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/HarshSingh16/0722ac13318618ebbccbbd6861f71387.js"></script>
Save HarshSingh16/0722ac13318618ebbccbbd6861f71387 to your computer and use it in GitHub Desktop.

Download ZIP

Raw

	#Adding tokens to our dictionary
	Tokens=["<PAD>","<SOS>","<EOS>","<OUT>"]
	for token in Tokens:
	dict_integer[token]=len(dict_integer)+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment