kabirahuja2431/cls_sep.py

Last active October 8, 2019 14:35

Star () You must be signed in to star a gist
Fork () You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/kabirahuja2431/5cd0f8f0b7bcd0b1e77f5d7b358f3af5.js"></script>
Save kabirahuja2431/5cd0f8f0b7bcd0b1e77f5d7b358f3af5 to your computer and use it in GitHub Desktop.

Download ZIP

Raw

	tokens = ['[CLS]'] + tokens + ['[SEP]']
	print(tokens)
	# Out: ['[CLS]', 'i', 'really', 'enjoyed', 'this', 'movie', 'a', 'lot', '.', '[SEP]']

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment