Skip to content

Instantly share code, notes, and snippets.

@khuangaf

khuangaf/gist:48469a3d0042ce723c7b1bdc9002a7ac

Created March 14, 2020 16:40
Show Gist options
  • Save khuangaf/48469a3d0042ce723c7b1bdc9002a7ac to your computer and use it in GitHub Desktop.
Save khuangaf/48469a3d0042ce723c7b1bdc9002a7ac to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
tokenizer.convert_tokens_to_ids(tokens)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment