Skip to content

Instantly share code, notes, and snippets.

@schio

schio/add_cls_and_sep.py

Created January 18, 2020 15:22
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save schio/16109d80280cca87112050ba00de8899 to your computer and use it in GitHub Desktop.
Save schio/16109d80280cca87112050ba00de8899 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
add_cls_and_sep.py
bert_tokens = []
bert_tokens.append("[CLS]")
for token in tokens:
bert_tokens.append(token)
bert_tokens.append("[SEP]")
bert_tokens
>>> ['[CLS]', 'this', 'here', "'", 's', 'an', 'example', 'of', 'using', 'the', 'bert', 'token', '##izer', '[SEP]']
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment