Skip to content

Instantly share code, notes, and snippets.

@prateekjoshi565

prateekjoshi565/testing_tokenizer_bert.py

Last active July 17, 2020 14:34
Show Gist options
  • Save prateekjoshi565/48bc03ad0e2a66cc3c1b065cecc373db to your computer and use it in GitHub Desktop.
Save prateekjoshi565/48bc03ad0e2a66cc3c1b065cecc373db to your computer and use it in GitHub Desktop.
Download ZIP
Raw
testing_tokenizer_bert.py
# sample data
text = ["this is a bert model tutorial", "we will fine-tune a bert model"]
# encode text
sent_id = tokenizer.batch_encode_plus(text, padding=True)
# output
print(sent_id)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment