Skip to content

Instantly share code, notes, and snippets.

@a7v8x

a7v8x/tokenization_output.txt

Created May 5, 2020 18:35
Show Gist options
  • Save a7v8x/84ddadf9e9cf69df508da637c7826756 to your computer and use it in GitHub Desktop.
Save a7v8x/84ddadf9e9cf69df508da637c7826756 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
tokenization_output.txt
tokenized ['[CLS]', 'test', 'token', '##ization', 'sentence', '.', 'followed', 'by', 'another', 'sentence', '[SEP]']
{
'token_ids': [101, 3231, 19204, 3989, 6251, 1012, 2628, 2011, 2178, 6251, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0],
'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment