Skip to content

Instantly share code, notes, and snippets.

@hanneshapke

hanneshapke/adding_of_CLS_and_SEP_tokens.py

Created March 9, 2020 17:47
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save hanneshapke/e657f8df643c174ff8dca81309f6f677 to your computer and use it in GitHub Desktop.
Save hanneshapke/e657f8df643c174ff8dca81309f6f677 to your computer and use it in GitHub Desktop.
Download ZIP
adding_of_CLS_and_SEP_tokens
Raw
adding_of_CLS_and_SEP_tokens.py
CLS_ID = tf.constant(101, dtype=tf.int64)
SEP_ID = tf.constant(102, dtype=tf.int64)
start_tokens = tf.fill([tf.shape(text)[0], 1], CLS_ID)
end_tokens = tf.fill([tf.shape(text)[0], 1], SEP_ID)
tokens = tokens[:, :sequence_length - 2]
tokens = tf.concat([start_tokens, tokens, end_tokens], axis=1)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment