Skip to content

Instantly share code, notes, and snippets.

@aravindpai

aravindpai/bpe_10.py

Created May 22, 2020 19:53
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save aravindpai/558b49865ff281b56ba7fbbd8d8a00af to your computer and use it in GitHub Desktop.
Save aravindpai/558b49865ff281b56ba7fbbd8d8a00af to your computer and use it in GitHub Desktop.
Download ZIP
Raw
bpe_10.py
#applying BPE to OOV
oov ='lowest'
#tokenize OOV into characters
oov = " ".join(list(oov))
#append </w>
oov = oov + ' </w>'
#create a dictionary
oov = { oov : 1}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment