Skip to content

Instantly share code, notes, and snippets.

@aravindpai
Created May 22, 2020 19:53
Show Gist options
  • Save aravindpai/558b49865ff281b56ba7fbbd8d8a00af to your computer and use it in GitHub Desktop.
Save aravindpai/558b49865ff281b56ba7fbbd8d8a00af to your computer and use it in GitHub Desktop.
#applying BPE to OOV
oov ='lowest'
#tokenize OOV into characters
oov = " ".join(list(oov))
#append </w>
oov = oov + ' </w>'
#create a dictionary
oov = { oov : 1}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment