Skip to content

Instantly share code, notes, and snippets.

@aravindpai

aravindpai/bpe_10.py

Created May 22, 2020
Embed
What would you like to do?
#applying BPE to OOV
oov ='lowest'
#tokenize OOV into characters
oov = " ".join(list(oov))
#append </w>
oov = oov + ' </w>'
#create a dictionary
oov = { oov : 1}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment