Skip to content

Instantly share code, notes, and snippets.

@aravindpai

aravindpai/bpe_11.py

Created May 22, 2020
Embed
What would you like to do?
i=0
while(True):
#compute frequency
pairs = get_stats(oov)
#extract keys
pairs = pairs.keys()
#find the pairs available in the learned operations
ind=[merges.index(i) for i in pairs if i in merges]
if(len(ind)==0):
print("\nBPE Completed...")
break
#choose the most frequent learned operation
best = merges[min(ind)]
#merge the best pair
oov = merge_vocab(best, oov)
print("Iteration ",i+1, list(oov.keys())[0])
i=i+1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment