Skip to content

Instantly share code, notes, and snippets.

@bluedistro

bluedistro/save_tokenizer.py

Created January 1, 2019 19:23
Show Gist options
  • Save bluedistro/907dfd4957b01fa790421527fcf1c106 to your computer and use it in GitHub Desktop.
Save bluedistro/907dfd4957b01fa790421527fcf1c106 to your computer and use it in GitHub Desktop.
Download ZIP
save tokenizer
Raw
save_tokenizer.py
# save the tokenizer
with open(os.path.join(tokenizer_path, 'tokenizer_m1.pickle'), 'wb') as handle:
pk.dump(tokenizer, handle, protocol=pk.HIGHEST_PROTOCOL)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment