Skip to content

Instantly share code, notes, and snippets.

@bfarzin
Last active August 11, 2019 13:39
Show Gist options
  • Save bfarzin/f9407c0b0f2690f36dd6d51c8ef56944 to your computer and use it in GitHub Desktop.
Save bfarzin/f9407c0b0f2690f36dd6d51c8ef56944 to your computer and use it in GitHub Desktop.
Display the source blob
Display the rendered blob
Raw
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@tanny411
Copy link

tanny411 commented Jun 7, 2019

i believe this is how i should use it:
mycust_tok = CustomTokenizer(SPTokenizer,model_prefix)
But i still have the error. Can you help with the full modified code?

@bfarzin
Copy link
Author

bfarzin commented Jun 7, 2019

itos was wrong also. I updated the example above.

@tanny411
Copy link

tanny411 commented Jun 7, 2019

Thank you so much. Sorry I didnt notice the change earlier. It works. Much appreciated.

@bfarzin
Copy link
Author

bfarzin commented Jun 8, 2019

No problem. I am glad I cleaned it up for my own good!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment