Skip to content

Instantly share code, notes, and snippets.

@c7x43t

c7x43t/word_tokenizer_regex.js

Last active October 2, 2021 16:02
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save c7x43t/2df558e6fcc2f07d3b8b9b6023904b9f to your computer and use it in GitHub Desktop.
Save c7x43t/2df558e6fcc2f07d3b8b9b6023904b9f to your computer and use it in GitHub Desktop.
Download ZIP
Raw
word_tokenizer_regex.js
var reg=/(?:^|\s?)([\w\u00C0-\u017Fa-zA-Z-']+)(?:$|\s?)/g;
// __
function tokenize(text){
var tokens = Array.from(text.matchAll(/(?:^|\s?)([\w\u00C0-\u017Fa-zA-Z-']+)(?:$|\s?)/g)).map(e=>e[1]);
return tokens;
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment