Skip to content

Instantly share code, notes, and snippets.

@backupbrain

backupbrain/tokenizer_run.js

Created October 15, 2018 18:53
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save backupbrain/2c992f03acc4cc4c55f284090c461110 to your computer and use it in GitHub Desktop.
Save backupbrain/2c992f03acc4cc4c55f284090c461110 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
tokenizer_run.js
Tokenizer.prototype.run = function(text) {
formattedText = this.prepareCodeForTokenization(text);
tokens = this.getTokenLines(formattedText);
return tokens;
}
Tokenizer.prototype.prepareCodeForTokenization = function(text) {
text = this.stripComments(text);
text = this.trimWhitespace(text);
text = this.resetNewlines(text);
text = this.padTokens(text);
return text;
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment