Skip to content

Instantly share code, notes, and snippets.

@backupbrain

backupbrain/tokenizer_reset_new_lines.js

Created October 15, 2018 18:51
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save backupbrain/b992459191ef74d47af92365bec0ba49 to your computer and use it in GitHub Desktop.
Save backupbrain/b992459191ef74d47af92365bec0ba49 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
tokenizer_reset_new_lines.js
Tokenizer.prototype.resetNewlines = function(text) {
nlText = text.replace(/\n/g, "").replace(/\;/g, ";\n");
nlText = nlText.replace(/\}/g, "}\n").replace(/\{/g, "\n{\n");
nlText = nlText.replace(/\n[ ]+/g, "\n");
nlText = nlText.replace(/\n$/, "");
return nlText;
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment