Skip to content

Instantly share code, notes, and snippets.

@andreis
Created Mar 21, 2014
Embed
What would you like to do?
func Tokenize(s string) []string {
out := []string{}
for l, r := 0, 0; r < len(s); {
for l = r; isSep(s[l]); l++ {
}
for r = l; r < len(s) && !isSep(s[r]); r++ {
}
out = append(out, s[l:r])
}
return out
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment