Last active
January 18, 2020 06:43
-
-
Save korakot/1547acb30dc98d967fdf6aff193f8824 to your computer and use it in GitHub Desktop.
Longest matching Thai word tokenization
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from marisa_trie import Trie | |
| # wordlist = ... | |
| trie = Trie(wordlist) | |
| def lmcut(text): | |
| for w in reversed(trie.prefixes(text)): | |
| if w==text: | |
| yield [w] | |
| else: | |
| for ww in lmcut(text[len(w):]): | |
| yield [w]+ww | |
| words = next(lmcut("สวัสดีครับคุณครู")) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment