import tokenizers
tokenizers.__version__
- NLP 관련 다양한 패키지를 제공하고 있으며, 특히 언어 모델 (language models) 을 학습하기 위하여 세 가지 패키지가 유용
package | note |
---|---|
transformers | Transformer 기반 (masked) language models 알고리즘, 기학습된 모델을 제공 |
tokenizers | transformers 에서 사용할 수 있는 토크나이저들을 학습/사용할 수 있는 기능 제공. transformers 와 분리된 패키지로 제공 |
nlp | 데이터셋 및 평가 척도 (evaluation metrics) 을 제공 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Working example for my blog post at: | |
# http://danijar.com/variable-sequence-lengths-in-tensorflow/ | |
import functools | |
import sets | |
import tensorflow as tf | |
from tensorflow.models.rnn import rnn_cell | |
from tensorflow.models.rnn import rnn | |
def lazy_property(function): |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{0: 'tench, Tinca tinca', | |
1: 'goldfish, Carassius auratus', | |
2: 'great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias', | |
3: 'tiger shark, Galeocerdo cuvieri', | |
4: 'hammerhead, hammerhead shark', | |
5: 'electric ray, crampfish, numbfish, torpedo', | |
6: 'stingray', | |
7: 'cock', | |
8: 'hen', | |
9: 'ostrich, Struthio camelus', |