Skip to content

Instantly share code, notes, and snippets.

@aidiary
Created Mar 27, 2020
Embed
What would you like to do?
出現回数を棒グラフで綺麗に表示する
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
sns.set(style='darkgrid')
sns.set(font_scale=1.5)
plt.rcParams['figure.figsize'] = (10, 5)
token_lengths = [len(token) for token in tokenizer.vocab.keys()]
sns.countplot(token_lengths)
plt.title('Vocab Token Lengths')
plt.xlabel('Token Length')
plt.ylabel('# of Tokens')
print('Maximum token length:', max(token_lengths))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment