yemba_dataset2.py

max_word_len = df.yb.str.len().max()
max_word_len_utf8 = df.yb_utf8.str.len().max()
nb_labels = len(df.word_type.unique())
nb_words = df.shape[0]
print("Number of words: ", nb_words)
print("Number of labels: ", nb_labels)
print("Max word length: {} characters and {} bytes".format(max_word_len, max_word_len_utf8))

Further, can I ask how do we usually handle the situation where the training data misses some label categories while missing categories could happen in production?

@pancodia thanks for getting back to me. Sorry for the late reply. I was traveling. Did you find a fix? If yes, can you share or do you still need help?