Skip to content

Instantly share code, notes, and snippets.

View bjascob's full-sized avatar

Brad Jascob bjascob

  • Colorado
View GitHub Profile
@bjascob
bjascob / Test_Bart_Perplexity_Token_Level.py
Last active August 22, 2023 12:28
Bart token level perplexity.py
#!/usr/bin/python3
from transformers import AutoModelForMaskedLM, set_seed
from bart_token_level_perplexity import BartPerplexityTester
# Use seq_len=256 as a standard for testing.
if __name__ == '__main__':
device = 'cuda:0'
model_name ='facebook/bart-base'