Skip to content

Instantly share code, notes, and snippets.

View rsk's full-sized avatar
🎯
Focusing

rsk

🎯
Focusing
View GitHub Profile
@rsk
rsk / text_summary.py
Created November 28, 2023 10:08
text summarization
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, pipeline
def summarize_text(text, model_name):
# Initialize tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
# Encode the text and generate a summary
tokens_input = tokenizer.encode("summarize: " + text, return_tensors='pt', max_length=512, truncation=True)
ids = model.generate(tokens_input, min_length=80, max_length=120)