Skip to content

Instantly share code, notes, and snippets.

@manmohan24nov

manmohan24nov/xlnet_summerize.py

Created November 3, 2020 20:19
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save manmohan24nov/b32bc2422b0b6633f0126aca77517110 to your computer and use it in GitHub Desktop.
Save manmohan24nov/b32bc2422b0b6633f0126aca77517110 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
xlnet_summerize.py
>>> from summarizer import TransformerSummarizer
>>> import re
>>> xlnet_model = TransformerSummarizer(transformer_type="XLNet",transformer_model_key="xlnet-base-cased")
>>> text = " ".join(tweet_data)
>>> TEXT_CLEANING_RE = "@\S+|https?:\S+|http?:\S|[^A-Za-z0-9]+"
>>> text = re.sub(TEXT_CLEANING_RE, ' ', str(text).lower()).strip()
>>> summerize = ''.join(xlnet_model(text, min_length=60, max_length=120))
>>> summerize
"The fixwithohimai and chidiodinkalu look ahead to tomorrow's presidential election. The uselections2020 overnight show will feature guests on both sides of the at trump s defeat.
A new poll shows potus leading in one of the most important swing states pennsylvania."
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment