Skip to content

Instantly share code, notes, and snippets.

@somyamohanty

somyamohanty/text_collocation.py

Created August 14, 2014 03:18
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save somyamohanty/12b14dea5fb0f4eedffa to your computer and use it in GitHub Desktop.
Save somyamohanty/12b14dea5fb0f4eedffa to your computer and use it in GitHub Desktop.
Download ZIP
Raw
text_collocation.py
import nltk
docs = ['hi this is a test', 'testing done for now', 'today is a test']
docs_l = []
for sentence in docs:
tokens = nltk.word_tokenize(sentence)
docs_l.append(tokens)
finder = BigramCollocationFinder.from_documents(docs_l)
bigram_measures = nltk.collocations.BigramAssocMeasures()
print(finder.score_ngrams(bigram_measures.raw_freq))
finder.ngram_fd.viewitems()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment