Skip to content

Instantly share code, notes, and snippets.

@AdroitAnandAI

AdroitAnandAI/wordcount.py

Created June 6, 2021 05:47
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save AdroitAnandAI/f61e2a5e7b8af20807cb608382b1f29b to your computer and use it in GitHub Desktop.
Save AdroitAnandAI/f61e2a5e7b8af20807cb608382b1f29b to your computer and use it in GitHub Desktop.
Download ZIP
To count words
Raw
wordcount.py
# Word Count using Spark
# https://spark.apache.org/examples.html
text_file = sc.textFile("./sample.txt") //local file
counts = text_file.flatMap(lambda line: line.split(" ")) \
.map(lambda word: (word, 1)) \
.reduceByKey(lambda a, b: a + b)
counts.saveAsTextFile("./count_output")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment