Skip to content

Instantly share code, notes, and snippets.

@pdsphil
Created September 19, 2009 20:15
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save pdsphil/189563 to your computer and use it in GitHub Desktop.
Save pdsphil/189563 to your computer and use it in GitHub Desktop.
#!/usr/bin/ruby
# must place in dir containing the wukong code
$: << File.dirname(__FILE__)+'/wukong/lib'
require 'wukong'
module WordCount
class Mapper < Wukong::Streamer::LineStreamer
# Emit each word in the line.
def process line
words = line.strip.split(" ").reject(&:blank?)
yield [words[4], 1]
end
end
class Reducer < Wukong::Streamer::ListReducer
def finalize
total = 0
counts = values.map(&:last).map(&:to_i)
counts.each {|x| total += x}
yield [ key, total ]
end
end
end
Wukong::Script.new(WordCount::Mapper, WordCount::Reducer).run
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment