Created

Embed URL

HTTPS clone URL

SSH clone URL

You can clone with HTTPS or SSH.

Download Gist

Download all tweets from the twitter search API for a given search term (limited to their maximum of 1500)

View scrape_tweets.rb
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
#! /usr/bin/env ruby
require "fileutils"
search_term = ARGV[0]
if search_term
time = Time.now
directory_path = File.dirname(__FILE__) + "/tweets/" + search_term + "_" + time.to_i.to_s
FileUtils.mkdir_p(directory_path)
directory = Dir.new(directory_path)
(1..15).each do |i|
`curl "http://search.twitter.com/search.json?q=#{search_term}&rpp=100&page=#{i}&include_entities=true&result_type=mixed" > #{directory.path}/#{i}.json`
end
puts "Scraped to #{directory_path}"
else
puts "./scrape_tweets.rb <your search term in quotes>"
end

Can you make it download images too?

chid commented

To anyone who finds this, note that this does not work anymore as Twitter has disabled their 1.0 API and now requires authentication for searches https://dev.twitter.com/docs/auth

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.