Skip to content

Instantly share code, notes, and snippets.

@stefl

stefl/scrape_tweets.rb

Created March 25, 2012 14:47
Show Gist options
  • Star 7 You must be signed in to star a gist
  • Fork 3 You must be signed in to fork a gist
  • Save stefl/2196581 to your computer and use it in GitHub Desktop.
Save stefl/2196581 to your computer and use it in GitHub Desktop.
Download ZIP
Download all tweets from the twitter search API for a given search term (limited to their maximum of 1500)
Raw
scrape_tweets.rb
#! /usr/bin/env ruby
require "fileutils"
search_term = ARGV[0]
if search_term
time = Time.now
directory_path = File.dirname(__FILE__) + "/tweets/" + search_term + "_" + time.to_i.to_s
FileUtils.mkdir_p(directory_path)
directory = Dir.new(directory_path)
(1..15).each do |i|
`curl "http://search.twitter.com/search.json?q=#{search_term}&rpp=100&page=#{i}&include_entities=true&result_type=mixed" > #{directory.path}/#{i}.json`
end
puts "Scraped to #{directory_path}"
else
puts "./scrape_tweets.rb <your search term in quotes>"
end
@jadedgnome
Copy link

Can you make it download images too?

@chid
Copy link

chid commented May 23, 2014

To anyone who finds this, note that this does not work anymore as Twitter has disabled their 1.0 API and now requires authentication for searches https://dev.twitter.com/docs/auth

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment