Skip to content

Instantly share code, notes, and snippets.

@mort
Forked from guillermo/twitter_backup.rb
Created November 3, 2009 09:27
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save mort/224913 to your computer and use it in GitHub Desktop.
Save mort/224913 to your computer and use it in GitHub Desktop.
Tweaking Guillermo's original script to add support for accounts with a huge amount of tweets ;)
require 'twitter'
# TwitterBackup.new(user,pass).backup!
# Will save tweets to files
# ~/.twitter_backup/tweets/tweet_id.yaml # Full yaml tweet
# ~/.twitter_backup/tweets/tweet_id.txt # Only tweet text
# Search locally in your tweets
# cat ~/.twitter_backup/tweets/*.txt | grep -i 'some text'
class TwitterBackup
def initialize(user,pass)
@user,@pass = user, pass
@twitter = Twitter::Base.new(Twitter::HTTPAuth.new(@user,@pass))
end
def total
@twitter.verify_credentials.statuses_count
end
def remaining_hits
@twitter.rate_limit_status.remaining_hits
end
def backup!
puts "Backing up #{total} tweets for #{@user}"
mkdir
page = start_page
res = [1,2,3]
while (res.size > 0)
puts "Fetching page #{page}"
begin
res = @twitter.user_timeline(:page => page)
rescue Twitter::RateLimitExceeded
puts "Twitter API exception: rate limit exceeded"
save_latest_page(page)
exit
end
res.each do |tweet|
puts "Processing tweet #{tweet.id}: #{tweet.text}"
save(tweet)
end
page += 1
end
end
private
def save(tweet)
write_yaml(tweet.id,tweet)
write_txt(tweet.id,tweet.text)
end
def save_latest_page(page)
f = File.open(backup_dir+"/.latest_page",'w')
f.write page
f.close
end
def start_page
f_path = backup_dir+"/.latest_page"
n = if File.exists?(f_path)
f = File.open(f_path, 'r')
v = f.readline
f.close
File.unlink(f_path)
v
else
1
end
n
end
def write_yaml(id,data)
f = File.open(backup_dir+"/#{id.to_s}.yaml",'w')
f.write data.to_yaml
f.close
end
def write_txt(id,data)
f = File.open(backup_dir+"/#{id.to_s}.txt",'w')
f.write data.to_yaml
f.close
end
def backup_dir
File.expand_path("~/.twitter_backup/tweets")
end
def mkdir
`mkdir -p #{backup_dir}`
end
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment