Skip to content

Instantly share code, notes, and snippets.

@guillermo
Forked from mort/twitter_backup.rb
Created December 10, 2011 12:05
Show Gist options
  • Save guillermo/1454999 to your computer and use it in GitHub Desktop.
Save guillermo/1454999 to your computer and use it in GitHub Desktop.
Tweaking mort's fork of the Tweaking Guillermo's original script to add support for accounts with a huge amount of tweets ;), to use the new twitter api.
#!/usr/bin/env ruby
=begin
DESCRIPTION
This little program will save all tweets from a user timelime to files
~/.twitter_backup/tweets/tweet_id.yaml # Full yaml tweet
~/.twitter_backup/tweets/tweet_id.txt # Only tweet text
You can search your tweets with:
cat ~/.twitter_backup/tweets/*.txt | grep -i 'some text'
or
grep -rin "keyword" ~/.twitter_backup/tweets/*.txt
INSTALL
$ chmod +x ./twitter_backup.rb
EXECUTION
$ ./twitter_backup.rb user_name
FROM RUBY
TwitterBackup.new(user).backup!
NOTES
To get higher rate limit, set this values and move to the code section. :-P
Twitter.configure do |config|
config.consumer_key = ""
config.consumer_secret = ""
config.oauth_token = ""
config.oauth_token_secret = ""
end
=end
require 'rubygems'
require 'twitter'
class TwitterBackup
def initialize(user)
@user = user
end
def total
Twitter.user(@user).statuses
end
def remaining_hits
Twitter.rate_limit_status.remaining_hits
end
def backup!
puts "Backing up #{total} tweets for #{@user}"
mkdir
page = start_page.to_i
begin
puts "Fetching page #{page}"
begin
res = Twitter.user_timeline(@user, :count => 200, :trim_user => true, :page => page)
rescue Twitter::Error => e
puts "Twitter API exception: #{e.message}"
save_latest_page(page)
if(e.retry_after == 0)
puts "Retrying"
else
puts "Retrying after #{e.retry_after} seconds"
sleep e.retry_after
end
retry
end
res.each do |tweet|
puts "Processing tweet #{tweet.id}: #{tweet.text}"
save(tweet)
end
page += 1
end while (res.size > 0)
ensure
save_latest_page(page)
end
private
def save(tweet)
write_yaml(tweet.id,tweet)
write_txt(tweet.id,tweet.text)
end
def save_latest_page(page)
f = File.open(backup_dir+"/.latest_page",'w')
f.write page
f.close
end
def start_page
f_path = backup_dir+"/.latest_page"
n = if File.exists?(f_path)
f = File.open(f_path, 'r')
v = f.readline
f.close
File.unlink(f_path)
v
else
1
end
n
end
def write_yaml(id,data)
f = File.open(backup_dir+"/#{id.to_s}.yaml",'w')
f.write data.to_yaml
f.close
end
def write_txt(id,data)
f = File.open(backup_dir+"/#{id.to_s}.txt",'w')
f.write data.to_yaml
f.close
end
def backup_dir
File.expand_path("~/.twitter_backup/tweets")
end
def mkdir
`mkdir -p #{backup_dir}`
end
end
if $0 == __FILE__
TwitterBackup.new(ARGV.join(" ")).backup!
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment