Skip to content

Instantly share code, notes, and snippets.

@teward
Forked from joshwand/tumblr-likes-downloader.rb
Last active August 29, 2015 14:08
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save teward/66a4a1ac3d58f1285702 to your computer and use it in GitHub Desktop.
Save teward/66a4a1ac3d58f1285702 to your computer and use it in GitHub Desktop.
Updated version that fixes a couple bugs
require 'tumblr_client'
require 'mechanize'
require 'date'
require 'yaml'
require 'uri'
=begin
To configure:
1) Go to this URL: https://www.tumblr.com/oauth/apps
2) Add an application. Use the following settings:
Application Name: Personal Tumblr likes downloader
Application Website: This gist.
App Store URL: (blank)
Google Play Store URL: (blank)
Application Description: ruby script to download personal likes
Administrative Contact Email: your email address
Default callback URL: This gist.
3) A new application shows up, and shows the OAuth Consumer key.
Click "Explore API" below it and allow the app access.
4) Click the "Ruby" link in the box underneath the Authentication
section. You'll be using the four lines after the line that
reads like this: client = Tumblr::Client.new({
5) The following is the mapping of the ExploreAPI -> the config
block below. Copy and paste the values accordingly:
:APIValue -> Config block variable in this script
:consumer_key -> config.consumer_key
:consumer_secret -> config.consumer_secret
:oauth_token -> config.oauth_token
:oauth_token_secret -> config.oauth_token_secret
=end
Tumblr.configure do |config|
config.consumer_key = "consumer_key"
config.consumer_secret = "consumer_secret"
config.oauth_token = "oauth_token"
config.oauth_token_secret = "oauth_token_secret"
end
THREADPOOL_SIZE = 4
SEGMENT_SIZE = 25
HISTORY_FILE = File.join(File.dirname(__FILE__), "previous.yml")
previous_dled_ids = YAML.load(File.open(HISTORY_FILE)) rescue []
directory = "tumblr-likes"
client = Tumblr::Client.new
likes = client.likes
liked_count = likes["liked_count"]
puts liked_count
likes = []
(0...liked_count).each do |i|
if i==0 or i%SEGMENT_SIZE==0
p "getting #{SEGMENT_SIZE} more likes starting at #{i}: #{likes.count} thus far"
client.likes({:limit => SEGMENT_SIZE, :offset => i})["liked_posts"].each do |like|
likes << like if like['type'] == 'photo' and !previous_dled_ids.include?(like['id'])
end
end
end
if likes.empty?
p "no new likes!"
exit 0
end
puts "#{likes.count} new likes!"
# some of this code comes from https://github.com/jamiew/tumblr-photo-downloader
already_had = 0
threads = []
if likes.count < THREADPOOL_SIZE
SLICE_SIZE = likes.count
else
SLICE_SIZE = likes.count / THREADPOOL_SIZE
end
likes.each_slice(SLICE_SIZE).each do |group|
threads << Thread.new {
begin
p "launching thread #{threads.size}"
group.each do |like|
i = 0
like["photos"].each do |photo|
url = photo["original_size"]["url"]
filename = "#{like["blog_name"]}-#{like["slug"]}-"
filename += "#{i}-" if i > 0
filename += File.basename(URI.parse(url).path.split('?')[0])
if File.exists?("#{directory}/#{filename}")
puts "Already have #{url}"
already_had += 1
else
begin
puts "Saving photo #{url}"
file = Mechanize.new.get(url)
file.save_as("#{directory}/#{filename}")
rescue Mechanize::ResponseCodeError => e
puts "Error #{e.response_code} getting file for #{url}"
end
end
i += 1
previous_dled_ids << like['id']
end
end
rescue Exception => e
puts "unhandled exception:, #{$!}"
end
p "closing thread"
}
end
threads.each{|t| t.join }
YAML.dump(previous_dled_ids, File.open(HISTORY_FILE, "w"))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment