Created
January 17, 2017 22:43
-
-
Save peterc/13858a03d631e7f9046bf1ed40ade113 to your computer and use it in GitHub Desktop.
Downloads Vines associated with a certain account
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# vinescrape.rb goes through an account's Vines and downloads them, skipping revines | |
# I had to write it because I didn't get notified of the download deadline and | |
# with no email address on their site, it wouldn't let me get my archive.. | |
# so I came up with this approach instead :-D | |
# | |
# To use, just change the VINE_USER_ID string below to the long set of digits that makes up YOUR Vine ID | |
# Then ruby vinescrape.rb | |
# Works fine with no dependencies on Ruby 2.4 on OS X, should work fine elsewhere too | |
VINE_USER_ID = "970482265149054976" | |
USER_AGENT = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/#{41 + rand(10)}.0.2228.0 Safari/537.36" | |
require 'open-uri' | |
require 'json' | |
# Given a URL, we go GET it | |
def fetch_data(url) | |
open(url, "User-Agent" => USER_AGENT).read | |
end | |
page = 1 | |
# First, we need to fetch the first 'page' of results | |
url = "https://vine.co/api/timelines/users/#{VINE_USER_ID}?page=#{page}" | |
data = JSON.parse(fetch_data(url)) | |
total_vines = data['data']['count'].to_i | |
STDERR.puts "#{total_vines} vines to fetch" | |
sleep 1 | |
i = 0 | |
downloaded = 0 | |
loop do | |
url = "https://vine.co/api/timelines/users/#{VINE_USER_ID}?page=#{page}" | |
STDERR.puts "Fetching page #{page} of vines" | |
data = JSON.parse(fetch_data(url)) | |
vines = data['data']['records'] | |
vines.each do |vine| | |
i += 1 | |
STDERR.puts "Processing #{i}/#{total_vines}" | |
# Skip "revines" | |
unless vine['userId'].to_s == VINE_USER_ID | |
STDERR.puts " Revine - not downloading" | |
next | |
end | |
# Get the URL of the MP4 video | |
url = vine['videoDashUrl'] | |
# Download and save the video locally | |
STDERR.puts " Downloading #{url}" | |
begin | |
File.open("#{vine['postId']}.mp4", "w") { |f| f.puts fetch_data(url) } | |
STDERR.puts " Downloaded!" | |
downloaded += 1 | |
# Sleep a bit to not look too heavy on the server | |
sleep rand(3) + 1 | |
rescue | |
STDERR.puts " Download FAILED" | |
end | |
end | |
break if i >= total_vines | |
page += 1 | |
sleep rand(2) + 1 | |
end | |
STDERR.puts "#{i} vines processed, #{downloaded} vines downloaded" |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
And yes the code is a mess. I looked at how Vine's private API worked and threw it together in 20 minutes before they stopped me being able to download my Vines. If it breaks for you or if Vine blocks it, sorry!