public
Last active

Answer to a recent LRUG question about downloading of images in parallel. I noticed everyone seemed to shy away from threads, but with j.u.c you get a lot for free including battle hardened concurrency semantics.

  • Download Gist
gistfile1.rb
Ruby
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
#!/usr/bin/env jruby
java_import java.util.concurrent.Executors
 
require 'net/http'
 
class ThreadedDownloader
 
def initialize(poolsize = 50)
@executor = Executors.new_fixed_thread_pool(poolsize)
end
 
def download(file_list = [])
file_list.each {|f| download_file(f)}
@executor.shutdown if @executor
end
 
def download_file(src, dest = nil)
@executor.submit do
uri = URI(src)
Net::HTTP.start(uri.host) do |http|
resp = http.get(uri.path)
dest = dest.nil? ? File.basename(uri.path) : dest
open(dest, "wb") do |file|
file.write(resp.body)
puts "Downloaded #{dest}"
end
end
end
end
end
 
files = [
"http://www.cedmagic.com/featured/he-man/he-man.jpg",
"http://images1.wikia.nocookie.net/__cb20090716165928/heman/images/c/c3/Heman83.jpg",
"http://geektyrant.com/storage/0999-post-images/heman7162012.jpeg?__SQUARESPACE_CACHEVERSION=1342543799392",
"http://www.dvdactive.com/images/reviews/screenshot/2004/8/heman3_copy0.jpg",
"http://media.comicvine.com/uploads/9/98086/1888897-superman_flying.jpg"
]
 
ThreadedDownloader.new.download(files)

Please sign in to comment on this gist.

Something went wrong with that request. Please try again.