Skip to content

Instantly share code, notes, and snippets.

@jaikoo
Last active December 12, 2015 09:49
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jaikoo/4754975 to your computer and use it in GitHub Desktop.
Save jaikoo/4754975 to your computer and use it in GitHub Desktop.
Answer to a recent LRUG question about downloading of images in parallel. I noticed everyone seemed to shy away from threads, but with j.u.c you get a lot for free including battle hardened concurrency semantics.
#!/usr/bin/env jruby
java_import java.util.concurrent.Executors
require 'net/http'
class ThreadedDownloader
def initialize(poolsize = 50)
@executor = Executors.new_fixed_thread_pool(poolsize)
end
def download(file_list = [])
file_list.each {|f| download_file(f)}
@executor.shutdown if @executor
end
def download_file(src, dest = nil)
@executor.submit do
uri = URI(src)
Net::HTTP.start(uri.host) do |http|
resp = http.get(uri.path)
dest = dest.nil? ? File.basename(uri.path) : dest
open(dest, "wb") do |file|
file.write(resp.body)
puts "Downloaded #{dest}"
end
end
end
end
end
files = [
"http://www.cedmagic.com/featured/he-man/he-man.jpg",
"http://images1.wikia.nocookie.net/__cb20090716165928/heman/images/c/c3/Heman83.jpg",
"http://geektyrant.com/storage/0999-post-images/heman7162012.jpeg?__SQUARESPACE_CACHEVERSION=1342543799392",
"http://www.dvdactive.com/images/reviews/screenshot/2004/8/heman3_copy0.jpg",
"http://media.comicvine.com/uploads/9/98086/1888897-superman_flying.jpg"
]
ThreadedDownloader.new.download(files)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment