Skip to content

Instantly share code, notes, and snippets.

@isalgueiro

isalgueiro/parallel_http.R

Created November 29, 2017 13:06
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save isalgueiro/36a08947bc43f8fc63f20c3c2d6b1f1d to your computer and use it in GitHub Desktop.
Save isalgueiro/36a08947bc43f8fc63f20c3c2d6b1f1d to your computer and use it in GitHub Desktop.
Download ZIP
HOW-TO get parallel HTTP connections in R
Raw
parallel_http.R
library(parallel)
library(httr)
library(RCurl)
detectCores()
f <- function(x) {
#h <- httr::handle('http://slowwly.robertomurray.co.uk/')
#httr::GET('http://slowwly.robertomurray.co.uk/delay/2000/url/http://www.google.co.uk', handle = h)
RCurl::httpGET('http://slowwly.robertomurray.co.uk/delay/2000/url/http://www.google.co.uk')
#Sys.sleep(2)
return(T)
}
vf <- Vectorize(f)
x <- 1:6
system.time(pvec(x, vf, mc.cores = 1))
system.time(pvec(x, vf, mc.cores = 2))
system.time(pvec(x, vf, mc.cores = 4))
system.time(pvec(x, vf, mc.cores = 6))
@isalgueiro
Copy link
Author

isalgueiro commented Nov 29, 2017 •
edited

Using httr I couldn't manage to make it launch parallel HTTP connections; mc.cores = 6 execution is almost as slow as mc.cores = 1. With RCurl parallel results are faster.

httr 6 cores

> system.time(pvec(x, vf, mc.cores = 6))
   user  system elapsed 
  0.024   0.080  12.201

RCurl 6 cores

> system.time(pvec(x, vf, mc.cores = 6))
   user  system elapsed 
  0.048   0.080   2.260

More info:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment