curl() connection

curl.R

# Synchronous implementation of curl() connection
curl <- function(url){
  con <- rawConnection(raw(0), "r+");
  cat("Downloading ")
  req <- httr::GET(url, httr::write_stream(function(x){
    cat(".")
    writeBin(x, con)
  }))
  httr::stop_for_status(req)
  seek(con, 0)
  con
}

# Does not parse-stream. Downloads all data first.
diamonds <- jsonlite::stream_in(curl("https://jeroenooms.github.io/data/diamonds.json"))

# Also does not support gzip
flights <- jsonlite::stream_in(gzcon(curl("https://jeroenooms.github.io/data/flights.json.gz")))

You'd have to put the stream_in() inside the write_stream() in order for it to do what you wanted. (Or provide a completely different implementation based on event streams, possibly built on top of shiny)

I'm surprised the second one doesn't work because curl normally handles the unzipping.

Making it behave like a real connection would be hard

Yes that was I was afraid of. Doing the incremental parsing in the callback is ugly though, makes things much more complicated for the user. Would be fine for javascript, but not really for R. I really like the behavior of connections. Maybe I'll ask on r-devel if there would be a way to implement this.

This is now possible with the new curl package.