Using Cloud Storage resumable upload with a signed URL in Ruby

stream_upload.rb

# Copyright 2018 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# A Google Cloud Storage streaming upload example using Signed URLs in Ruby.
# https://cloud.google.com/storage/docs/xml-api/resumable-upload
# https://cloud.google.com/storage/docs/access-control/create-signed-urls-program
# https://www.github.com/GoogleCloudPlatform/google-cloud-ruby
# Disclaimer: This example doesn't have connection interruption retries 

# Requirements
# Install google-cloud-storage gem
# gem install google-cloud-storage

# Example usage:
# cat file.ext | ruby stream_upload.rb "bucket-name" "file-name" "text/plain"

require "google/cloud/storage"
require "net/http"

unless ARGV.size == 3
  puts 'Example usage: ruby stream_upload.rb "bucket-name" "file-name" "text/plain"'
  exit
end
  
storage = Google::Cloud::Storage.new
bucket = storage.bucket ARGV.shift
file = bucket.file ARGV.shift, skip_lookup: true # file may not exist.
signed_url = file.signed_url(method: "POST",
                             content_type: ARGV.shift,
                             headers: {"x-goog-resumable":"start"})

# Ruby example of getting the resumable session.
# Create the HTTP object
uri = URI.parse(signed_url)
https = Net::HTTP.new(uri.host, uri.port)
https.use_ssl = true
# https.set_debug_output $stderr # debugging purposes only

# Prepare POST request to get resumable session
headers = {"x-goog-resumable": "start"}
request = Net::HTTP::Post.new(uri.request_uri, headers)
request.content_type = "text/plain"

# Send POST request to get resumable session URL.
response = https.request(request)

# Process response
resumable_session_url = response["location"]

# Continue with PUTs
# chunk_size can be increased but uploaded chunks need to be a multiple of 256 kibibytes or 262,144 bytes.
chunk_size = 256*1024
last_byte = 0
total_bytes = 0
headers = {}

file = STDIN
while !file.eof?
  chunk = file.read(chunk_size)
  total_bytes += chunk.size

  if chunk.size == chunk_size && !file.eof?
    headers["Content-Range"] = "bytes #{last_byte}-#{last_byte+chunk_size-1}/*"
  else
    headers["Content-Range"] = "bytes #{last_byte}-#{last_byte+chunk.size-1}/#{total_bytes}"
  end
  headers["Content-Length"] = chunk.size.to_s
  last_byte += chunk.size

  request = Net::HTTP::Put.new(resumable_session_url, headers)
  request.content_type = "text/plain"
  request.body = chunk
  response = https.request(request)

  case response.code
  when "200"
    puts "Completed upload"
  when "308"
    puts response["Range"]
  else
    raise "Unhandled response code #{response.code}"
  end
end

@frankyn Wow, that you so much for posting this!! It is based on Resumable Uploads with the XML API, correct? Is there a reason you didn't implement Step 4—Query Google Cloud Storage for the upload status? (I feel bad asking, because this was clearly already a lot of work!)

I used Resumable Uploads of Unknown Size which returns a 308 status in response received to track uploaded bytes otherwise it may fail with one of the following interruptions listed in recommended practices. This example doesn't provide interruption handling at the moment.

This example doesn't provide interruption handling at the moment.

OK, gotcha!

Thanks @frankyn This helps me a lot as Google have not put lot of effort in documenting this properly and are not that helpful!