Skip to content

Instantly share code, notes, and snippets.

@stask
Last active December 26, 2015 05:49
Show Gist options
  • Save stask/7103154 to your computer and use it in GitHub Desktop.
Save stask/7103154 to your computer and use it in GitHub Desktop.
Downloading non-public files via REST API from S3 in Clojure
(ns agent-smith.s3
(:require [clojure.data.codec.base64 :as b64]
[clojure.string :as s]
[clojure.java.io :refer [copy file]]
[clj-http.client :as client]
[clj-time.core :as t]
[clj-time.format :as tf]))
(defn- sign [data key]
(let [mac (javax.crypto.Mac/getInstance "HmacSHA1")
key-spec (javax.crypto.spec.SecretKeySpec. (.getBytes key "UTF8") "HmacSHA1")]
(.init mac key-spec)
(->> (.getBytes data "UTF8")
(.doFinal mac)
b64/encode
String.)))
(defn- get-resource [{:keys [api-key api-secret-key]} path options]
(let [[_ bucket path] (re-find #"^s3://([^/]+)/(.+)$" path)
timestamp (tf/unparse (tf/formatters :rfc822) (t/now))
request (s/join \newline ["GET" "" "" timestamp (str "/" bucket "/" path)])
authorization (str "AWS " api-key ":" (sign request api-secret-key))
url (str "https://" bucket ".s3.amazonaws.com/" path)
headers {"Date" timestamp, "Authorization" authorization}]
(client/get url (assoc options :headers headers))))
(defn download
"Downloads a file from S3. Returns clj-http response.
The remote path should be s3://BUCKET_NAME/PATH/TO/OBJECT."
[aws-creds remote-path]
(get-resource aws-creds remote-path {:as :byte-array}))
(defn download-to
"Downloads a file from S3 to local path.
The remote path should be s3://BUCKET_NAME/PATH/TO/OBJECT."
[aws-creds remote-path local-path]
(let [resource (get-resource aws-creds remote-path {:as :stream})]
(copy (:body resource) (file local-path))))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment