Skip to content

Instantly share code, notes, and snippets.

@mheffner
Created October 3, 2012 18:11
Show Gist options
  • Save mheffner/3828710 to your computer and use it in GitHub Desktop.
Save mheffner/3828710 to your computer and use it in GitHub Desktop.
Upload Travis CI builds to S3

Upload Travis CI builds to S3

This will demonstrate how to upload build files from Travis CI to S3.

NOTE: Keys have been changed to protect the innocent.

Step 1: Create an S3 policy.

Create an S3 ACL policy, see s3_policy.json for an example.

Step 2: Generate the policy and signature.

Use gen_upload_policy.php to generate the base64 policy and signature. This requires your S3 secret key to generate the signature:

$ ./gen_upload_policy.php ./s3_policy.json lvbpI6ICIyMDA4LTErLTAxVDtyO+m202n03
S3_POLICY="eyAiZXhwaXshdGlvbpI6ICIyMDA4LTErLTAxVDtyOjAwOjAwLjAsMFoiLAogICJjb25kaXRpb25zPjogWwoJeyJidWNrZXQiOiAiczNwaG90b3MubW9hcHAubmV0IiB9LAogICAgWyJzdGFydHMtd2l0aCIsICIka2V5IiwgIkxpdmVTaG90cy8iXSwKICAgIHsiYWNsIjogInB1YmxpYy1yZWFkIiB9LAoJWyJlcSIsICIkQ29udGVudC1UeXBlIiwgImltYWdlL2pwZWciXSwKICBdCn0K"
S3_SIGNATURE="20uh08kU75ADHL49NyhYRgZW8BY="

You'll use the S3_POLICY and S3_SIGNATURE variables in your .travis.yml config file.

Step 3: Setup .travis.yml.

Take the variables from above and set them as environment variables in your .travis.yml. Also set the S3_ACCESS_KEY to your S3 access key (not the secret one) and S3_BUCKET to the name of the S3 bucket. See the example .travis.yml file for details.

Step 4: Create the upload script

Create an upload script that uploads files to the appropriate S3 bucket. See the example s3_upload.sh that uploads all *.jar files under the top-level directory target/ to S3.

Add the script location to the .travis.yml after_success hook.

language: java
jdk:
- openjdk6
env:
global:
- S3_POLICY="eyAiZXhwaXshdGlvbpI6ICIyMDA4LTErLTAxVDtyOjAwOjAwLjAsMFoiLAogICJjb25kaXRpb25zPjogWwoJeyJidWNrZXQiOiAiczNwaG90b3MubW9hcHAubmV0IiB9LAogICAgWyJzdGFydHMtd2l0aCIsICIka2V5IiwgIkxpdmVTaG90cy8iXSwKICAgIHsiYWNsIjogInB1YmxpYy1yZWFkIiB9LAoJWyJlcSIsICIkQ29udGVudC1UeXBlIiwgImltYWdlL2pwZWciXSwKICBdCn0K"
- S3_SIGNATURE="20uh08kU75ADHL49NyhYRgZW8BY="
- S3_ACCESS_KEY="2EO6H8MX1X8YWEA0V432"
- S3_BUCKET="johnsmith"
after_success: ./scripts/s3upload.sh
#!/usr/bin/env php
<?php
/*
* From: http://raamdev.com/2008/amazon-s3-hmac-signatures-without-pear-or-php5/
*/
/*
* Calculate HMAC-SHA1 according to RFC2104
* See http://www.faqs.org/rfcs/rfc2104.html
*/
function hmacsha1($key,$data) {
$blocksize=64;
$hashfunc='sha1';
if (strlen($key)>$blocksize)
$key=pack('H*', $hashfunc($key));
$key=str_pad($key,$blocksize,chr(0x00));
$ipad=str_repeat(chr(0x36),$blocksize);
$opad=str_repeat(chr(0x5c),$blocksize);
$hmac = pack(
'H*',$hashfunc(
($key^$opad).pack(
'H*',$hashfunc(
($key^$ipad).$data
)
)
)
);
return bin2hex($hmac);
}
/*
* Used to encode a field for Amazon Auth
* (taken from the Amazon S3 PHP example library)
*/
function hex2b64($str)
{
$raw = '';
for ($i=0; $i < strlen($str); $i+=2)
{
$raw .= chr(hexdec(substr($str, $i, 2)));
}
return base64_encode($raw);
}
if (count($argv) != 3) {
echo "Usage: " . $argv[0] . " <S3 Policy File> <S3 secret key>\n";
exit(1);
}
$policy = file_get_contents($argv[1]);
$secret = $argv[2];
/*
* Base64 encode the Policy Document and then
* create HMAC SHA-1 signature of the base64 encoded policy
* using the secret key. Finally, encode it for Amazon Authentication.
*/
$base64_policy = base64_encode($policy);
$signature = hex2b64(hmacsha1($secret, $base64_policy));
echo "S3_POLICY=\"" . $base64_policy . "\"\nS3_SIGNATURE=\"" . $signature . "\"\n"
?>
{
"expiration": "2015-01-01T12:00:00.000Z",
"conditions": [
{"acl": "private" },
{"bucket": "johnsmith" },
["starts-with", "$key", "jars/"],
["content-length-range", 2048, 268435456],
["eq", "$Content-Type", "application/octet-stream"]
]
}
#!/bin/bash
# From:
# http://raamdev.com/2008/using-curl-to-upload-files-via-post-to-amazon-s3/
GIT_VERSION=`git describe`
for FILE in `ls target/*.jar`; do
BASE=`basename $FILE`
curl \
-F "key=jars/$GIT_VERSION/$BASE" \
-F "acl=private" \
-F "AWSAccessKeyId=$S3_ACCESS_KEY" \
-F "Policy=$S3_POLICY" \
-F "Signature=$S3_SIGNATURE" \
-F "Content-Type=application/octet-stream" \
-F "file=@$FILE" \
https://s3.amazonaws.com/$S3_BUCKET
if [ $? -ne 0 ]; then
exit 1
fi
done
exit 0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment