Skip to content

Instantly share code, notes, and snippets.

@chrismdp
Last active March 5, 2024 12:57
Show Gist options
  • Save chrismdp/6c6b6c825b07f680e710 to your computer and use it in GitHub Desktop.
Save chrismdp/6c6b6c825b07f680e710 to your computer and use it in GitHub Desktop.
Uploading to S3 in 18 lines of Shell (used to upload builds for http://soltrader.net)
# You don't need Fog in Ruby or some other library to upload to S3 -- shell works perfectly fine
# This is how I upload my new Sol Trader builds (http://soltrader.net)
# Based on a modified script from here: http://tmont.com/blargh/2014/1/uploading-to-s3-in-bash
S3KEY="my aws key"
S3SECRET="my aws secret" # pass these in
function putS3
{
path=$1
file=$2
aws_path=$3
bucket='my-aws-bucket'
date=$(date +"%a, %d %b %Y %T %z")
acl="x-amz-acl:public-read"
content_type='application/x-compressed-tar'
string="PUT\n\n$content_type\n$date\n$acl\n/$bucket$aws_path$file"
signature=$(echo -en "${string}" | openssl sha1 -hmac "${S3SECRET}" -binary | base64)
curl -X PUT -T "$path/$file" \
-H "Host: $bucket.s3.amazonaws.com" \
-H "Date: $date" \
-H "Content-Type: $content_type" \
-H "$acl" \
-H "Authorization: AWS ${S3KEY}:$signature" \
"https://$bucket.s3.amazonaws.com$aws_path$file"
}
for file in "$path"/*; do
putS3 "$path" "${file##*/}" "/path/on/s3/to/files/"
done
@ChestersGarage
Copy link

Stoked to come across this. We are using it to upload files from AIX servers, which doesn't support (read: admins won't install) the AWS cli and related stuff. Thanks!

@pkhetan
Copy link

pkhetan commented Sep 24, 2018

My company is using rook for s3 storage. So I have different base URL for my bucket. "http://abc.rook.com" like this.
so if I create a new bucket with let say "piyush" and put some object inside "test" then my URL becomes like this "http://abc.rook.com/piyush/test/abc.txt". I am able to put object using s3api but not with above curl script.
Please suggest what change I need to do.

@atonamy
Copy link

atonamy commented Aug 21, 2020

Hi chrismdp,

I ran the same script in my environment it is giving out an error.
SignatureDoesNotMatchThe request signature we calculated does not match the signature you provided. Check your key and signing method.AKI*****************LQPUT

Can you please help me with that..
I am also trying to create a download shell script as well if you have any information regarding that do let me know.

Thanks In Advance.
:) 👍

I have the same problem

The request signature we calculated does not match the signature you provided

How to fix that?

@micaelomota
Copy link

micaelomota commented Sep 14, 2020

For those with root access, just install awscli and be happy doing that in only one line https://aws.amazon.com/getting-started/hands-on/backup-to-s3-cli/

@sanlodhi
Copy link

How can I test my connection to S3 bucket

@sanlodhi
Copy link

sanlodhi commented Oct 29, 2020

Hi bro ,may you please help with.....

How do I check my connection to S3 is successful or not?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment