Common S3 and S3 Glacier actions using the AWS S3, S3API and Glacier APIs

README.md

A gist showing some common S3 and S3 Glacier actions using the API's listed below.

Actions

• Create an S3 Bucket
• Upload files to bucket choosing different storage classe
• Download objects from S3
• Restore objects from S3
• Configure an S3 lifecycle policy
• Create Glacier vault
• Upload archive to vault
• Request an inventory in Glacier
• Request an archive retrieval in Glacier
• Describe a Glacier job request
• List Glacier jobs
• Get Glacier job output
• Sample security policy for Glacier Topic

APIs

API	Documentation
S3 API	https://docs.aws.amazon.com/cli/latest/reference/s3/
S3 API	https://docs.aws.amazon.com/cli/latest/reference/s3api/index.html#cli-aws-s3api
Glacier API	https://docs.aws.amazon.com/cli/latest/reference/glacier/index.html#cli-aws-glacier

Define some common variables

S3_BUCKETNAME=YOURBUCKETNAME${RANDOM}
S3_STANDARD_UPLOAD_1=koala.jpg
S3_STANDARD_UPLOAD_2=emu.jpg
S3_GLACIER_UPLOAD_1=kangaroo.jpg
S3_GLACIER_UPLOAD_2=galah.jpg
GLACIER_VAULT_NAME=YOURVAULTNAME
GLACIER_ARCHIVE=glacier-archive.zip
AWS_PROFILE=YOUR_AWS_CLI_PROFILE_NAME

Create an S3 Bucket

https://docs.aws.amazon.com/cli/latest/reference/s3/mb.html

aws s3 mb s3://${S3_BUCKETNAME} --profile ${AWS_PROFILE}

Upload files to bucket choosing different storage classes

https://docs.aws.amazon.com/cli/latest/reference/s3api/put-object.html

aws s3api put-object \
 --bucket ${S3_BUCKETNAME} \
 --storage-class STANDARD \
 --key ${S3_STANDARD_UPLOAD_1} \
 --body uploads/${S3_STANDARD_UPLOAD_1} \
 --profile ${AWS_PROFILE}

aws s3api put-object \
 --bucket ${S3_BUCKETNAME} \
 --storage-class ONEZONE_IA \
 --key ${S3_STANDARD_UPLOAD_2} \
 --body uploads/${S3_STANDARD_UPLOAD_2} \
 --profile ${AWS_PROFILE}

aws s3api put-object \
 --bucket ${S3_BUCKETNAME} \
 --storage-class GLACIER \
 --key ${S3_GLACIER_UPLOAD_1} \
 --body uploads/${S3_GLACIER_UPLOAD_1} \
 --profile ${AWS_PROFILE}

aws s3api put-object \
 --bucket ${S3_BUCKETNAME} \
 --storage-class DEEP_ARCHIVE \
 --key ${S3_GLACIER_UPLOAD_2} \
 --body uploads/${S3_GLACIER_UPLOAD_2} \
 --profile ${AWS_PROFILE}

Download objects from S3

https://docs.aws.amazon.com/cli/latest/reference/s3api/get-object.html

aws s3api get-object \
 --bucket ${S3_BUCKETNAME} \
 --key ${S3_STANDARD_UPLOAD_1} \
 --profile ${AWS_PROFILE} \
 downloads/${S3_STANDARD_UPLOAD_1}

Restore objects from S3

https://docs.aws.amazon.com/cli/latest/reference/s3api/restore-object.html

aws s3api restore-object \
 --bucket ${S3_BUCKETNAME} \
 --key ${S3_GLACIER_UPLOAD_1} \
 --restore-request Days=1 \
 --profile ${AWS_PROFILE}

Configure an S3 lifecycle policy

https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-lifecycle-configuration.html

aws s3api put-bucket-lifecycle-configuration \
 --bucket ${S3_BUCKETNAME} \
 --lifecycle-configuration file://json/lifecycle.json \
 --profile ${AWS_PROFILE}

Create Glacier vault

https://docs.aws.amazon.com/cli/latest/reference/glacier/create-vault.html

aws glacier create-vault \
 --account-id - \
 --vault-name ${GLACIER_VAULT_NAME} \
 --profile ${AWS_PROFILE}

Upload archive to vault

https://docs.aws.amazon.com/cli/latest/reference/glacier/upload-archive.html

aws glacier upload-archive \
 --account-id - \
 --vault-name ${GLACIER_VAULT_NAME} \
 --body uploads/${GLACIER_ARCHIVE} \
 --profile ${AWS_PROFILE}

Request an inventory in Glacier

https://docs.aws.amazon.com/cli/latest/reference/glacier/initiate-job.html

aws glacier initiate-job \
 --account-id - \
 --vault-name ${GLACIER_VAULT_NAME} \
 --job-parameters file://json/inventory-retrieval.json \
 --profile ${AWS_PROFILE}

Request an archive retrieval in Glacier

https://docs.aws.amazon.com/cli/latest/reference/glacier/initiate-job.html

aws glacier initiate-job \
 --account-id - \
 --vault-name ${GLACIER_VAULT_NAME} \
 --job-parameters file://json/archive-retrieval.json \
 --profile ${AWS_PROFILE}

Describe a Glacier job request

https://docs.aws.amazon.com/cli/latest/reference/glacier/describe-job.html

aws glacier describe-job \
 --account-id - \
 --vault-name ${GLACIER_VAULT_NAME} \
 --profile ${AWS_PROFILE} \
  --job-id JOB_ID

List Glacier jobs

https://docs.aws.amazon.com/cli/latest/reference/glacier/list-jobs.html

aws glacier list-jobs \
 --account-id - \
 --vault-name ${GLACIER_VAULT_NAME} \
 --profile ${AWS_PROFILE}

Get Glacier job output

https://docs.aws.amazon.com/cli/latest/reference/glacier/get-job-output.html

aws glacier get-job-output \
 --account-id - \
 --vault-name ${GLACIER_VAULT_NAME} \
 --job-id JOB_ID \
 --profile ${AWS_PROFILE} \
  glacier-jobs-out

Sample security policy for Glacier Topic

Limited policy

{
  "Version": "2008-10-17",
  "Id": "GlacierNotificationPolicy",
  "Statement": [
    {
      "Sid": "GlacierNotificationAllow",
      "Effect": "Allow",
      "Principal": {
        "Service": "glacier.amazonaws.com"
      },
      "Action": [
        "SNS:Publish"
      ],
      "Resource": "TOPIC_ARN"
     }
  ]
}

Default policy

{
  "Version": "2008-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "__default_statement_ID",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": [
        "SNS:Publish",
        "SNS:RemovePermission",
        "SNS:SetTopicAttributes",
        "SNS:DeleteTopic",
        "SNS:ListSubscriptionsByTopic",
        "SNS:GetTopicAttributes",
        "SNS:Receive",
        "SNS:AddPermission",
        "SNS:Subscribe"
      ],
      "Resource": "TOPIC_ARN",
      "Condition": {
        "StringEquals": {
          "AWS:SourceOwner": "AWS_ACCOUNT_ID"
        }
      }
    }
  ]
}

archive-retrieval.json

{
  "Type": "archive-retrieval",
  "Format" : "JSON",
  "ArchiveId": "ADD_THE_ARCHIVE_ID",
  "Description": "Test example for retrieving Glacier archive",
  "SNSTopic": "ADD_SNS_TOPIC_ARN"
}

inventory-retrieval.json

{
  "Type": "inventory-retrieval",
  "Format" : "JSON",
  "Description": "Inventory retrieval request",
  "SNSTopic": "ADD_SNS_TOPIC_ARN"
}

lifecycle.json

{
    "Rules": [
        {
	    "ID": "Move files to Glacier",
            "Prefix": "/",
            "Status": "Enabled",
            "Transitions": [
                {
                    "Days": 2,
                    "StorageClass": "GLACIER"
                }
            ]
        }
    ]
}