aws-quickies.md

AWS Quickies

Introduction

A collection of handy AWS commands to get start and get around.

This is organized by services.

AWS CLI Installation

Refer to Amazon's documentation on installing the CLI. There is also a Docker option if you want an installation-less option. (Obviously, you should be comfortable with Docker for this.)

It's also reasonable to install the AWS CLI via Homebrew and other package managers.

Configuration

$ aws configure
AWS Access Key ID [None]: your-iam-user-access-key
AWS Secret Access Key [None]: your-iam-user-secret-access-key
Default region name [None]: us-west-2
Default output format [None]: text
$ aws configure --profile alt
AWS Access Key ID [None]: your-alternate-iam-account-access-key
AWS Secret Access Key [None]: your-alternate-iam-account-secret-access-key
Default region name [None]: us-west-2
Default output format [None]: text

$ ls -l ~/.aws
total 24
drwxr-xr-x    5 gkyc  staff   160 20 Oct 16:16 ./
drwxr-xr-x+ 128 gkyc  staff  4096 11 Nov 15:16 ../
-rw-------    1 gkyc  staff   242 11 Nov 15:15 config
-rw-------    1 gkyc  staff   687 11 Nov 15:15 credentials

$ cat ~/.aws/credentials
[default]
aws_access_key_id = your-iam-account-access-key
aws_secret_access_key = your-iam-account-secret-access-key

[alt]
aws_access_key_id = your-alternate-iam-account-access-key
aws_secret_access_key = your-alternate-iam-account-secret-access-key

$ cat ~/.aws/config     
[default]
region = us-west-2
output = text

[profile alt]
region = us-west-2
output = text

Security Best Practice

1. Create an IAM user. (You can still give full administrative access to this user.)
   1. Use a strong password.
   2. Turn on MFA
   3. Create an access key pair for this IAM user.
   4. Configure your AWS CLI with this access key pair
2. For your root user:
   1. Use a strong password.
   2. Turn on MFA
   3. Do not generate/use an access pair unless you are confident of your security practices

Credential/account check (default text output):

$ aws sts get-caller-identity
534zzzzzz160    arn:aws:iam::534zzzzzz160:root  534zzzzzz160

Use --output json to see the fields:

$ aws --output json sts get-caller-identity
{
    "UserId": "534zzzzzz160",
    "Account": "534zzzzzz160",
    "Arn": "arn:aws:iam::534zzzzzz160:root"
}

Shell commands

1. Streaming gunzip of a .gz file; useful for inspecting a log/CSV file.

$ gzcat /path/to/file.gz | less

2. Inspecting the contents of a .tar.gz file:

$ gzcat /path/to/file.tar.gz | tar tv | less

3. Finding a specific files within a .tar.gz file:

$ gzcat /path/to/file.tar.gz | tar tv | grep nameFragment

4. Pretty-printing a gzipped JSON file:

$ gzcat /path/to/file.json.gz | jq '.' | less

S3 (CLI doc)

1. Examine the contents of a bucket.

$ aws s3 ls --recursive s3://your-bucket-name

2. Copy (download) content from a bucket. The reverse (upload) also works.

$ aws s3 cp s3://source-bucket-name /path/of/local/destination

3. Recursively copies new/updated files from bucket to your local end. This is handy for reading EMR log files.

$ aws s3 sync s3://source-bucket-name /path/of/local/destination

4. The reverse also works: to upload a tree of files from your local laptop to your bucket. This is handy for bulk transfer of data files in a complex folder hierarchy. Experiment to get the hang of it.

$ aws s3 sync /path/of/local/destination s3://source-bucket-name 

5. Browse an arbitrary bucket (especially public buckets not owned by you):

   Use the URL https://s3.console.aws.amazon.com/s3/buckets/some-bucket-name (substitute some-bucket-name for a bucket of interest) in your browser to examine the bucket's content.

EMR (CLI doc)

An EMR cluster is identified by a synthetic id that is generated at cluster creation time.

Use the JSON format along with jq to pull that out:

$ aws --output json emr list-clusters | jq '.Clusters[].Id'
"j-21C2XNA48DDK8"
"j-33DBKO0TFB61C"
"j-G9HXGLETIE8Y"
...

A slightly more elaborate format:

$ aws --output json emr list-clusters | jq -r '.Clusters[]| .Id + " " + .Name + " " + .Status.State'
j-21C2XNA48DDK8 c732-emr-4x-m6gd.xl-corr TERMINATED
j-33DBKO0TFB61C c732-emr-4x-m6gd.xl-corr TERMINATED
j-G9HXGLETIE8Y c732-emr-2x-m4.2xl TERMINATED
...

(Refer to this jq tutorial)

Redshift (CLI doc)

A Redshift cluster is identified by the name that you specified at creation time.

$ aws redshift list-clusters

To pause a specific cluster:

$ aws redshift pause-cluster --cluster-identifier your-cluster-name

To resume a specific cluster:

$ aws redshift resume-cluster --cluster-identifier your-cluster-name

To examine a specific cluster

aws --output json redshift describe-clusters --cluster-identifier your-cluster-name | grep ClusterStatus

To delete a specific cluster:

$ aws redshift delete-cluster --skip-final-cluster-snapshot --cluster-identifier your-cluster-name

Reference

Refer to https://github.com/overcoil/c732-quickies for a set of handy shell aliases and functions for working with AWS (among others). The AWS CLI itself has an internal alias feature but I handroll this mainly for consistency with the other tools (e.g., Docker, kubectl, etc).