Skip to content

Instantly share code, notes, and snippets.

@micklove micklove/cli.md

Last active Mar 28, 2020
Embed
What would you like to do?

Useful AWS command line tools

awscli

The AWS Command Line Interface (CLI) is a unified tool to manage your AWS services.

Installation details

nb: See http://embedyoutube.org/ for embedding of youtube vids in markdown.


AWS awscli training

The best way to learn the awscli tool, is to use it, next, it's to watch the people that created the tool, show it off each year, at the AWS Re-invent conference. These videos are great.

AWS cli training video 2015

2015 aws cli

AWS cli training video 2015

2016 aws cli


youtube reinvent 2017

AWS Configure

the aws configure command is used to configure the credentials required to allow the cli to sign in to AWS services. More details here


Alias files

See https://github.com/awslabs/awscli-aliases/blob/master/alias for aws cli aliases

See here for a sample aws alias file


AWS Profiles

TODO


Specs

The jmespath specification is what the awscli tool uses to query / filter aws cli commands, using the --query parameter.

e.g.

## Retrieve the AWS roles for the `build` account, but only show the RoleName propery.
AWS_PROFILE=build \
  aws iam list-roles \
  --query Roles[].RoleName

Tools

jq - terminal

Command line tool, probably the most used, for querying / filtering json in the terminal. Does not necessarily follow the JMESPATH spec, nor functions.

Quick install:

brew install jq
# Windows install - chocolately
choco install jq

jq queries in the browser


jp - terminal

Command line tool, for using the SAME queries (follows the JMESPATH spec)as the awscli command. Quick Install:

brew tap jmespath/jmespath
brew install jp
Example - more at bottom of page
AWS_PROFILE=build aws sts get-caller-identity | jp -u 'Account'

The -u flag is used to ensure the output is NOT quoted. (similar to the raw, -r, flag in jq)

jp github pages

jmespath.org test queries online


jpterm - terminal

Command line tool, for using the same queries as the awscli command.

AWS_PROFILE=build aws sts get-caller-identity | jpterm

Use ctrl+p to swap between output modes. (e.g. Output the command OR the result set)

Further details on the jpterm github pages.


awscli - very useful tools

Escape an existing json file

Some aws commands require you to encode json, when passing commands.

e.g.

cmd

# Get some sample json
AWS_PROFILE=build aws sts get-caller-identity > id-example.json
cat id-example.json

output

{
    "Account": "1234567890",
    "UserId": "SOMEKEY:d911123",
    "Arn": "arn:aws:sts::1234567890:assumed-role/SomeUser/d911123"
}

encode with escape chars - use jp to_string method

jp -f trust-policy.json "to_string(@)"

output

"{\"Account\":\"972211316688\",\"Arn\":\"arn:aws:sts::1234567890:assumed-role/SomeUser/d911123\",\"UserId\":\"SOMEKEY:d911123:d911123\"}"

Nb: To save using an intermediate file, the file redirect operator can be used, use the following

jp -f <(AWS_PROFILE=build aws sts get-caller-identity)  "to_string(@)"

Example decoding of already escaped json string

To do the opposite (decode the json string), use the jq 'fromjson' command.

See the Policy property returned in the json
aws lambda get-policy --function-name my-lambda
{
    "Policy": "{\"Version\":\"2012-10-17\",\"Id\":\"default\",\"Statement\":[{\"Sid\":\"lambda-598abcdefg-90d1-41e6-83e2-90d5b0c6d08a\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"s3.amazonaws.com\"},\"Action\":\"lambda:InvokeFunction\",\"Resource\":\"arn:aws:lambda:ap-southeast-2:1234567890:function:somefunc\",\"Condition\":{\"StringEquals\":{\"AWS:SourceAccount\":\"1234567890\"},\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:s3:::some-bucket\"}}}]}",
    "RevisionId": "abcdefg-3f7a-4fa7-8863-88eca0b4c90c"
}
Now, decode the policy, using the jq fromjson method
aws lambda get-policy \
--function-name tmp_datalake_inventory \
--query "Policy" | jq '. | fromjson'

{
  "Version": "2012-10-17",
  "Id": "default",
  "Statement": [
    {
      "Sid": "lambda-598fbcde-90d1-41e6-83e2-90d5b0c6d08a",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "lambda:InvokeFunction",
      "Resource": "arn:aws:lambda:ap-southeast-2:12345678:function:somefunc",
      "Condition": {
        "StringEquals": {
          "AWS:SourceAccount": "1234567890"
        },
        "ArnLike": {
          "AWS:SourceArn": "arn:aws:s3:::some-bucket"
        }
      }
    }
  ]
}

Find all roles containing a given string, e.g. "pipeline"

    AWS_PROFILE=some-aws-profile-name \
    aws iam list-roles \
      --query 'Roles[?contains(RoleName, `pipeline`)].RoleName'

Examples

loop through each region showing uniq ec2 types

This command helps to locate any rogue instances, in other regions, that you may have forgotten about.

export ENV=some-aws-profile-name
for region in $(AWS_PROFILE="${ENV}" aws ec2 describe-regions --query 'Regions[].RegionName | sort(@)' --output text); \
do echo $region; \
    AWS_PROFILE="${ENV}" \
    aws ec2 describe-instances \
        --region "${region}" \
    --filter Name=instance-state-name,Values=running \
    --query 'Reservations[].Instances[].{InstanceType:InstanceType}' \
    | jq -r '.[].InstanceType' \
    | sort \
    | uniq -c \
    | sort -r;
done
ap-northeast-1
ap-northeast-2
ap-south-1
ap-southeast-1
ap-southeast-2
  24 t2.medium
   6 c4.8xlarge
   5 t2.micro
   3 c4.large
   2 m4.large
   1 m3.medium
ca-central-1
eu-central-1
eu-north-1
eu-west-1
eu-west-2
eu-west-3
sa-east-1
us-east-1
us-east-2
us-west-1
us-west-2

Match iam roles, with a prefix, e.g. pipeline-* in a given environment

Get INLINE ROLES

#Get INLINE ROLES
export ENV=some-aws-profile-name
for role in $(AWS_PROFILE=${ENV} aws iam list-roles --query "Roles[?contains(RoleName, 'ping')].RoleName" --output text)
do echo ${role}
  for policy in $(AWS_PROFILE="${ENV}" aws iam list-role-policies --role-name="${role}" --query PolicyNames --output text)
  do echo ${role}_${policy}
    AWS_PROFILE=${ENV} aws iam get-role-policy --role-name="${role}" --policy-name="${policy}" \
    > "ROLE-${role}-INLINE_ROLE_POLICY-${policy}.json"
  done
done

Get ATTACHED ROLE POLICIES

(and write them to a file)

export ENV=some-aws-profile-name
for role in $(AWS_PROFILE=${ENV} aws iam list-roles --query "Roles[?contains(RoleName, 'ping')].RoleName" --output text)
do echo ${role}
  for policy in $(AWS_PROFILE="${ENV}" aws iam list-attached-role-policies --role-name="${role}" --query "AttachedPolicies[].PolicyArn" --output text)
  do echo ${role}_${policy} - $(basename ${policy})
    AWS_PROFILE=${ENV} aws iam get-policy-version \
      --policy-arn "${policy}" \
      --version-id $(AWS_PROFILE=${ENV} aws iam get-policy --policy-arn="${policy}" --query "Policy.DefaultVersionId" --output text) \
      > "ROLE-${role}-ATTACHED_POLICY-$(basename ${policy}).json"
  done
done

diff 2 json files with different sort orders

# Diff json files with different formatting. Enables all sort of jq tricks, e.g. to diff only a subset of the content.
diff <(jq . --sort-keys ${file_1}) <(jq . --sort-keys ${file_2}) 

awslogs

https://github.com/jorgebastida/awslogs

AWS_PROFILE=someprofile \
   awslogs get /aws/codebuild/some-code-build-thing 
    --start='2h' \
    --filter-pattern "ami" > code-build-stuff-last-2-hours.txt

OR (for interactive tail)

AWS_PROFILE=someprofile awslogs get /aws/lambda/somelambda \
  --start='3h' \
  --filter-pattern="mysearchthing" \
  --query=message
  --watch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.