micklove/cli.md

## cli.md

      
    Raw
  

              cli.md
            
          
    Useful AWS command line tools

awscli

The AWS Command Line Interface (CLI) is a unified tool to manage your AWS services.
Installation details
nb: See http://embedyoutube.org/ for embedding of youtube vids in markdown.

AWS awscli training

The best way to learn the awscli tool, is to use it, next, it's to watch the people that created the tool, show it off each year, at the AWS Re-invent conference. These videos are great.
AWS cli training video 2015


AWS cli training video 2015


youtube reinvent 2017
AWS Configure

the aws configure command is used to configure the credentials required to allow the cli to sign in to AWS services. More details here

Alias files

See https://github.com/awslabs/awscli-aliases/blob/master/alias for aws cli aliases
See here for a sample aws alias file

AWS Profiles

TODO

Specs

The jmespath specification is what
the awscli tool uses to query / filter aws cli commands, using the --query parameter.
e.g.
## Retrieve the AWS roles for the `build` account, but only show the RoleName propery.
AWS_PROFILE=build \
  aws iam list-roles \
  --query Roles[].RoleName

Tools

jq - terminal

Command line tool, probably the most used, for querying / filtering json in the terminal.
Does not necessarily follow the JMESPATH spec, nor functions.
Quick install:
brew install jq
# Windows install - chocolately
choco install jq
jq queries in the browser

jp - terminal

Command line tool, for using the SAME queries (follows the JMESPATH spec)as the awscli command.
Quick Install:
brew tap jmespath/jmespath
brew install jp
Example - more at bottom of page

AWS_PROFILE=build aws sts get-caller-identity | jp -u 'Account'
The -u flag is used to ensure the output is NOT quoted. (similar to the raw, -r, flag in jq)
jp github pages
jmespath.org test queries online

jpterm - terminal

Command line tool, for using the same queries as the awscli command.
AWS_PROFILE=build aws sts get-caller-identity | jpterm
Use ctrl+p to swap between output modes. (e.g. Output the command OR the result set)
Further details on the jpterm github pages.

awscli - very useful tools

Escape an existing json file

Some aws commands require you to encode json, when passing commands.
e.g.
cmd
# Get some sample json
AWS_PROFILE=build aws sts get-caller-identity > id-example.json
cat id-example.json
output
{
    "Account": "1234567890",
    "UserId": "SOMEKEY:d911123",
    "Arn": "arn:aws:sts::1234567890:assumed-role/SomeUser/d911123"
}
encode with escape chars - use jp to_string method
jp -f trust-policy.json "to_string(@)"
output
"{\"Account\":\"972211316688\",\"Arn\":\"arn:aws:sts::1234567890:assumed-role/SomeUser/d911123\",\"UserId\":\"SOMEKEY:d911123:d911123\"}"
Nb: To save using an intermediate file, the file redirect operator can be used, use the following
jp -f <(AWS_PROFILE=build aws sts get-caller-identity)  "to_string(@)"

Example decoding of already escaped json string

To do the opposite (decode the json string), use the jq 'fromjson' command.
See the Policy property returned in the json

aws lambda get-policy --function-name my-lambda
{
    "Policy": "{\"Version\":\"2012-10-17\",\"Id\":\"default\",\"Statement\":[{\"Sid\":\"lambda-598abcdefg-90d1-41e6-83e2-90d5b0c6d08a\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"s3.amazonaws.com\"},\"Action\":\"lambda:InvokeFunction\",\"Resource\":\"arn:aws:lambda:ap-southeast-2:1234567890:function:somefunc\",\"Condition\":{\"StringEquals\":{\"AWS:SourceAccount\":\"1234567890\"},\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:s3:::some-bucket\"}}}]}",
    "RevisionId": "abcdefg-3f7a-4fa7-8863-88eca0b4c90c"
}

Now, decode the policy, using the jq fromjson method

aws lambda get-policy \
--function-name tmp_datalake_inventory \
--query "Policy" | jq '. | fromjson'

{
  "Version": "2012-10-17",
  "Id": "default",
  "Statement": [
    {
      "Sid": "lambda-598fbcde-90d1-41e6-83e2-90d5b0c6d08a",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "lambda:InvokeFunction",
      "Resource": "arn:aws:lambda:ap-southeast-2:12345678:function:somefunc",
      "Condition": {
        "StringEquals": {
          "AWS:SourceAccount": "1234567890"
        },
        "ArnLike": {
          "AWS:SourceArn": "arn:aws:s3:::some-bucket"
        }
      }
    }
  ]
}

Find all roles containing a given string, e.g. "pipeline"

    AWS_PROFILE=some-aws-profile-name \
    aws iam list-roles \
      --query 'Roles[?contains(RoleName, `pipeline`)].RoleName'
Examples

loop through each region showing uniq ec2 types

This command helps to locate any rogue instances, in other regions, that you may have forgotten about.
export ENV=some-aws-profile-name
for region in $(AWS_PROFILE="${ENV}" aws ec2 describe-regions --query 'Regions[].RegionName | sort(@)' --output text); \
do echo $region; \
    AWS_PROFILE="${ENV}" \
    aws ec2 describe-instances \
        --region "${region}" \
    --filter Name=instance-state-name,Values=running \
    --query 'Reservations[].Instances[].{InstanceType:InstanceType}' \
    | jq -r '.[].InstanceType' \
    | sort \
    | uniq -c \
    | sort -r;
done
ap-northeast-1
ap-northeast-2
ap-south-1
ap-southeast-1
ap-southeast-2
  24 t2.medium
   6 c4.8xlarge
   5 t2.micro
   3 c4.large
   2 m4.large
   1 m3.medium
ca-central-1
eu-central-1
eu-north-1
eu-west-1
eu-west-2
eu-west-3
sa-east-1
us-east-1
us-east-2
us-west-1
us-west-2


Match iam roles, with a prefix, e.g. pipeline-* in a given environment

Get INLINE ROLES

#Get INLINE ROLES
export ENV=some-aws-profile-name
for role in $(AWS_PROFILE=${ENV} aws iam list-roles --query "Roles[?contains(RoleName, 'ping')].RoleName" --output text)
do echo ${role}
  for policy in $(AWS_PROFILE="${ENV}" aws iam list-role-policies --role-name="${role}" --query PolicyNames --output text)
  do echo ${role}_${policy}
    AWS_PROFILE=${ENV} aws iam get-role-policy --role-name="${role}" --policy-name="${policy}" \
    > "ROLE-${role}-INLINE_ROLE_POLICY-${policy}.json"
  done
done
Get ATTACHED ROLE POLICIES

(and write them to a file)
export ENV=some-aws-profile-name
for role in $(AWS_PROFILE=${ENV} aws iam list-roles --query "Roles[?contains(RoleName, 'ping')].RoleName" --output text)
do echo ${role}
  for policy in $(AWS_PROFILE="${ENV}" aws iam list-attached-role-policies --role-name="${role}" --query "AttachedPolicies[].PolicyArn" --output text)
  do echo ${role}_${policy} - $(basename ${policy})
    AWS_PROFILE=${ENV} aws iam get-policy-version \
      --policy-arn "${policy}" \
      --version-id $(AWS_PROFILE=${ENV} aws iam get-policy --policy-arn="${policy}" --query "Policy.DefaultVersionId" --output text) \
      > "ROLE-${role}-ATTACHED_POLICY-$(basename ${policy}).json"
  done
done


diff 2 json files with different sort orders

# Diff json files with different formatting. Enables all sort of jq tricks, e.g. to diff only a subset of the content.
diff <(jq . --sort-keys ${file_1}) <(jq . --sort-keys ${file_2}) 
awslogs

https://github.com/jorgebastida/awslogs
AWS_PROFILE=someprofile \
   awslogs get /aws/codebuild/some-code-build-thing 
    --start='2h' \
    --filter-pattern "ami" > code-build-stuff-last-2-hours.txt
OR (for interactive tail)
AWS_PROFILE=someprofile awslogs get /aws/lambda/somelambda \
  --start='3h' \
  --filter-pattern="mysearchthing" \
  --query=message
  --watch