Notes on AWS CLI usage

aws-notes.md

AWS Notes

All config, credentials are stored by aws in ~/.aws.

Note: In any command, a variable written in all capital (example: SSO-PROFILE-NAME) has to be replaced by the appropriate value.

Installing AWS CLI v2

Official install without root

The official docs only show how to install aws-cli as root user. If you don't have root access, you can follow these steps:

$ cd Downloads
$ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
$ unzip awscliv2.zip
$ ./aws/install -i ~/aws-cli -b ~/aws-cli/bin
$ ~/aws-cli/bin/aws --version

Via pip, unofficial package awscliv2

AWS CLI v2 is not available for install in PyPi. As a workaround, we can install the unofficial package awscliv2 from pip:

$ pip3 install awscliv2
$ awscliv2 --install
$ alias aws=awsv2                          # put this in ~/.bashrc

Note: If Warning: Input is not a terminal (fd=0). shows, this may be because you forgot to do awscliv2 --install. Run it now, then this issue should be resolved. If the issue still occurs, try installing official aws-cli without root.

Single Sign On (SSO)

Blog on AWS CLI Usage with SSO - need to read later!

Basic Usage

• aws configure sso: Interactively configure SSO Credentials on local machine.
  • Asks for login url, region.
  • Opens browser window where you can click Allow to give aws CLI access. Now close the browser window.
  • (CLI) Asks for details to create new SSO profile: default region, profile name SSO-PROFILE-NAME, etc.
• aws sso login --profile SSO-PROFILE: Actually login to the SSO profile - this will again open a browser window where you have to click Allow.
  Note: This step has to be repeated periodically, because the credentials expire in a short time (not sure of exact - maybe a few hours or 1 day?)
• Now you can use this SSO profile in Python code like this:

import boto3
boto3.setup_default_session(profile_name='SSO-PROFILE-NAME')        # Here enter the SSO Profile name which was created above
my_profile = boto3.client('sts').get_caller_identity()     # If credentials invalid / expired / anything else, will raise exception
# Now use boto3 as normal - create client / resource, etc.

Note: Basically any aws command works almost the same way with / without SSO - the only difference is that when using SSO, you have to specify your profile for any command.

For example:

• List all S3 buckets I have access to:
  • aws s3 ls --> normal IAM account (no SSO)
  • aws s3 ls --profile SSO-PROFILE-NAME --> using SSO Profile

Specifying SSO Profile every time can be tedious, so default SSO Profile can be configured. Don't know how right now, need to check the docs.

PS: If you don't want SSO configure or login to automatically open a browser (for example, in a headless server), you can do this - BROWSER=true aws sso login --profile SSO-PROFILE-NAME.

PPS: If you get the error botocore.exceptions.UnauthorizedSSOTokenError while using SSO profile, it means SSO session expired, so simply do aws sso login --profile SSO_PROFILE again. SSO session usually expires within a day.

AWS Code Commit

After doing aws configure sso:

$ pip install git-remote-codecommit
$ git clone codecommit://SSO-PROFILE-NAME@NAME-OF-REPOSITORY        # replace SSO-PROFILE-NAME, NAME-OF-REPOSITORY with your details

AWS Lambda

Identify unused Lambda Layers (i.e., no Lambda function is using these layers):

lambda_client = boto3.client('lambda', region_name='us-west-2')
all_layers = lambda_client.list_layers()['Layers']      # list of all Lambda layers
all_layer_arns = set(layer['LayerArn'] for layer in all_layers)
all_functions = lambda_client.list_functions()['Functions']     # list of all Lambda functions
used_layer_arns = set(
    layer['Arn'].rpartition(':')[0]       # remove layer version from the end of the ARN
    for function in all_functions
    for layer in function.get('Layers', [])
)
unused_layer_arns = all_layer_arns - used_layer_arns    
for layer_arn in unused_layer_arns:
    layer_name = layer_arn.rpartition(':')[2]
    print('Unused Layer:', layer_name)

AWS S3

• Upload to S3:

s3_resource = boto3.resource('s3', region_name='us-west-2')
bucket = s3_resource.Bucket('bucket-name')
bucket.put_object(Key='index.html', Body=data, ContentType='text/html')           # directly upload bytes
bucket.upload_file()          # upload file (by name)
bucket.upload_fileobj()       # upload bytes from any file-like object (eg. BytesIO)

Note: You can get the underlying boto3.client from a boto3.resource() like this: s3_resource.meta.client.

Fargate / Docker ECR Private Repo

Use docker login to store credentials in ~/.docker/config.json, then use Docker commands normally:

$ aws ecr get-login-password --profile SSO_PROFILE_NAME | docker login --username AWS --password-stdin ECR_URI
$ docker pull ECR_URI

Replace SSO_PROFILE_NAME and ECR_URI with your details. ECR_URI is usually of the form ******.dkr.ecr.us-west-2.amazonaws.com/IMAGE_NAME.

Note: AWS is NOT a detail to be replaced - just write AWS exactly.

Fargate CPU, Memory configurations

Source (blog) - corrected vCPU of cpu=256, 512 - original blog wrongly shows as 25 vCPU instead of right .25 vCPU.

CPU Value	Memory Value
256 (.25 vCPU)	0.5GB, 1GB and 2GB
512 (.5 vCPU)	Min 1GB and Max 4GB, in 1GB increments
1024 (1 vCPU)	Min 2GB and Max 8GB, in 1GB increments
2048 (2 vCPU)	Min 4GB and Max 16GB, in 1GB increments
4096 (4 vCPU)	Min 8GB and Max 30GB, in 1GB increments

Misc

• Credentials (Access Key Id, Secret Access Key, Session Token) can be obtained by going to SSO start page, clicking on your application and selecting Command Line or Programmatic Access.
• Remove all saved credentials: Delete folder ~/.aws.