Skip to content

Instantly share code, notes, and snippets.

@ryansimms
Last active February 22, 2024 04:55
Show Gist options
  • Save ryansimms/808214137d219be649e010a07af44bad to your computer and use it in GitHub Desktop.
Save ryansimms/808214137d219be649e010a07af44bad to your computer and use it in GitHub Desktop.
Deploying to Elastic Beanstalk via CircleCi 2.0

Deploying to Elastic Beanstalk via CircleCi 2.0

I got to here after spending hours trying to deploy to an Elastic Beanstalk instance via CircleCi 2.0 so I thought I'd write up what worked for me to hopefully help others. Shout out to RobertoSchneiders who's steps for getting it to work with CircleCi 1.0 were my starting point.

For the record, I'm not the most server-savvy of developers so there may be a better way of doing this.

Setup a user on AWS IAM to use for deployments

  • Add user here
  • Set a username and select Programmatic access as the Access type
  • Click 'Create group' on the user permissions page
  • Set a group name and search for the AdministratorAccess-AWSElasticBeanstalk and AmazonS3FullAccess policies and select them
    • Note: This used to be just AWSElasticBeanstalkFullAccess, but that's since been deprecated
  • Create the group so it's assigned to your new user
  • Review and create the user

Setup an Elastic Beanstalk application

  • 'Create New Application'
  • 'Create New Environment'
  • 'Create web server'
  • On the Environment Information page where it asks you to set the 'Environment name', set it to something with the Git branch name in e.g. BRANCHNAME-my-application
    • I do this as I have a staging branch and the master branch so in our EB config, we'll be replacing BRANCHNAME with the $CIRCLE_BRANCH environment variable provided by CircleCi so when deploying the staging branch for example, it will know to deploy to the staging-my-application environment on Elastic Beanstalk
  • Follow the rest of the setup as you require

Add deployment user environment variables to CircleCi

  • Project Settings > Environment Variables
    • AWS_ACCESS_KEY_ID
    • AWS_SECRET_ACCESS_KEY

Add .elasticbeanstalk/config.yml config to application code

  • Update the values in the below snippet to match what you setup.
branch-defaults:
  master:
    environment: $CIRCLE_BRANCH-my-application
global:
  application_name: My Application
  default_ec2_keyname: XXXXXX
  default_platform: 64bit Amazon Linux 2017.09 v2.6.2 running Ruby 2.4 (Passenger Standalone)
  default_region: eu-west-1
  sc: git

Note: Ensure the application_name is exactly what you called your application in Elastic Beanstalk when you did the 'Create New Application' step.

Also Note: Do not set a profile: value here, the profile will be set based on the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables you setup.

Update your .circleci/config.yml

  • The following is what I think is the bare minimum I needed to get only the master or staging branches on your Git repo to deploy.
  • Update $CIRCLE_BRANCH-my-application to the environment name you set in Elastic Beanstalk.
version: 2
jobs:
  deploy:
    working_directory: ~/app
    docker:
      - image: circleci/ruby:2.4.3
    steps:
      - checkout

      - run:
          name: Installing deployment dependencies
          working_directory: /
          command: |
            sudo apt-get -y -qq update
            sudo apt-get install python3-pip python3-dev build-essential
            sudo pip3 install awsebcli

      - run:
          name: Deploying
          command: eb deploy $CIRCLE_BRANCH-my-application

workflows:
  version: 2
  build:
    jobs:
      - deploy:
          filters:
            branches:
              only:
                - staging
                - master

Cross fingers

  • Commit and let CircleCi do it's thing. If all goes well you should see it updating on the Elastic Beanstalk dashboard as the 'Deploying' step is running in CircleCi.
@howardkitto
Copy link

I have my env vars set up (I can see them via - run : echo ${AWS_ACCESS_KEY_ID}

But get this error:

sudo eb deploy myApp-env
ERROR: CredentialsError - Operation Denied. You appear to have no credentials
Exited with code 4

Here is my config:
.circleci/config.yml

version: 2
jobs:
  deploy:
    working_directory: /go/src
    docker:
      - image: circleci/golang
    steps:
      - checkout
      - run: sudo apt-get -y -qq update --assume-yes
      - run: sudo apt-get install python-pip python-dev build-essential --assume-yes
      - run: sudo pip install awsebcli --upgrade
      - run : echo ${AWS_ACCESS_KEY_ID}
      - run : sudo eb deploy myApp-env
workflows:
  version: 2
  build_and_test:
    jobs:
      - deploy

.elasticbeanstalk/config.yml

branch-defaults:
  develop:
    environment: myApp-env
  master:
    environment: myApp-env
global:
  application_name: myApp
  default_platform: Docker 18.03.1-ce
  default_region: eu-west-2
  sc: git

Any ideas what could be wrong? (I'm guessing a dumb typo!)

@ryansimms
Copy link
Author

@howardkitto - have you tried the eb deploy command without sudo ? e.g. - run: eb deploy myApp-env

@howardkitto
Copy link

Huge thanks @ryansimms that was the problem - obvious now I look at it!

@chemitaxis
Copy link

So good!! Thanks for sharing!!

@amosuro
Copy link

amosuro commented Oct 15, 2018

Thanks @ryansimms, great starting point for me!

@lifenautjoe
Copy link

Getting
TypeError - must be str, not NoneType
when running

eb deploy $CIRCLE_BRANCH-xxx

Any idea anyone?

@ryansimms
Copy link
Author

@lifenautjoe - sorry, missed your comment, but checking out your config on openbook, I'm guess you managed to resolve the issue?

@KahTim
Copy link

KahTim commented Nov 24, 2018

Hey @ryansimms, my .elasticbeanstalk/config files doesn't seem to have access to environment variables I've added in CircleCI, how did you get your config file to access $CIRCLE_BRANCH
Mine deploys it as a string during eb deploy, output as below:

Uploading $EC2_APP_NAME/xxx.zip to S3. This may take a while.
Upload Complete.
Application $EC2_APP_NAME has been created.

.elasticbeanstalk/config.yml

branch-defaults:
  master:
    environment: $EB_ENVIRONMENT
global:
  application_name: $EC2_APP_NAME
  default_ec2_keyname: $EC2_KEYNAME
  default_platform: arn:aws:elasticbeanstalk:ap-southeast-1::platform/Puma with Ruby
    2.3 running on 64bit Amazon Linux/2.8.2
  default_region: ap-southeast-1

@ryansimms
Copy link
Author

Hey @KahTim - apologies for the delayed response (no notifications on gists yet)
It sounds like you may not have setup your environment variables on CircleCi - $CIRCLE_BRANCH is given by default by CircleCi, but if you have custom variables then you'll need to add them to your project on CircleCi via the Project settings > Build Settings > Environment Variables page.

Hope that helps

@rofc
Copy link

rofc commented Jan 3, 2019

hi @ryansimms, I think what @KahTim is trying to say is how did you create the .elasticbeanstalk/config.yml file since variables are being took as string instead of environment variables. There is no detailed step about this. Thanks!

@jakesylvestre
Copy link

Thanks! This was helpful.

@4rch18
Copy link

4rch18 commented Feb 13, 2019

There's another way to handle AWS keys in CircleCI:

  • Project Settings > Permissions > AWS
    • Access Key ID
    • Secret Access Key

As given on the page [Feb 13 2019]:

Set the AWS keypair to be used for authenticating against AWS services during your builds. Credentials are installed on your containers into the ~/.aws/config and ~/.aws/credentials properties files. These are read by common AWS libraries such as the Java SDK, Python's boto, and the Ruby SDK.

We recommend that you create a unique IAM user for use by CircleCI.

@vitr
Copy link

vitr commented Feb 27, 2019

thanks for the great gist! I've been using it for about 6 months, unfortunately, it stopped working yesterday. the error is

 Complete output from command python setup.py egg_info:
    error in awsebcli setup command: Invalid environment marker: sys_platform == "win32" and python_version >= "3.6"

and the full log

#!/bin/bash -eo pipefail
sudo apt-get -y -qq update
sudo apt-get install python-pip python-dev build-essential
sudo pip install awsebcli --upgrade
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://packagecloud.io trusty InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 4E6910DFCB68C9CD

W: Failed to fetch https://packagecloud.io/circleci/trusty/ubuntu/dists/trusty/InRelease  

W: Some index files failed to download. They have been ignored, or old ones used instead.

Reading package lists... Done

Building dependency tree       


Reading state information... Done

build-essential is already the newest version.
python-pip is already the newest version.
The following extra packages will be installed:
  libpython-dev libpython2.7 libpython2.7-dev libpython2.7-minimal
  libpython2.7-stdlib python2.7 python2.7-dev python2.7-minimal
Suggested packages:
  python2.7-doc
The following NEW packages will be installed:
  libpython-dev libpython2.7-dev python-dev python2.7-dev
The following packages will be upgraded:
  libpython2.7 libpython2.7-minimal libpython2.7-stdlib python2.7
  python2.7-minimal
5 upgraded, 4 newly installed, 0 to remove and 245 not upgraded.
Need to get 26.9 MB of archives.
After this operation, 34.2 MB of additional disk space will be used.


0% [Working]0% [Connecting to archive.ubuntu.com (91.189.88.162)]0% [Connecting to archive.ubuntu.com (91.189.88.162)]0% [Waiting for headers]0% [Waiting for headers]Get:1 http://archive.ubuntu.com/ubuntu/ trusty-updates/main python2.7 amd64 2.7.6-8ubuntu0.5 [197 kB]

0% [1 python2.7 2,293 B/197 kB 1%]Get:2 http://archive.ubuntu.com/ubuntu/ trusty-updates/main libpython2.7 amd64 2.7.6-8ubuntu0.5 [1,041 kB]

5% [Working]Get:3 http://archive.ubuntu.com/ubuntu/ trusty-updates/main libpython2.7-stdlib amd64 2.7.6-8ubuntu0.5 [1,872 kB]

Get:4 http://archive.ubuntu.com/ubuntu/ trusty-updates/main python2.7-minimal amd64 2.7.6-8ubuntu0.5 [1,186 kB]

Get:5 http://archive.ubuntu.com/ubuntu/ trusty-updates/main libpython2.7-minimal amd64 2.7.6-8ubuntu0.5 [308 kB]

Get:6 http://archive.ubuntu.com/ubuntu/ trusty-updates/main libpython2.7-dev amd64 2.7.6-8ubuntu0.5 [22.0 MB]

17% [6 libpython2.7-dev 33.2 kB/22.0 MB 0%]70% [6 libpython2.7-dev 14.2 MB/22.0 MB 65%]Get:7 http://archive.ubuntu.com/ubuntu/ trusty/main libpython-dev amd64 2.7.5-5ubuntu3 [7,078 B]

Get:8 http://archive.ubuntu.com/ubuntu/ trusty-updates/main python2.7-dev amd64 2.7.6-8ubuntu0.5 [269 kB]

100% [Waiting for headers]Get:9 http://archive.ubuntu.com/ubuntu/ trusty/main python-dev amd64 2.7.5-5ubuntu3 [1,166 B]

Fetched 26.9 MB in 4s (5,920 kB/s)
debconf: unable to initialize frontend: Dialog
debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
debconf: falling back to frontend: Readline
(Reading database ... 71348 files and directories currently installed.)
Preparing to unpack .../python2.7_2.7.6-8ubuntu0.5_amd64.deb ...
Unpacking python2.7 (2.7.6-8ubuntu0.5) over (2.7.6-8ubuntu0.3) ...
Preparing to unpack .../libpython2.7_2.7.6-8ubuntu0.5_amd64.deb ...
Unpacking libpython2.7:amd64 (2.7.6-8ubuntu0.5) over (2.7.6-8ubuntu0.3) ...
Preparing to unpack .../libpython2.7-stdlib_2.7.6-8ubuntu0.5_amd64.deb ...
Unpacking libpython2.7-stdlib:amd64 (2.7.6-8ubuntu0.5) over (2.7.6-8ubuntu0.3) ...
Preparing to unpack .../python2.7-minimal_2.7.6-8ubuntu0.5_amd64.deb ...
Unpacking python2.7-minimal (2.7.6-8ubuntu0.5) over (2.7.6-8ubuntu0.3) ...
Preparing to unpack .../libpython2.7-minimal_2.7.6-8ubuntu0.5_amd64.deb ...
Unpacking libpython2.7-minimal:amd64 (2.7.6-8ubuntu0.5) over (2.7.6-8ubuntu0.3) ...
Selecting previously unselected package libpython2.7-dev:amd64.
Preparing to unpack .../libpython2.7-dev_2.7.6-8ubuntu0.5_amd64.deb ...
Unpacking libpython2.7-dev:amd64 (2.7.6-8ubuntu0.5) ...
Selecting previously unselected package libpython-dev:amd64.
Preparing to unpack .../libpython-dev_2.7.5-5ubuntu3_amd64.deb ...
Unpacking libpython-dev:amd64 (2.7.5-5ubuntu3) ...
Selecting previously unselected package python2.7-dev.
Preparing to unpack .../python2.7-dev_2.7.6-8ubuntu0.5_amd64.deb ...
Unpacking python2.7-dev (2.7.6-8ubuntu0.5) ...
Selecting previously unselected package python-dev.
Preparing to unpack .../python-dev_2.7.5-5ubuntu3_amd64.deb ...
Unpacking python-dev (2.7.5-5ubuntu3) ...
Processing triggers for mime-support (3.54ubuntu1.1) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Setting up libpython2.7-minimal:amd64 (2.7.6-8ubuntu0.5) ...
Setting up python2.7-minimal (2.7.6-8ubuntu0.5) ...
Setting up libpython2.7-stdlib:amd64 (2.7.6-8ubuntu0.5) ...
Setting up python2.7 (2.7.6-8ubuntu0.5) ...
Setting up libpython2.7:amd64 (2.7.6-8ubuntu0.5) ...
Setting up libpython2.7-dev:amd64 (2.7.6-8ubuntu0.5) ...
Setting up libpython-dev:amd64 (2.7.5-5ubuntu3) ...
Setting up python2.7-dev (2.7.6-8ubuntu0.5) ...
Setting up python-dev (2.7.5-5ubuntu3) ...
Processing triggers for libc-bin (2.19-0ubuntu6.13) ...
Downloading/unpacking awsebcli
  Downloading awsebcli-3.14.13.tar.gz (241kB): 241kB downloaded
  Running setup.py (path:/tmp/pip_build_root/awsebcli/setup.py) egg_info for package awsebcli
    error in awsebcli setup command: Invalid environment marker: sys_platform == "win32" and python_version >= "3.6"
    Complete output from command python setup.py egg_info:
    error in awsebcli setup command: Invalid environment marker: sys_platform == "win32" and python_version >= "3.6"

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /tmp/pip_build_root/awsebcli
Storing debug log for failure in /home/circleci/.pip/pip.log
Exited with code 1

@ryansimms
Copy link
Author

@vitr - We just experienced this on one of our node repos, but surprisingly not for our rails apps which use the same sort of config.

Afraid it's beyond my dev-ops knowledge as to why it happens, but this revision seems to fix the issue:
https://gist.github.com/ryansimms/808214137d219be649e010a07af44bad/revisions#diff-efeb8e8fc639c88ae04766794608b44bR73

^^ one liner adding sudo pip install --upgrade setuptools before the sudo pip install awsebcli --upgrade command.

@vitr
Copy link

vitr commented Feb 28, 2019

@ryansimms thanks for replaying! I believe it works for some configurations, but the main difference in my CI is that I use

machine: true

and you're inside some docker image

    docker:
      - image: circleci/ruby:2.4.3

I ended up with a docker way fix, I post it here as alternative solution, if you don't mind

      - run:
          name: Making AWS profile
          command: |
            mkdir  ~/.aws
            echo -e "[profile eb-cli]\naws_access_key_id=$AWS_ACCESS_KEY_ID\naws_secret_access_key=$AWS_SECRET_ACCESS_KEY\n" > ~/.aws/config

      - run:
          name: Deploying to Elastic Beanstalk
          working_directory: /home/circleci/project/my-project/
          command: |
            docker run --rm  -v $PWD:/aws -v ~/.aws:/root/.aws vitr/aws-eb-cli deploy

Here I use pre-built docker image with AWS EB Cli inside https://hub.docker.com/r/vitr/aws-eb-cli
This allowed me to speed up my CI almost twice. With apt and pip commands EB deployment of my project took about 2 mins, now it takes just 1 min, as, I guess, installing eb cli with all pip dependencies is a way longer process than pulling up a docker image with pre-installed eb cli.

@ryansimms
Copy link
Author

@vitr Sounds good! We have used the config on this gist on a number of projects and find that deployments do tend to slow down over time so anything to help speed things up is always welcome, thanks for sharing!

@princedhimanz
Copy link

Thanks @ryansimms it work well for nodeJS also by making some changes.Thanks a lot dude

@Klaney
Copy link

Klaney commented Apr 11, 2019

Thanks @ryansimms for the quick guide, just a heads up to anybody else who may be getting the No Environment found for EnvironmentName = 'whatever-env' error on deploy. Check to make sure your default_region in your .elasticbeanstalk/config.yml is set correctly. I didn't change it, and got that unhelpful error.

@upugo-dev
Copy link

 Complete output from command python setup.py egg_info:
    error in awsebcli setup command: Invalid environment marker: sys_platform == "win32" and python_version >= "3.6"

For anyone reading, I fixed this error by simply fixing the awsebcli to a known stable version.

      - run:
          name: Install python, pip, and the AWS EB CLI
          command: |
            sudo apt-get install python-pip python-dev
            sudo pip install awsebcli==3.14.11

Great, I can get the eb cli installed but still can't run eb deploy! I can run eb deploy from my local machine, and my .elasticbeanstalk/config.yml file is committed to the repo that CircleCi is checking out. Any idea what is going on, @ryansimms?

@ryansimms
Copy link
Author

@upugo-dev what error do you get when the deploy command runs?

@upugo-dev
Copy link

upugo-dev commented Jun 27, 2019

@ryansimms thanks so much for the reply. Sorry I wasn't very clear when I wrote my previous message; I was receiving the

ERROR: This directory has not been set up with the EB CLI
You must first run "eb init".

Which I have now fixed... drumroll please... I'd removed checkout from my circleci .config file, eb init was looking in a completely empty directory! I have now added the checkout line back in and it is working fine deployment-wise, but my app isn't running; I'm getting

┌──────────────────────────────────────────────────┐
│ npm update check failed │
│ Try running with sudo or get access │
│ to the local update config store via │
│ sudo chown -R $USER:$(id -gn $USER) /tmp/.config │
└──────────────────────────────────────────────────┘

for some reason, on my instance node logs. I'm thinking of clearing the npm cache and re-deploying.

@ryansimms
Copy link
Author

I'd removed checkout from my circleci .config file

@upugo-dev whoops! glad deployment is now working as expected. In regards to your app I'm afraid I'm not too versed with npm so probably not the best person to ask, but if you find a solution and think it's helpful for others finding this gist then please do share.

@upugo-dev
Copy link

Looking into it, its an issue with connecting to my mongo atlas db, so nothing to do with deployment at all. Deployment working perfectly now thanks in some part to you! cheers

@upugo-dev
Copy link

upugo-dev commented Jul 11, 2019

hey, I've had success without locking the awsebcli version by doing:

- run: 
          name: Install python, pip, and the AWS EB CLI
          command: |
            sudo apt update
            sudo apt install -y python-pip python-dev
            sudo pip install awsebcli

credit to: https://discuss.circleci.com/t/eb-command-not-found/24461

this is on circleci docker image - image: circleci/node:10.15.1

@rickywid
Copy link

Thanks. This worked perfectly.

I was first getting a

ERROR: This directory has not been set up with the EB CLI
You must first run "eb init".

To fix this I had .elasticbeanstalk folder inside my .gitignore file. I simply removed it and pushed the changes and everything seemed to work fine.

@IsaiahJTurner
Copy link

 deploy:
    working_directory: ~/zu
    docker:
      - image: circleci/ruby:2.6.2
    steps:
      - checkout
      - run:
          name: Installing deployment dependencies
          working_directory: /
          command: |
            sudo apt-get -y -qq update
            sudo apt-get install python3-pip python3-dev build-essential
            sudo pip3 install awsebcli

worked for me. awsebcli supports python3 now

@greendinosaur
Copy link

Thanks for sharing this. It was an excellent starting point for me to use circleci to deploy to Elastic Beanstalk.

If you don't want to store your config file in your repo, you can also use circleci environment variables and use additional eb commands.
Within CircleCI, I created a context called aws and used this to store the application name and environment name as environment variables. I then added the following job to the circleci config.yml file. Make sure you use the aws context when calling the job.

I've also used the build number ($CIRCLE_BUILD_NUM) as the description for the deployment package.

  deploy:
    docker:
      - image: circleci/python:3.7.7
    steps:
      - checkout
      - run:
          name: installing aws dependencies
          command: |
            sudo pip install --upgrade setuptools
            sudo pip install awsebcli --upgrade
      - run:
          name: deploying
          command: |
            eb init --platform 'Python 3.7 running on 64bit Amazon Linux 2' --region 'us-east-2' $BEANSTALK_APPLICATION_NAME
            eb use $BEANSTALK_ENVIRONMENT_NAME
            eb deploy $BEANSTALK_ENVIRONMENT_NAME --message $CIRCLE_BUILD_NUM

@ryansimms
Copy link
Author

Updated gist based on @IsaiahJTurner 's suggestion (cheers!) as it seems Python2 is being phased out which was resulting in errors like:

pyrsistent requires Python '>=3.5' but the running Python is 2.7.13

@ryansimms
Copy link
Author

For those who may experience the error:

No matching distribution found for botocore<1.22.0,>=1.21.0 (from awsebcli)

A temporary fix seems to be replacing sudo pip install awsebcli --upgrade with sudo pip3 install awsebcli botocore==1.19.63 --upgrade as mentioned here.

@toluwalope19
Copy link

@DeBraid were you able to solve the linking to s3 issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment