Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Install Python 3.6 in Amazon Linux
# A virtualenv running Python3.6 on Amazon Linux/EC2 (approximately) simulates the Python 3.6 Docker container used by Lambda
# and can be used for developing/testing Python 3.6 Lambda functions
# This script installs Python 3.6 on an EC2 instance running Amazon Linux and creates a virtualenv running this version of Python
# This is required because Amazon Linux does not come with Python 3.6 pre-installed
# and several packages available in Amazon Linux are not available in the Lambda Python 3.6 runtime
# The script has been tested successfully on a t2.micro EC2 instance (Root device type: ebs; Virtualization type: hvm)
# running Amazon Linux AMI 2017.03.0 (HVM), SSD Volume Type - ami-c58c1dd3
# and was developed with the help of AWS Support
# The steps in this script are:
# - install pre-reqs
# - install Python 3.6
# - create virtualenv
# install pre-requisites
sudo yum -y groupinstall development
sudo yum -y install zlib-devel
sudo yum -y install openssl-devel
# Installing openssl-devel alone seems to result in SSL errors in pip (see https://medium.com/@moreless/pip-complains-there-is-no-ssl-support-in-python-edbdce548852)
# Need to install OpenSSL also to avoid these errors
wget https://github.com/openssl/openssl/archive/OpenSSL_1_0_2l.tar.gz
tar -zxvf OpenSSL_1_0_2l.tar.gz
cd openssl-OpenSSL_1_0_2l/
./config shared
make
sudo make install
export LD_LIBRARY_PATH=/usr/local/ssl/lib/
cd ..
rm OpenSSL_1_0_2l.tar.gz
rm -rf openssl-OpenSSL_1_0_2l/
# Install Python 3.6
wget https://www.python.org/ftp/python/3.6.0/Python-3.6.0.tar.xz
tar xJf Python-3.6.0.tar.xz
cd Python-3.6.0
./configure
make
sudo make install
cd ..
rm Python-3.6.0.tar.xz
sudo rm -rf Python-3.6.0
# Create virtualenv running Python 3.6
sudo pip install --upgrade virtualenv
virtualenv -p python3 MYVENV
source MYVENV/bin/activate
python --version
# Python 3.6.0
@jgranq

This comment has been minimized.

Copy link

@jgranq jgranq commented Sep 9, 2017

python --version
prints
Python 2.7.12

but

python3 --version
prints
Python 3.6.0

@hughesjj

This comment has been minimized.

Copy link

@hughesjj hughesjj commented Dec 2, 2017

I would recommend using a newer version of openssl and python 3.6 with this script (currently, openssl 1.1.0g and and python 3.6.3)

@shax71

This comment has been minimized.

Copy link

@shax71 shax71 commented Dec 9, 2017

Very helpful. Thank you.

@gilinachum

This comment has been minimized.

Copy link

@gilinachum gilinachum commented Dec 19, 2017

One can save time compiling OpenSSL by yum installing it.

@colorlessmorning

This comment has been minimized.

Copy link

@colorlessmorning colorlessmorning commented Mar 2, 2018

Really helpful.

@adeiltonsousa

This comment has been minimized.

Copy link

@adeiltonsousa adeiltonsousa commented Apr 26, 2018

Thanks!!

Only in important detail:
For Amazon Linux, pip is not installed, it will be neccessary:

$ curl -O https://bootstrap.pypa.io/get-pip.py
and
$ python3 get-pip.py --user

@weiliangz

This comment has been minimized.

Copy link

@weiliangz weiliangz commented May 28, 2018

This is really a helpful guide. Thanks a lot!

@dilsilva

This comment has been minimized.

Copy link

@dilsilva dilsilva commented Sep 4, 2018

Nice, thx!

@civanescu

This comment has been minimized.

Copy link

@civanescu civanescu commented Mar 14, 2019

For Python3.6.8 this script didn't work. After all the work it says pip configured with SSL but ssl library is missing.
The procedure I manage to successfully obtain was:
amazon-linux-extra install python3
sudo yum -y groupinstall development
sudo yum -y install zlib-devel
sudo yum -y install openssl-devel bzip2-devel
curl -O https://www.python.org/ftp/python/3.6.8/Python-3.6.8.tgz
tar xzvf Python-3.6.8.tgz
cd Python-3.6.8
./configure
make altinstall
yum remove python3

The result:
python3.6 --version

Python 3.6.8

I believe that if I would use make install I won't have to remove the old python 3.6.2 (that comes with AMI Extra).

@TJ-jack

This comment has been minimized.

Copy link

@TJ-jack TJ-jack commented Apr 13, 2019

Excellent steps few updates from me.

I was using RHEL 7.6 AWS.

My approach:

  1. sudo yum -y groupinstall development
  2. sudo yum -y install zlib-devel
  3. sudo yum -y install openssl-devel bzip2-devel
  4. sudo yum remove python
  5. curl -O https://www.python.org/ftp/python/3.6.8/Python-3.6.8.tgz
  6. tar xzvf Python-3.6.8.tgz
  7. cd Python-3.6.8
  8. sudo ./configure --enable-optimizations
  9. sudo make install
  10. python --version
  11. which python python3
  12. follow http://outofmyhead.olssonandjones.com/2018/02/24/how-to-install-python-3-x-on-amazon-ec2-instance/
@mcassuranceiq

This comment has been minimized.

Copy link

@mcassuranceiq mcassuranceiq commented May 26, 2019

Thanks!!

Only in important detail:
For Amazon Linux, pip is not installed, it will be neccessary:

$ curl -O https://bootstrap.pypa.io/get-pip.py
and
$ python3 get-pip.py --user

This worked for me Thanks!!!

@joshwolff1

This comment has been minimized.

Copy link

@joshwolff1 joshwolff1 commented Jul 12, 2019

Beautiful

@edthrn

This comment has been minimized.

Copy link

@edthrn edthrn commented Jul 20, 2019

Helpful, thanks 👍

@jeremyhofman

This comment has been minimized.

Copy link

@jeremyhofman jeremyhofman commented Sep 20, 2019

Excellent steps few updates from me.

I was using RHEL 7.6 AWS.

My approach:

  1. sudo yum -y groupinstall development
  2. sudo yum -y install zlib-devel
  3. sudo yum -y install openssl-devel bzip2-devel
  4. sudo yum remove python
  5. curl -O https://www.python.org/ftp/python/3.6.8/Python-3.6.8.tgz
  6. tar xzvf Python-3.6.8.tgz
  7. cd Python-3.6.8
  8. sudo ./configure --enable-optimizations
  9. sudo make install
  10. python --version
  11. which python python3
  12. follow http://outofmyhead.olssonandjones.com/2018/02/24/how-to-install-python-3-x-on-amazon-ec2-instance/

In addition, on step #8, the --enable-shared flag can/should be included as well. Leaving it out can yield the following error message if a package depends on packages that are cpython-based (I think that's why, anyway. In my case I'm working with a C++ library that has a Python wrapper).

ImportError: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory

@owfm

This comment has been minimized.

Copy link

@owfm owfm commented Sep 24, 2019

Would it work to turn this script into a Dockerfile and do testing as a container, rather than spinning up an EC2 instance?

@espoelstra

This comment has been minimized.

Copy link

@espoelstra espoelstra commented Jun 8, 2020

There is an amazonlinux container available with tags for 2018xxxx (Amazon Linux 1) and the default/latest points to Amazon Linux 2 I believe.

You can enable epel for the amazonlinux container using the amazon-linux-extras install epel and then when you yum install things or yum search it should be able to access most things from epel as well.

If you are doing this in Docker for a deployable "builder" you probably don't want to use yum install @development as that pulls in a huge amount of packages (123 when I just ran it, weighing in at 121mb downloaded and ~141mb installed according to an uninstall, while git seems to pull in about half of these, there are quite a few other likely unused packages as well). Using @development or yum groupinstall development might be less tedious if you are in hurry than honing in on only the packages you need to compile Python and/or native extensions by fixing failures one at a time.

Personally I'm both lazy and OCD so I've gotten tired of handling the various quirks as new Python versions come out and just use Pyenv to install different versions of Python than what are available in the official repositories (on many distros and macOS and Windows).

Sadly it appears in our specific use case we ended up needing to use centos:7 for our base image because the version of gcc coming from amazonlinux was having issues compiling our code where the centos:7 version worked consistently.

FROM centos:7

RUN yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

RUN yum install -y \
# shared build deps
    make \
    gcc-c++ \
    git \
# Python C extension deps
    proj-devel \
    unzip \
    autoconf \
    automake \
    libtool \
    cmake \
    proj-epsg \
    libpng \
    libpng-devel \
# pyenv/python only build deps
    xz-devel \
    zlib-devel \
    bzip2-devel \
    readline-devel \
    sqlite sqlite-devel \
    openssl-devel \
    libffi-devel \
    yum-utils \
    && yum clean all

# Install specific python 3 version and use that pip3 to get some pre-reqs
# Maybe don't add --enable-optimizations unless you are actually running intensive
# python apps in the container as it takes a looooong time to compile
# If only using pyenv for this initial Python install, no need to put the PATH and
# pyenv init things into the .bashrc for the root user or a container specific user
RUN curl https://pyenv.run | bash \
     && export PATH="/root/.pyenv/bin:$PATH" \
     && env PYTHON_CONFIGURE_OPTS="--enable-shared" $(pyenv which python-build) 3.6.10 /usr/local/ \
     && pip3 install awscli

Skeleton of an AmazonLinux Dockerfile, can basically swap out the first 3 lines of the centos version with this and get a similar result.

FROM amazonlinux
# The amazon-linux-extras enables epel instead of installing an rpm from a URL
RUN amazon-linux-extras install epel && \
   yum install -y \
# Convenience group that adds a bit too much
#    @development \
    git \
    yum-utils \
    && yum clean all
@serg06

This comment has been minimized.

Copy link

@serg06 serg06 commented Jan 21, 2021

Here's an easy copy-paste install script for Python 3.6.12 on the default Amazon Linux 2 EC2 instance, based on the replies above.

sudo yum -y groupinstall development
sudo yum -y install zlib-devel
sudo yum -y install openssl-devel bzip2-devel
sudo yum remove python
curl -O https://www.python.org/ftp/python/3.6.12/Python-3.6.12.tgz
tar xzvf Python-3.6.12.tgz
cd Python-3.6.12
sudo ./configure --enable-optimizations --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"
sudo make install

Then run Python3 with:

python3

And use pip with:

python3 -m pip
@ducsiu

This comment has been minimized.

Copy link

@ducsiu ducsiu commented Oct 6, 2021

Thank you so much !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment