Skip to content

Instantly share code, notes, and snippets.

@marknca

marknca/install-build-env.sh

Last active January 14, 2016 05:43
Show Gist options
  • Save marknca/64727a6297d050752f44 to your computer and use it in GitHub Desktop.
Save marknca/64727a6297d050752f44 to your computer and use it in GitHub Desktop.
Download ZIP
Python extension build environment for AWS Lambda
Raw
install-build-env.sh
#! /usr/bin/env bash
# Install the required libraries for a Python build environment on Amazon Linux
# Note, you only really need lines #6 & 7, the rest of the lines install
# pre-requisities for the SciPy stack
yum -y update
yum -y groupinstall "Development Tools"
yum -y install Cython --enablerepo=epel
@astewart-twist
Copy link

I don't quite understand. How does installing Cython on an ec2 instance help with packaging extension modules for Lambda? Also, the --target option to pip does not appear to be working for Cython.

(env)[ec2-user@ip-172-31-43-163 ~]$ pip install Cython -t ~/build
You are using pip version 6.0.8, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting Cython
  Using cached Cython-0.23.4.tar.gz
    Unable to find pgen, not compiling formal grammar.
Installing collected packages: Cython
  Running setup.py install for Cython
    Unable to find pgen, not compiling formal grammar.
    cythoning /tmp/pip-build-sqlMPS/Cython/Cython/Plex/Scanners.py to /tmp/pip-build-sqlMPS/Cython/Cython/Plex/Scanners.c
    building 'Cython.Plex.Scanners' extension
    gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c /tmp/pip-build-sqlMPS/Cython/Cython/Plex/Scanners.c -o build/temp.linux-x86_64-2.7/tmp/pip-build-sqlMPS/Cython/Cython/Plex/Scanners.o
    gcc -pthread -shared build/temp.linux-x86_64-2.7/tmp/pip-build-sqlMPS/Cython/Cython/Plex/Scanners.o -L/usr/lib64 -lpython2.7 -o build/lib.linux-x86_64-2.7/Cython/Plex/Scanners.so
    cythoning /tmp/pip-build-sqlMPS/Cython/Cython/Plex/Actions.py to /tmp/pip-build-sqlMPS/Cython/Cython/Plex/Actions.c
    building 'Cython.Plex.Actions' extension
    gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c /tmp/pip-build-sqlMPS/Cython/Cython/Plex/Actions.c -o build/temp.linux-x86_64-2.7/tmp/pip-build-sqlMPS/Cython/Cython/Plex/Actions.o
    gcc -pthread -shared build/temp.linux-x86_64-2.7/tmp/pip-build-sqlMPS/Cython/Cython/Plex/Actions.o -L/usr/lib64 -lpython2.7 -o build/lib.linux-x86_64-2.7/Cython/Plex/Actions.so
    cythoning /tmp/pip-build-sqlMPS/Cython/Cython/Compiler/Lexicon.py to /tmp/pip-build-sqlMPS/Cython/Cython/Compiler/Lexicon.c
    building 'Cython.Compiler.Lexicon' extension
    gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c /tmp/pip-build-sqlMPS/Cython/Cython/Compiler/Lexicon.c -o build/temp.linux-x86_64-2.7/tmp/pip-build-sqlMPS/Cython/Cython/Compiler/Lexicon.o
    gcc -pthread -shared build/temp.linux-x86_64-2.7/tmp/pip-build-sqlMPS/Cython/Cython/Compiler/Lexicon.o -L/usr/lib64 -lpython2.7 -o build/lib.linux-x86_64-2.7/Cython/Compiler/Lexicon.so
    cythoning /tmp/pip-build-sqlMPS/Cython/Cython/Compiler/Scanning.py to /tmp/pip-build-sqlMPS/Cython/Cython/Compiler/Scanning.c
    building 'Cython.Compiler.Scanning' extension
    gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c /tmp/pip-build-sqlMPS/Cython/Cython/Compiler/Scanning.c -o build/temp.linux-x86_64-2.7/tmp/pip-build-sqlMPS/Cython/Cython/Compiler/Scanning.o
    gcc -pthread -shared build/temp.linux-x86_64-2.7/tmp/pip-build-sqlMPS/Cython/Cython/Compiler/Scanning.o -L/usr/lib64 -lpython2.7 -o build/lib.linux-x86_64-2.7/Cython/Compiler/Scanning.so
    cythoning /tmp/pip-build-sqlMPS/Cython/Cython/Compiler/Parsing.py to /tmp/pip-build-sqlMPS/Cython/Cython/Compiler/Parsing.c
    building 'Cython.Compiler.Parsing' extension
    gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c /tmp/pip-build-sqlMPS/Cython/Cython/Compiler/Parsing.c -o build/temp.linux-x86_64-2.7/tmp/pip-build-sqlMPS/Cython/Cython/Compiler/Parsing.o
    gcc -pthread -shared build/temp.linux-x86_64-2.7/tmp/pip-build-sqlMPS/Cython/Cython/Compiler/Parsing.o -L/usr/lib64 -lpython2.7 -o build/lib.linux-x86_64-2.7/Cython/Compiler/Parsing.so
    cythoning /tmp/pip-build-sqlMPS/Cython/Cython/Compiler/Visitor.py to /tmp/pip-build-sqlMPS/Cython/Cython/Compiler/Visitor.c
    building 'Cython.Compiler.Visitor' extension
    gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c /tmp/pip-build-sqlMPS/Cython/Cython/Compiler/Visitor.c -o build/temp.linux-x86_64-2.7/tmp/pip-build-sqlMPS/Cython/Cython/Compiler/Visitor.o
    gcc -pthread -shared build/temp.linux-x86_64-2.7/tmp/pip-build-sqlMPS/Cython/Cython/Compiler/Visitor.o -L/usr/lib64 -lpython2.7 -o build/lib.linux-x86_64-2.7/Cython/Compiler/Visitor.so
    cythoning /tmp/pip-build-sqlMPS/Cython/Cython/Compiler/FlowControl.py to /tmp/pip-build-sqlMPS/Cython/Cython/Compiler/FlowControl.c
    building 'Cython.Compiler.FlowControl' extension
    gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c /tmp/pip-build-sqlMPS/Cython/Cython/Compiler/FlowControl.c -o build/temp.linux-x86_64-2.7/tmp/pip-build-sqlMPS/Cython/Cython/Compiler/FlowControl.o
    gcc -pthread -shared build/temp.linux-x86_64-2.7/tmp/pip-build-sqlMPS/Cython/Cython/Compiler/FlowControl.o -L/usr/lib64 -lpython2.7 -o build/lib.linux-x86_64-2.7/Cython/Compiler/FlowControl.so
    cythoning /tmp/pip-build-sqlMPS/Cython/Cython/Compiler/Code.py to /tmp/pip-build-sqlMPS/Cython/Cython/Compiler/Code.c
    building 'Cython.Compiler.Code' extension
    gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c /tmp/pip-build-sqlMPS/Cython/Cython/Compiler/Code.c -o build/temp.linux-x86_64-2.7/tmp/pip-build-sqlMPS/Cython/Cython/Compiler/Code.o
    gcc -pthread -shared build/temp.linux-x86_64-2.7/tmp/pip-build-sqlMPS/Cython/Cython/Compiler/Code.o -L/usr/lib64 -lpython2.7 -o build/lib.linux-x86_64-2.7/Cython/Compiler/Code.so
    cythoning /tmp/pip-build-sqlMPS/Cython/Cython/Runtime/refnanny.pyx to /tmp/pip-build-sqlMPS/Cython/Cython/Runtime/refnanny.c
    building 'Cython.Runtime.refnanny' extension
    gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c /tmp/pip-build-sqlMPS/Cython/Cython/Runtime/refnanny.c -o build/temp.linux-x86_64-2.7/tmp/pip-build-sqlMPS/Cython/Cython/Runtime/refnanny.o
    gcc -pthread -shared build/temp.linux-x86_64-2.7/tmp/pip-build-sqlMPS/Cython/Cython/Runtime/refnanny.o -L/usr/lib64 -lpython2.7 -o build/lib.linux-x86_64-2.7/Cython/Runtime/refnanny.so
    cythoning /tmp/pip-build-sqlMPS/Cython/Cython/Tempita/_tempita.py to /tmp/pip-build-sqlMPS/Cython/Cython/Tempita/_tempita.c
    building 'Cython.Tempita._tempita' extension
    gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c /tmp/pip-build-sqlMPS/Cython/Cython/Tempita/_tempita.c -o build/temp.linux-x86_64-2.7/tmp/pip-build-sqlMPS/Cython/Cython/Tempita/_tempita.o
    gcc -pthread -shared build/temp.linux-x86_64-2.7/tmp/pip-build-sqlMPS/Cython/Cython/Tempita/_tempita.o -L/usr/lib64 -lpython2.7 -o build/lib.linux-x86_64-2.7/Cython/Tempita/_tempita.so
    Installing cython script to /tmp/tmposbS8p/bin
    Installing cygdb script to /tmp/tmposbS8p/bin
    Installing cythonize script to /tmp/tmposbS8p/bin
Successfully installed Cython
Exception:
Traceback (most recent call last):
  File "/home/ec2-user/env/local/lib/python2.7/site-packages/pip/basecommand.py", line 232, in main
    status = self.run(options, args)
  File "/home/ec2-user/env/local/lib/python2.7/site-packages/pip/commands/install.py", line 390, in run
    for item in os.listdir(lib_dir):
OSError: [Errno 2] No such file or directory: '/tmp/tmposbS8p/lib/python/'

The resulting ~/build directory is completely empty.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment