Skip to content

Instantly share code, notes, and snippets.

@irr

irr/aws_pandas_layer.md

Forked from antoinedelia/aws_pandas_layer.md
Created December 6, 2022 13:29
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save irr/c293b1f37511d2490859f9432af3ec6d to your computer and use it in GitHub Desktop.
Save irr/c293b1f37511d2490859f9432af3ec6d to your computer and use it in GitHub Desktop.
Download ZIP
How to make a Python pandas Layer for AWS Lambda
Raw
aws_pandas_layer.md

How to make pandas as an AWS layer

  1. Install pandas to local directory with pip: pip install -t . pandas

  2. Remove pandas and numpy directories rm -r pandas numpy

  3. Download the Linux distribution for pandas (choose the Python version that you want to use): https://pypi.org/project/pandas/#files

  4. Download the Linux distribution for numpy (must be the same as the pandas one): https://pypi.org/project/numpy/#files

  5. Move the downloaded files to your current directory and unzip them: unzip numpy-1.16.1-cp36-cp36m-manylinux1_x86_64.whl and unzip pandas-0.24.1-cp36-cp36m-manylinux1_x86_64.whl

    a. Note: Lambda with Python 3.7 requires pytz lib: $ pip install -t . pytz

  6. Remove unecessary files: rm -r *.whl *.dist-info __pycache__

  7. Make the following directory (change the version of Python according to what you downloaded earlier): lambda_layers/python/lib/python3.8/site-packages/.

  8. Move your whole directory into the site-packages folder.

  9. Go to cd lambda_layers and run: zip -r pandas_lambda_layer.zip *

  10. Upload the zip to S3.

  11. Create a Layer that uses the file uploaded to S3.

  12. Create a Lambda function (must uses the same Python version than before).

  13. You can test that everything works by using the code in lambda_function.py

  14. Have fun with pandas!

Sources:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment