Skip to content

Instantly share code, notes, and snippets.

@antoinedelia
Last active April 2, 2024 08:29
Show Gist options
  • Save antoinedelia/4060330cb5b1bba77371b9417b921b7b to your computer and use it in GitHub Desktop.
Save antoinedelia/4060330cb5b1bba77371b9417b921b7b to your computer and use it in GitHub Desktop.
How to make a Python pandas Layer for AWS Lambda

How to make pandas as an AWS layer

  1. Install pandas to local directory with pip: pip install -t . pandas

  2. Remove pandas and numpy directories rm -r pandas numpy

  3. Download the Linux distribution for pandas (choose the Python version that you want to use): https://pypi.org/project/pandas/#files

  4. Download the Linux distribution for numpy (must be the same as the pandas one): https://pypi.org/project/numpy/#files

  5. Move the downloaded files to your current directory and unzip them: unzip numpy-1.16.1-cp36-cp36m-manylinux1_x86_64.whl and unzip pandas-0.24.1-cp36-cp36m-manylinux1_x86_64.whl

    a. Note: Lambda with Python 3.7 requires pytz lib: $ pip install -t . pytz

  6. Remove unecessary files: rm -r *.whl *.dist-info __pycache__

  7. Make the following directory (change the version of Python according to what you downloaded earlier): lambda_layers/python/lib/python3.8/site-packages/.

  8. Move your whole directory into the site-packages folder.

  9. Go to cd lambda_layers and run: zip -r pandas_lambda_layer.zip *

  10. Upload the zip to S3.

  11. Create a Layer that uses the file uploaded to S3.

  12. Create a Lambda function (must uses the same Python version than before).

  13. You can test that everything works by using the code in lambda_function.py

  14. Have fun with pandas!

Sources:

@amzar96
Copy link

amzar96 commented Oct 3, 2023

After a few tried, I got this error

Response
{
  "errorMessage": "Unable to import module 'lambda_function': Error importing numpy: you should not try to import numpy from\n        its source directory; please exit the numpy source tree, and relaunch\n        your python interpreter from there.",
  "errorType": "Runtime.ImportModuleError",
  "stackTrace": []
}

Is it expected?

@ashfaqsharif
Copy link

@amzar96 Did you managed to fic the issue? I am facing the similar.

@amzar96
Copy link

amzar96 commented Apr 2, 2024

@amzar96 Did you managed to fic the issue? I am facing the similar.

You must use the same Python version you define when creating the layer.

  1. Create virtualenv in local
  2. pip install libraries
  3. follow the above step

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment