Skip to content

Instantly share code, notes, and snippets.

@allen-munsch
Forked from JamesChevalier/lambda_function.py
Last active November 28, 2023 17:35
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save allen-munsch/ad8faf9c04b72aa8d0808fa8953bc639 to your computer and use it in GitHub Desktop.
Save allen-munsch/ad8faf9c04b72aa8d0808fa8953bc639 to your computer and use it in GitHub Desktop.
How to install lxml in Amazon Lambda
sudo yum install update
sudo yum groupinstall -y "Development Tools"
sudo yum install -y python-devel libpng-devel freetype-devel gcc72-c++

If you want to use docker, you can use the following command:

  docker run -it lambci/lambda:build-python3.7 bash

Otherwise, launch an EC2 instance and ssh in. From there, we can use virtual environments to package our code.

virtualenv v-env
source v-env/bin/activate
pip install lxml
deactivate

mkdir python
cp -a ./v-env/lib/python3.7/site-packages/. ./python
cp -a ./v-env/lib64/python3.7/site-packages/. ./python

zip -r9 ../layer37.zip python

Which will give us a zipped folder containing a structure like

  ./python/
    lxml/
      ...

Where each of the underlying .so files correctly references the existing system libraries. We can use this structure as a layer. To download it, if you used docker, open another terminal window and run:

  docker ps (note the Container ID!)
  docker cp {container id}:/var/layer37.zip .

Or from the EC2 instance:

  scp -i {key location} ec2-user@{instance dns address}:layer37.zip ./layer.zip

We can add it to any existing Lambda functions which need lxml and don't currently have the improperly packaged lxml. To create the layer, navigate to the Lambda Layers console and follow the prompts to create a new layer, uploading the layer37.zip

file. Be careful to select the compatible runtime environments. Now, navigate to your Lambda function and add the layer! You'll be able to successfully import lxml.

START RequestId: 066db9dd-a447-446d-b6bc-85b885121dd7 Version: $LATEST
(4, 3, 4, 0)
completed all processes
END RequestId: 066db9dd-a447-446d-b6bc-85b885121dd7
REPORT RequestId: 066db9dd-a447-446d-b6bc-85b885121dd7	Duration: 1.31 ms	Billed Duration: 100 ms 	Memory Size: 3008 MB	Max Memory Used: 65 MB	

If you want to do this using a deployment package, simply navigate to the inside of your site packages folder, zip those files, and download that artifact to your local machine or build server. From there run zip -g {zip_name} {files/folders to include} where you might include for instance function.py

.

from lxml import etree
def lambda_handler(event, context):
return {'content' : 'ok'}

See: https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

  • Start a new EC2 instance with the Amazon Linux AMI
  • sudo yum install gcc gcc-c++ libjpeg-devel zlib-devel libevent-devel libxml2-devel libxslt-devel libpng-devel
  • sudo yum install python36-devel python36-pip
  • virtualenv venv_ok
  • source venv_ok/bin/activate
  • easy_install lxml
  • zip -9 bundle.zip lambda_function.py
  • for dir in $VIRTUAL_ENV/{lib,lib64}/python3.6/site-packages;do pushd $dir;zip -r9 $HOME/bundle.zip .;popd;done
  • Upload the bundle.zip file to your Lambda function
    • This assumes a default Handler set to lambda_function.lambda_handler
  • Delete your EC2 instance
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment