Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
ImageMagick Static Binaries for AWS Lambda
#!/usr/bin/env bash
# Must be run on an Amazon Linux AMI that matches AWS Lambda's runtime which can be found at:
# https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html
#
# As of May 21, 2019, this is:
# Amazon Linux AMI 2018.03.0 (ami-0756fbca465a59a30)
#
# You need to prepend PATH with the folder containing these binaries in your Lambda function
# to ensure these newer binaries are used.
#
# In a NodeJS runtime, you would add something like the following to the top of
# your Lambda function file:
# process.env['PATH'] = process.env['LAMBDA_TASK_ROOT'] + '/imagemagick/bin:' + process.env['PATH']
#
# This works with both ImageMagick v6.x and v7.x
# version=6.9.10-23
version=7.0.8-45
sudo yum -y install libpng-devel libjpeg-devel libtiff-devel gcc
curl -O https://imagemagick.org/download/ImageMagick-$version.tar.gz
tar zxvf ImageMagick-$version.tar.gz
cd ImageMagick-$version
./configure --prefix=/var/task/imagemagick --enable-shared=no --enable-static=yes
make
sudo make install
tar zcvf ~/imagemagick.tgz /var/task/imagemagick/
@jvrraghu

This comment has been minimized.

Copy link

@jvrraghu jvrraghu commented Apr 27, 2017

How to install imagemagick on EC2 instance with shared libraries as binary file.

@andypmw

This comment has been minimized.

Copy link

@andypmw andypmw commented Oct 2, 2017

It is helpful. Thanks for the gist

@baardoopic

This comment has been minimized.

Copy link

@baardoopic baardoopic commented Oct 2, 2017

Thanks for the code! Could you comment on how to then include the binaries in the Lambda script? We are using

var gm = require('gm').subClass({ imageMagick: true });

and it seems to still use the preinstalled ImageMagick version. Any tip would be greatly appreciated.

@mephir

This comment has been minimized.

Copy link

@mephir mephir commented Oct 12, 2017

@baardoopic You can see here how im using it https://github.com/mephir/aws-lambda-image/tree/webp-support. Basically whole magic is about overriding PATH and LD_LIBRARY_PATH. In releases section you will find already compiled binaries for aws lambda from latest im 7.0.7 source. BTW lambda does not require static-enabled.

@ezioruan

This comment has been minimized.

Copy link

@ezioruan ezioruan commented Oct 17, 2017

   var IM_PATH = process.env['LAMBDA_TASK_ROOT'] + "/imagemagick/bin/";
   process.env['LD_LIBRARY_PATH'] = '/var/task/imagemagick/lib/';
   process.env['PATH'] = process.env['PATH'] + ':' + IM_PATH;
   var gm = require('gm').subClass({
     imageMagick: true,
    appPath: '/var/task/imagemagick/bin/',
  });

remember to put the imagemagick floder in the root folder

@redserpent7

This comment has been minimized.

Copy link

@redserpent7 redserpent7 commented Nov 23, 2017

Is it possible to use this with Java?

@Enase

This comment has been minimized.

Copy link

@Enase Enase commented Mar 22, 2018

Compiled files are more than 250 Mb. Should I copy all the files?

@bensie

This comment has been minimized.

Copy link
Owner Author

@bensie bensie commented Jan 10, 2019

Gist updated:

  • Confirm support for latest ImageMagick v6.x and v7.x
  • Add note that using the Lambda-specific AMI is required
  • Add example code for prepending to PATH so calls look for the binaries in the right place
@lefthand

This comment has been minimized.

Copy link

@lefthand lefthand commented Jan 10, 2019

The share folder can be deleted successfully to help with file size. I also found that there are a handful of executables listed in bin all symlinked to magick. When I deployed with Sam, the zipping process turned all of the symlinks into full files (16mb each!). As such I've been removing any executable that I'm not using.

@leviwilson

This comment has been minimized.

Copy link

@leviwilson leviwilson commented Jan 30, 2019

@lefthand how much did you have to remove to allow it to be uploaded as a layer? I keep bumping into the size constraints...presumably b/c of what you'd mentioned with the symlinks :/

@puneeth8994

This comment has been minimized.

Copy link

@puneeth8994 puneeth8994 commented Apr 5, 2019

What about installing packages like freetype?
The fonts don't seem to work without it.

@chaddjohnson

This comment has been minimized.

Copy link

@chaddjohnson chaddjohnson commented Apr 11, 2019

It is possible to store symlinks as symlinks in zip files using the --symlinks option with the UNIX zip utility; for example

zip -q -x *.git* --symlinks -r ./deploy-package.zip .
@lefthand

This comment has been minimized.

Copy link

@lefthand lefthand commented May 16, 2019

@leviwilson I remove everything other than convert, identify, and magick in the /bin folder.

@chaddjohnson Unfortunately we don't have control over how Sam zips the files... yet.

@bensie

This comment has been minimized.

Copy link
Owner Author

@bensie bensie commented May 16, 2019

Gist updated:

  • The Lambda execution environment will invoke functions using Amazon Linux 2018.03 as its base as of May 21, 2019

This script has not yet been tested on the Amazon Linux 2 execution environment used by the nodejs10.x runtime.

@chaddjohnson

This comment has been minimized.

Copy link

@chaddjohnson chaddjohnson commented May 18, 2019

Has anyone experienced ImageMagick running considerably slower when compiled manually for Lambda?

We tried compiling both ImageMagick 6 and 7, and we also tried the same compilation options that the default ImageMagick install with Lambda uses, but things always run much more slowly.

@mt-sergio

This comment has been minimized.

Copy link

@mt-sergio mt-sergio commented May 27, 2019

I tried with nodejs10.x and latest Amazon Linux 2 without success.

Command failed: convert
convert: error while loading shared libraries: libtiff.so.5: cannot open shared object file: No such file or directory

When I add the files into the package they go under /opt folder, which is expected and automatically into $PATH (/opt/bin and /opt/lib), but maybe other folder are also critical.

Solution1: Just needed to run ldd /var/task/imagemagick/bin/convert, get that long list down to my computer, and add into the lib/ folder. 🤷‍♂️ .

Just got new one issue:

convert: UnableToOpenConfigureFile 'colors.xml' @ warning/configure.c/GetConfigureOptions/699.

Solution2: Include etc,shared and include imagemagick in serverless folder. Then, these .xml files will be discovered as they will be refereed correctly.

Issue3: I need gs :( I guess I will need to compile with ghostscript...

Why doesn't Imagemagick exist in https://github.com/mthenw/awesome-layers ??

@jp928

This comment has been minimized.

Copy link

@jp928 jp928 commented Jun 1, 2019

In my case, I can't get permission to execute GraphicsMagick from /var/task.
I copied to /tmp and give it permission 755 as mentioned: https://stackoverflow.com/questions/27708573/aws-lambda-making-video-thumbnails/29001078#29001078
Do you mind to share how did you overcome this issue?

@ronaksavla

This comment has been minimized.

Copy link

@ronaksavla ronaksavla commented Jul 19, 2019

I followed these steps to get Static Binaries for ImageMagic version 6.9.10-5
https://imagemagick.org/download/ImageMagick-6.9.10-55.tar.gz

Seems like the new update has removed convert modules. I confirmed with tech support
https://aws.amazon.com/blogs/compute/upcoming-updates-to-the-aws-lambda-execution-environment/

Lambda Code:
`var IM_PATH = "var/task/imagemagick/bin/";
process.env['LD_LIBRARY_PATH'] = 'var/task/imagemagick/lib/';
process.env['PATH'] = process.env['PATH'] + ':' + IM_PATH + ':' + process.env['LD_LIBRARY_PATH'];

var gm = require('gm').subClass({
imageMagick: true,
appPath: 'var/task/imagemagick/bin/',
});

gm('image.png')
.resize(240, 240, '!')
.write('/tmp/resize.png', function (err) {
if (!err) console.log('done');
else {
console.log(err);
}
});
`

Folder structure for Node js 8.10 & Node js 10.x lambdas:
image

Node gm version: "1.23.1"

Here are following errors:

  1. Node js 10.x lambda:
{ Error: Command failed: var/task/imagemagick/bin/convert: error while loading shared libraries: libgomp.so.1: cannot open shared object file: No such file or directory

    at ChildProcess.onExit (/var/task/node_modules/gm/lib/command.js:301:17)
    at ChildProcess.emit (events.js:189:13)
    at ChildProcess.EventEmitter.emit (domain.js:441:20)
    at maybeClose (internal/child_process.js:970:16)
    at Socket.stream.socket.on (internal/child_process.js:389:11)
    at Socket.emit (events.js:189:13)
    at Socket.EventEmitter.emit (domain.js:441:20)
    at Pipe._handle.close (net.js:597:12) code: 127, signal: null }
  1. Node js 8.10 lambda:
{ Error: Command failed: convert: UnableToOpenConfigureFile `delegates.xml' @ warning/configure.c/GetConfigureOptions/677.
convert: NoDecodeDelegateForThisImageFormat `PNG' @ error/constitute.c/ReadImage/560.
convert: NoImagesDefined `/tmp/resize.png' @ error/convert.c/ConvertImageCommand/3235.

    at ChildProcess.onExit (/var/task/node_modules/gm/lib/command.js:301:17)
    at emitTwo (events.js:126:13)
    at ChildProcess.emit (events.js:214:7)
    at maybeClose (internal/child_process.js:925:16)
    at Socket.stream.socket.on (internal/child_process.js:346:11)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at Pipe._handle.close [as _onclose] (net.js:567:12) code: 1, signal: null }
@peshkov3

This comment has been minimized.

Copy link

@peshkov3 peshkov3 commented Jul 26, 2019

Hi @ronaksavla. Have you find the solution?

@chaddjohnson

This comment has been minimized.

Copy link

@chaddjohnson chaddjohnson commented Jul 29, 2019

Here are instructions that worked well for my team and our app. Our app works with large images (e.g., 9000x13000) so it can use a lot of memory. ImageMagick 6 and uses less memory, so we use that.

  1. See https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html. The AMI amzn-ami-hvm-2017.03.1.20170812-x86_64-gp2 was used for these instructions.
  2. On the server, download the latest version of ImageMagick 6 (not 7 as it uses too much memory) from https://imagemagick.org/download/.
  3. Extract the source.
  4. Install dependencies: sudo yum install libpng-devel libjpeg-devel libtiff-devel gcc ImageMagick-devel xz-devel.x86_64 fontconfig-devel.x86_64 libxml2-devel.x86_64 libtool-ltdl-devel.
  5. Run ./configure --prefix=/var/task/imagemagick --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared=no --enable-static=yes --with-modules --with-perl=no --with-x=no --with-gslib=no --with-lcms2 --without-rsvg --with-xml --without-dps --without-included-ltdl --disable-hdri --with-quantum-depth=8 --disable-openmp in the source directory.
    1. Note the intentional use of --with-gslib=no. This forces use of the gs binary rather than the system library. See this discussion for details.
  6. Run make in the source directory.
  7. Run sudo make install in the source directory.
  8. Copy only the needed built files from the /var/task/imagemagick/bin directory into a local clone of your repository. If building ImageMagick 7, symlinks are present in the bin directory, so be sure to preserve symlinks using zip --symlinks or tar zcf.
  9. Commit the updated files.

We also had issues working with PDF files. To work around that, we did the following:

  1. In the instructions above, ensure use of --with-gslib=no.
  2. Download the latest version of Ghostscript for Linux x86 (64 bit) from https://www.ghostscript.com/download/gsdnld.html.
  3. Copy the gs-*-linux-x86_64 executable to bin/gs in your repository.
  4. Ensure the bin/gs binary is executable via chmod +x bin/gs.

Then in our Lambda function, above the handler definition, we include the following:

process.env['PATH'] = `${process.env['LAMBDA_TASK_ROOT']}/bin:${process.env['PATH']}`;
process.env['MAGICK_CONFIGURE_PATH'] = '/etc/ImageMagick';
@danhouldsworth

This comment has been minimized.

Copy link

@danhouldsworth danhouldsworth commented Aug 2, 2019

This thread has been super helpful - in that I now finally have a working version of IM with PDF support running on Lambda / Node v10.x
However it's also left me some questions :
@ronaksavla - did AWS say why they removed the convert modules? And do we mean the modules like /usr/lib64/ImageMagick-6.7.8/modules-Q16/coders/pdf.la ?
What caught me out during this journey is that Lambda/Node8.10 actually has ImageMagick 6.7.8.9-15.21.amzn1 RPM already installed. (I'm not sure if this was a recent change?) But convert & identify etc work natively out of the box, however it doesn't include PDF support. @bensie - with your script above, I'm pretty sure that won't give you extras such as PDF support, and will rely on the libs / modules included in the Lambda AMI for Node8.10. In which case - why not just use the ImageMagick 6.7.8.9 already in Lambda/Node8.10 ?
@chaddjohnson - thanks for the detailed tips on GS / PDF / gslib config. I found when using Lambda Node10.x I then also had to copy across many of the missing libs & config xmls like delegate.xml. Did you do this also? Or is this using Node8.10 and using the preinstalled IM libs?

@chaddjohnson

This comment has been minimized.

Copy link

@chaddjohnson chaddjohnson commented Aug 2, 2019

@danhouldsworth Glad that was helpful. I am still running this stuff on 8.10 — haven’t tried it in 10.x and don’t plan to for a while. I’m using preinstalled stuff except for the convert binary I compiled and the gs bin I added.

@adnascent

This comment has been minimized.

Copy link

@adnascent adnascent commented Aug 13, 2019

@chaddjohnson Thank you for sharing!
@danhouldsworth I had the copy the libraries over as well.
@jp928 I had to do the same thing, copy all the files in the tmp directory and chmod them in code. Then set PATH. Im not sure what everyone else is doing.

For the missing libraries I had to run the command "ldd" on the binaries to see what libraries were missing. EX:
ldd convert

@corbfon

This comment has been minimized.

Copy link

@corbfon corbfon commented Aug 17, 2019

@adnascent the ldd convert command ended up missing some libraries for me, such as libxml2.so.2 which is not listed as a shared lib for either convert or gs. Were you also checking shared libs for any other library?

I've been trying to get this working over the past couple of weeks on node 10 (also tried downgrading to node 8 and got the same errors as @ronaksavla). I'm considering switching gears and trying this with Lambda Layers. Has anyone tried with Layers before?

@corbfon

This comment has been minimized.

Copy link

@corbfon corbfon commented Aug 21, 2019

I've had success using layers like this on node10.x - https://github.com/serverlesspub/imagemagick-aws-lambda-2 - it worked out of the box, and then with some custom modifications. Going to approach ghostscript the same way.

@chaddjohnson

This comment has been minimized.

Copy link

@chaddjohnson chaddjohnson commented Jan 10, 2020

Updated instructions for nodejs10.x runtime

I was able to get this working. This should also work for nodejs12.x.

ImageMagick

Before proceeding, set up a Lambda Execution Environment. See https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html. The AMI amzn2-ami-hvm-2.0.20190313-x86_64-gp2 was used for this compilation.

  1. On the server, download the latest version of ImageMagick 6 or 7 from https://imagemagick.org/download/.
  2. Extract the source.
  3. Install dependencies: sudo yum install libpng-devel libjpeg-devel libtiff-devel gcc ImageMagick-devel xz-devel.x86_64 fontconfig-devel.x86_64 libxml2-devel.x86_64 libtool-ltdl-devel lcms2-devel.
  4. Run ./configure --prefix=/var/task/imagemagick --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared=no --enable-static=yes --with-modules --with-perl=no --with-x=no --with-gslib=no --with-lcms --without-rsvg --with-xml --without-dps --disable-hdri --with-quantum-depth=8 --disable-openmp in the source directory.
    1. Note the intentional use of --with-gslib=no. This forces use of the gs binary rather than the system library. See this discussion for details.
  5. Run make in the source directory.
  6. Run sudo make install in the source directory.
  7. Copy only the needed built files from the /var/task/imagemagick/bin directory into a local clone of your repository.
    1. If building ImageMagick 7, symlinks are present in the bin directory, so be sure to preserve symlinks using zip --symlinks or tar zcf.
  8. Copy libbz2.so.1, libexpat.so.1, libfontconfig.so.1, libfreetype.so.6, libgs.so.9, libjbig.so.2.0, libjpeg.so.62, liblcms2.so.2, liblzma.so.5, libpng15.so.15, libtiff.so.5, and libxml2.so.2 to lib/ in your repository from the server in /usr/lib64/. Be sure to copy the files that these symlink to, not the symlinks.

Then in the Lambda function, above the handler definition, include the following:

process.env['PATH'] = `${process.env['LAMBDA_TASK_ROOT']}/bin:${process.env['PATH']}`;
process.env['MAGICK_CONFIGURE_PATH'] = `${process.env['LAMBDA_TASK_ROOT']}/etc/ImageMagick`;

I also copied /etc/ImageMagick from the ami instance into my repository and am setting MAGICK_CONFIGURE_PATH -- I don't know if this is necessary:

process.env['MAGICK_CONFIGURE_PATH'] = `${process.env['LAMBDA_TASK_ROOT']}/etc/ImageMagick`;

Ghostscript (for PDF support)

  1. Download the latest version of Ghostscript for Linux x86 (64 bit) from https://www.ghostscript.com/download/gsdnld.html.
  2. Copy the gs-*-linux-x86_64 executable to bin/gs in your repository.
  3. Ensure the bin/gs binary is executable via chmod +x bin/gs.
  4. Copy libgs.so.9 to lib/ from the server in /usr/lib64/. Be sure to copy the file that this symlinks to, not the symlink.
@singhadarsh93

This comment has been minimized.

Copy link

@singhadarsh93 singhadarsh93 commented Jan 23, 2020

@chaddjohnson can you please write command for step 7 and 8 I am very new to linux , it will help , I am just stuck in this,
also there is no libexpat library in usr/lib64

@chaddjohnson

This comment has been minimized.

Copy link

@chaddjohnson chaddjohnson commented Jan 23, 2020

@singhadarsh93 Should be something like this:

  1. Run approximately these commands:

    On the server:

    cd /var/task/imagemagick
    tar zcf bin.tar.gz bin/convert bin/composite bin/identify
    

    On your computer (replace xxx.xxx.xxx.xxx with the server IP):

    ssh -i key.pem ec2-user@xxx.xxx.xxx.xxx 'cat /var/task/imagemagick/bin.tar.gz' > bin.tar.gz
    tar zxf bin.tar.gz
    

    Then there should be a bin directory on your local machine with the binaries.

  2. Run approximately these commands:

    On the server:

    mkdir /home/ec2-user/lib
    cd /usr/lib64
    cp -L libbz2.so.1 libexpat.so.1 libfontconfig.so.1 libfreetype.so.6 libgs.so.9 libjbig.so.2.0 libjpeg.so.62 liblcms2.so.2 liblzma.so.5 libpng15.so.15 libtiff.so.5 and libxml2.so.2 /home/ec2-user/lib/
    cd /home/ec2-user
    tar zcf lib.tar.gz lib/
    

    On your computer:

    ssh -i key.pem ec2-user@xxx.xxx.xxx.xxx 'cat lib.tar.gz' > lib.tar.gz
    tar zxf lib.tar.gz
    

    Then there should be a lib directory on your local machine with the .so files.

Not sure why libexpat isn't present. Maybe try yum install expat-devel. If that doesn't work, try searching via yum search expat, and install the expat devel 64-bit package.

@Rahul-Geeks

This comment has been minimized.

Copy link

@Rahul-Geeks Rahul-Geeks commented Apr 22, 2020

   var IM_PATH = process.env['LAMBDA_TASK_ROOT'] + "/imagemagick/bin/";
   process.env['LD_LIBRARY_PATH'] = '/var/task/imagemagick/lib/';
   process.env['PATH'] = process.env['PATH'] + ':' + IM_PATH;
   var gm = require('gm').subClass({
     imageMagick: true,
    appPath: '/var/task/imagemagick/bin/',
  });

remember to put the imagemagick floder in the root folder

I tried this after installing imagemagick in AWS lambda, but it automatically appends the path with "identify". And I dont know, where is that written.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.