Skip to content

Instantly share code, notes, and snippets.

@nanvel
Forked from pnc/debugging.md
Created August 6, 2017 15:31
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save nanvel/633165f0878a303dcf1883019e13bc86 to your computer and use it in GitHub Desktop.
Save nanvel/633165f0878a303dcf1883019e13bc86 to your computer and use it in GitHub Desktop.
Debugging exit status 127 of custom binaries on AWS Lambda

AWS Lambda allows you to run custom binaries as child processes.

However, if the packaged binary you're running on AWS Lambda uses shared libraries, they may not be available in the Lambda environment. If this is the case, your binary will terminate without any output. In my case, the exit status code was 127, which wasn't very helpful (typically this is "command not found.")

2015-11-18T00:50:10.731Z	521db901-8d8e-11e5-b9df-cd31cc90ece2	Calling phantom:  /var/task/phantomjs [ '/var/task/phantomjs-script.js' ]
2015-11-18T00:50:10.809Z	521db901-8d8e-11e5-b9df-cd31cc90ece2	child process exited with code 127

Linux's loader, ld.so, allows you (see manpage) to set an environment variable called LD_DEBUG that will output verbose information while the shared libraries are loaded.

Since Lambda doesn't let you set arbitrary environment variables, you need to set the environment at runtime prior to starting your child process. In Node.js, this looks like:

  process.env['LD_DEBUG'] = 'all';

  // However your progam executes your binary, e.g.
  // var phantom = childProcess.execFile('./phantomjs', args);

Re-run your Lambda function and you should get verbose information (including errors!) from the loader:

 7:	
 7:	file=libicudata.so.50 [0];  needed by /var/task/phantomjs [0]
 7:	find library=libicudata.so.50 [0]; searching
 7:	 search cache=/etc/ld.so.cache
 7:	 search path=/lib64/tls/x86_64:/lib64/tls:/lib64/x86_64:/lib64:/usr/lib64/tls/x86_64:/usr/lib64/tls:/usr/lib64/x86_64:/usr/lib64		(system search path)
 7:	  trying file=/lib64/tls/x86_64/libicudata.so.50
 7:	  trying file=/lib64/tls/libicudata.so.50
 7:	  trying file=/lib64/x86_64/libicudata.so.50
 7:	  trying file=/lib64/libicudata.so.50
 7:	  trying file=/usr/lib64/tls/x86_64/libicudata.so.50
 7:	  trying file=/usr/lib64/tls/libicudata.so.50
 7:	  trying file=/usr/lib64/x86_64/libicudata.so.50
 7:	  trying file=/usr/lib64/libicudata.so.50
 7:	
/var/task/phantomjs: error while loading shared libraries: libicudata.so.50: cannot open shared object file: No such file or directory

In this case, the version of libicudata PhantomJS was compiled against is not available in the Lambda environment (more info here.). As Amazon indicates in their documentation, it's important to use binaries built on the actual Amazon Linux AMIs.

Also, since Amazon regularly updates Lambda's environment to track changes in Amazon Linux, it's best to compile custom binaries statically when possible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment