Managing code dependencies in a project with nested Serverless and Node JS services.

serverless-code-dependency.md

How to get Serverless project file dependencies to work with Lambda

In AWS Lambdas, if you require a file like ../../../../lib you're out of luck. Serverless will not resolve that file correctly when creating the zip. This happens when the serverless.yml files are buried deep in your project directories. Node modules dependencies are handled correctly. (thats good).

To add lib, models, etc. correctly, I'm trying an approach using environment variables.

Here's our API directory structure. All of our shared files, like lib and models are at the root level. Many lambdas may need access to the same mode or library.

We have hundreds of lambdas and have run into issues when having so many in one serverless.yml file. This approach allows to have one project, The API, with many serverless.yml dividing it all up into manageable services that can share code.

projectRoot
  |__ lib // shared helpers
    |__ lib1.js
    |__ lib2.js
    |__ ...
  |__ models // shared models
    |__ service1Model.js
    |__ service2Model.js
    |__ ...
  |__ node_modules
  |__ test # shared test helpers
    |__ ...
  |__ package.json
  |__ services // lambdas that act like controllers
    |__service1
      |__ serverless.yml
      |__ tests
        |__ create_test.js
        |__ rpc1_test.js
        |__ ...
      |__ create.js
      |__ rpc1.js
      |__ ...
    |__service2
      |__ serverless.yml
      |__ tests
        |__ read_test.js
        |__ rpc_test.js
        |__ ...
      |__read.js
      |__rpc.js
      |__ ...
    |__ ...

Keep lambda dependencies small. Only include files that are needed for the function. Explicitly require only the models needed. In our case, all the libs are used. NOTE how we are setting environment variables.

# serverless.yml
service: service1
provider: aws

functions:
  create:
    handler: create.handler
    package:
      exclude:
      include:
        - create.js
        - ../../models/serviceModel1.js
        - ../../lib/**
    environment:
      LIB_PATH: lib
      MODELS_PATH: models

An example of what our lambda might look like. The important section is how we include the dependencies and how the path name is resolved using the environment variable found in process.env.

const lib = require(require('path').resolve(process.env.LIB_PATH, 'lib1'));
const model = require(require('path').resolve(process.env.MODELS_PATH, 'service1Model'));

module.exports.handler(event, context, callback) => {
  ...

Setting the environment variables for the local working project directory:

#!/bin/bash
export LIB_PATH="$(pwd)/lib";
export MODELS_PATH="$(pwd)/models";

Of course, these environment variables will need to be updated when changing projects.

Bonus

If you're using something like ava for TDD, you can export the env vars when you execute the test:

source scripts/set_env_vars.sh && ava -svw services/service1/test/foo_unit.js"

Alternate

A second option, using the environment variable approach, is to have the environment variable represent the structure at a higher level.

For example, the base of the project would contain a src directory that then contains the lib and models directories. This would only require defining one environment variable to represent the code path.

projectRoot
  |__ src
    |__ lib // shared helpers
      |__ lib1.js
      |__ lib2.js
      |__ ...
    |__ models // shared models
      |__ service1Model.js
      |__ service2Model.js
      |__ ...
  |__ node_modules
  |__ test # shared test helpers
    |__ ...
  |__ package.json
  |__ services // lambdas that act like controllers
    |__service1
      |__ serverless.yml
      |__ tests
        |__ create_test.js
        |__ rpc1_test.js
        |__ ...
      |__ create.js
      |__ rpc1.js
      |__ ...
    |__service2
      |__ serverless.yml
      |__ tests
        |__ read_test.js
        |__ rpc_test.js
        |__ ...
      |__read.js
      |__rpc.js
      |__ ...
    |__ ...

Finally

After 24 hours of thinking of this, simply use the environment variable to represent the project base. export BASE_PATH=/path/to/project. Leave models and lib at the root. In serverless.yml, set the Lambda environment variable to ./.

Then the require becomes:

const lib = require(require('path').resolve(process.env.BASE_PATH, 'lib/lib1'));
const model = require(require('path').resolve(process.env.BASE_PATH, 'model/service1Model'));

I love talking myself through these problems. ❤️

For my future self, we ended up relying heavily on NPM, node package manager. For directories that encapsulated functionality, we used a index.js and package.json files. Yarn is used to manage dependencies, using cached modules unless changes are made.