Encrypt secret configuration at rest in Vault. Access keys either:
1. At runtime using the API (either via REST or a client library, e.g. node-vault
- Encrypted variables never exposed to AWS/in console.
- Could be written in to a library and re-used based on approles.
- CI can remain ambivalent to secret management.
- All secret config access becomes asynchronous.
- Changes to secrets in vault require a lambda redeploy as AWS is unaware that anything has changed (could invalidate periodically and refetch)
- Vault downtime prevents new lambda containers from operating. Existing containers will be unaffected if loaded at startup.
- Approle has to be created per application. Kickstarter have open sourced a lambda to do this based on cloudformation updates (designed for ECS)
- Developers need to setup vault locally, local auth rather than approle must be used locally.
- CI needs to define vault secrets are present if reliable deployments are desired.
Unsure on what affect this has on cold-start time of the lambda.
e.g. CircleCI Deploy -> Fetch from Vault -> Publish to Lambda This is the approach next currently use
- Vault downtime does not affect the lambda.
- Secret access remains synchronous.
- Github/CI role can be used to access secrets.
- Vault secrets can be easily verified to exist by CI.
- Secrets now potentially stored in plaintext in AWS console/serverless deployment package.
- Secrets need to be rencrypted with KMS if not stored in plaintext.
- Developers need to setup vault locally and on CI.
It should be noted that all environment varaibles are encrypted automatically via AWS using a shared KMS key, however these variables are still visibile at rest by default.
There are two main serverless plugins in existent reflecting two main approaches for this.
https://docs.aws.amazon.com/lambda/latest/dg/env_variables.html#env_encrypt https://docs.aws.amazon.com/lambda/latest/dg/tutorial-env_console.html
This can then be decrypted using a snippet provided by AWS:
const AWS = require('aws-sdk');
const encrypted = process.env['PGPASSWORD'];
let decrypted;
function processEvent(event, context, callback) {
// TODO handle the event here
}
exports.handler = (event, context, callback) => {
if (decrypted) {
processEvent(event, context, callback);
} else {
// Decrypt code should run once and variables stored outside of the function
// handler so that these are decrypted once per container
const kms = new AWS.KMS();
kms.decrypt({ CiphertextBlob: new Buffer(encrypted, 'base64') }, (err, data) => {
if (err) {
console.log('Decrypt error:', err);
return callback(err);
}
decrypted = data.Plaintext.toString('ascii');
processEvent(event, context, callback);
});
}
};This incurs additional latency if decrypted at runtime source
latency to decrypt a secret using a KMS CMK is often around 2 seconds
- Uses existing AWS infrastructure.
- Only need to create a KMS key.
- Native to AWS lambda
- High latency (1-2s on cold starts)
- Need additional AWS permissions
- Not stored in vault.
- Not sharable between lamba projects.
- All secret config access becomes asynchronous.
Existing serverless plugin to handle encryption/decryption:
This incurs additional latency if decrypted at runtime source
latency to decrypt a secret using a KMS CMK is often around 2 seconds
- Uses existing AWS infrastructure.
- Only requires KMS
- Encrypted at rest and in transit
- Native to AWS lambda
- IAM KMS access handled by plugin
- High latency (1-2s on cold starts)
- Need additional AWS permissions
- Not stored in vault.
- Not sharable between lamba projects.
- All secret config access becomes asynchronous.
3. Encypt the variables at rest in SSM Parameter Store and decrypt prior to deploy
These values are then natively available in serverless reference on pros and cons at rest.
- Uses existing AWS infrastructure.
- Defined in serverless.yml natively
- Secret access doesn't change
- Validated on serverless deploy
- IAM role based
- Need additional AWS permissions
- Not stored in vault.
- Visibile in the AWS console.
- Visible using lambda
GetFunctionConfigurationAPI call. - Changes to secrets in KMS (not managed by lambda) potentially require a lambda redeploy as the lambda is unaware anything has changed (could refetch secrets periodically).
4. Encypt the variables at rest in SSM Parameter Store and access at runtime
Then access these at runtime using the SSM api.
https://github.com/SC5/serverless-kms-secrets
- Uses existing AWS infrastructure.
- Only need to create a KMS key.
- Low runtime cost to access SSM (~50ms max)
- IAM role based
- Need additional AWS permissions
- Not stored in vault.
- All secret config access becomes asynchronous.
- Changes to secrets in KMS (not managed by lambda) potentially require a lambda redeploy as the lambda is unaware anything has changed, alternatively invalidate secrets and re-fetch periodically
- CI needs to define KMS secrets are present if reliable deployments are desired.
Store variables in FT lastpass. Developers retrieve them and deploy locally/in circleCI Simplest solution, status quo.
- Simple, just add as environment variables.
- Each environment variable needs setting individually.
- No encryption at rest.
- Visible in the AWS console.
- Visible using lambda
GetFunctionConfigurationAPI call. - Not available programatically