How do I resolve the certificate expiration error that I receive when Lambda invokes dependencies for a Let's Encrypt certificate?

README.md

I get an SSL error when AWS Lambda invokes dependencies that use a Let's Encrypt certificate.

Short description

Some AWS Lambda .NET Core and Ruby runtimes are experiencing certificate errors due to an expired Let's Encrypt cross-signed DST Root CA X3. For compatibility purposes, Let's Encrypt certificates default to using a certificate chain that's cross-signed by the DST Root CA X3 certificate that expired on September 30, 2021. OpenSSL versions 1.0.2 and earlier return an error when one of the verification paths is invalid, which prevents the successful establishment of SSL/TLS connections.

The following resolution removes the expired CA from the CA bundle and forces the system to use the file provided by the layer instead of the file packaged with the base system. OpenSSL versions 1.0.2 and earlier are forced to validate Let's Encrypt certificates using the alternate path provided in the environment variables.

Important: The system's trust store is frequently updated to include new CA root certificates. The following resolution is applicable until a global patch is deployed by AWS. After the patch is deployed, be sure to roll back this resolution to avoid issues.

Resolution

1. Download a patched CA bundle file:

   wget https://cert-mitigation.s3.us-west-2.amazonaws.com/lambda-ca-bundle.zip -O ./lambda-ca-bundle.zip

2. Publish a Lambda layer that includes the .crt file that you downloaded in step 1:

   aws lambda publish-layer-version --layer-name ca-patch-211001 --zip-file fileb://lambda-ca-bundle.zip --compatible-runtimes runtimes

   Note: Replace runtimes with your function's appropriate runtime(s) from the following: dotnetcore1.0, dotnetcore2.0, dotnetcore2.1, dotnetcore3.1, ruby2.5, and ruby2.7.

   Next, add a layer to your Lambda function:

   aws lambda update-function-configuration --function-name your-function --layers your-layer-arn

   Note: Replace your-function with your function name and your-layer-arn with the Amazon Resource Name (ARN) for your layer. Be sure to note any environment variables that are returned from the preceding command.

   --or--

   If you have limitations using a layer to package your function, you can also bundle the ca-bundle.crt file (included in the ZIP archive from step 1) in your function's deployment package.

3. Set the environment variables SSL_CERT_DIR=/opt and SSL_CERT_FILE=/opt/ca-bundle.crt in your Lambda function's configuration. The SSL_CERT_FILE=/opt/ca-bundle.crt environment variable includes the bundle provided by the layer that you created earlier. For example:

   aws lambda update-function-configuration --function-name your-function --environment "Variables={SSL_CERT_FILE=/opt/ca-bundle.crt,SSL_CERT_DIR=/opt/}"

   Note: Replace your-function with your function name. If necessary, update the ca-bundle.crt path.

   Important: When you apply environment variables with the update-function-configuration command, everything in the Variables structure is replaced. To keep existing environment variables when you add new environment variables, include all existing values in your request.

   If the client requires an explicit parameter for the CA bundle path, update connection parameters on the client used for TLS connections in your function's code. The connection parameters must direct the TLS client to point to the certificates under /opt/ca-bundle.crt.

4. Invoke your Lambda function again to confirm that the issue is resolved.