Skip to content

Instantly share code, notes, and snippets.

@arkadiyt
Created December 15, 2021 17:07
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save arkadiyt/658fad75cee2b2979621f0f5a716a018 to your computer and use it in GitHub Desktop.
Save arkadiyt/658fad75cee2b2979621f0f5a716a018 to your computer and use it in GitHub Desktop.
Hello,
This message will help you determine if your Amazon SageMaker Canvas App is affected by the issue disclosed in CVE-2021-44228 [1] relating to Apache Log4j, and provide steps you can take to address the issue. Please note that these actions will not delete any of your datasets or model but will only apply the fix in the app needed for remediating the CVE.
1) Determine regions in which you have created a Canvas App.
2) Search Amazon SageMaker in AWS Console and click on it. This will take you to SageMaker Console Dashboard.
3) In the left Navigation, under “SageMaker Domain”, click on “Canvas”. This will take you to the list of domain users you have created.
4) Click on each domain user and follow these steps:
a. Domain user will take you to an individual listed user.
b. Scan the App Type column and if it says its “Canvas”, then click on the “Delete app” button.
c. Follow steps of “Delete app” as it asks you to confirm.
d. Once the app is deleted, it will change App Status from Ready/Pending to Deleted. It can take up to a minute for the App to be deleted.
e. You can relaunch Canvas from the menu item button on the right which says “Launch App” -> select “Canvas”. This will rebuild your Canvas App with the fix in the next 2-4 minutes.
5) Repeat steps 1 to 4 for other regions which have a Canvas App.
We ask that you perform these steps by 12:00 PM PST on December 15, 2021. If you are not able to, then SageMaker will force shutdown your app. This means that the next time you relaunch the Canvas app, it will need to be updated. However, since it has been auto-shutdown by the platform, it may take additional time to come back up when you relaunch it.
Should you have any questions or concerns please contact AWS Support [2].
[1] https://aws.amazon.com/security/security-bulletins/AWS-2021-006
[2] https://aws.amazon.com/support
Sincerely,
Amazon Web Services
Amazon Web Services, Inc. is a subsidiary of Amazon.com, Inc. Amazon.com is a registered trademark of Amazon.com, Inc. This message was produced and distributed by Amazon Web Services Inc., 410 Terry Ave. North, Seattle, WA 98109-5210
Hello,
This message will help you determine if your Amazon SageMaker endpoint is affected by the issue disclosed in CVE-2021-44228[1] relating to Apache Log4j, and provide steps you can take to address the issue.
Determining if your endpoint has been affected by the Apache Log4j issue:
While creating an Amazon SageMaker Model [2], you had an option to use either a prebuilt SageMaker Docker image [3] (such as a Deep Learning Container (DLC) image [4] or a SageMaker provided algorithm [5]) or bring your own custom-built container image.
If your SageMaker endpoint uses a Deep Learning Container (DLC) Image, your endpoint is not affected by the Apache Log4j issue and *no action is required*.
Some SageMaker-provided algorithms used a version of Apache Log4j affected by the issue disclosed in CVE-2021-44228. Therefore, if your endpoint was created before December 12, 2021 and uses a SageMaker-provided algorithm, we recommend that you update the endpoint to ensure that it uses the latest patched version of the image. (However, if your endpoint uses a SageMaker provided algorithm and was created after December 12, 2021, it will automatically use the latest patched image that addresses the issue and *no action* is required from you.)
If your SageMaker endpoint uses a custom-built container image, we recommend scanning your container image [6] and applying the necessary patches yourself before updating your endpoint.
The following are the steps for updating your endpoint:
There are 3 steps for updating your endpoint - 1/Creating the Model, 2/Creating the Endpoint Configuration, and 3/Updating the Endpoint:
Step 1 - Creating the Model: If you are using a prebuilt SageMaker Docker image and the image section [7] in the Container Definition [8] was configured to use the latest version of your container (such as the latest tag), then you can skip to step 2. Otherwise, you will need to create a new SageMaker Model with the patched version of the container specified in the image section in the Container Definition using the CreateModel API [2] action. You can copy over all the parameters except the container image path from your existing Model. In the image section of the Container Definition, specify the path to the patched container image.
Step 2 - Creating the Endpoint Configuration: Once you have created your SageMaker Model, clone your Endpoint Configurations to point to the newly created Model. This can be done by creating a new EndpointConfig using the CreateEndpointConfig [9] API action and specifying the name [10] of the Model from Step 1 that has the patched container image.
Please note that SageMaker Model and Endpoint Configuration are immutable and cannot be updated - you will need to create a new one with a different name to use in Step 3.
Step 3 - Updating the Endpoint: As the last step, update your endpoint using the UpdateEndpoint API action [11]. This will ensure that your SageMaker endpoint uses the latest patched version of the container image. If you are using autoscaling [12], we recommend that you set the RetainAllVariantProperties [13] to true while updating the endpoint. This will ensure that there are no changes to your current instance count and variant weights during the update. For instances that are in high demand (such as GPU or Inferentia instance types), you may encounter endpoint update failures due to insufficient capacity. If that happens, retry updating your endpoint.
You can validate that the updated endpoint is using the patched container image by calling the DescribeEndpoint API action [14] and checking the DeployedImages [15] section of the response.
Should you have any questions or concerns please contact AWS Support [16].
References:
[1] https://aws.amazon.com/security/security-bulletins/AWS-2021-006/
[2] https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateModel.html
[3] https://docs.aws.amazon.com/sagemaker/latest/dg/docker-containers-prebuilt.html
[4] https://github.com/aws/deep-learning-containers/blob/master/available_images.md
[5] https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-algo-docker-registry-paths.html
[6] https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-scanning.html
[7] https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ContainerDefinition.html#sagemaker-Type-ContainerDefinition-Image
[8] https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ContainerDefinition.html
[9] https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateEndpointConfig.html
[10] https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ProductionVariant.html#sagemaker-Type-ProductionVariant-ModelName
[11] https://docs.aws.amazon.com/sagemaker/latest/dg/API_UpdateEndpoint.html
[12] https://docs.aws.amazon.com/sagemaker/latest/dg/endpoint-auto-scaling.html
[13] https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_UpdateEndpoint.html#sagemaker-UpdateEndpoint-request-RetainAllVariantProperties
[14] https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeEndpoint.html
[15] https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_PendingProductionVariantSummary.html#sagemaker-Type-PendingProductionVariantSummary-DeployedImages
[16] https://aws.amazon.com/support
Sincerely,
Amazon Web Services
Amazon Web Services, Inc. is a subsidiary of Amazon.com, Inc. Amazon.com is a registered trademark of Amazon.com, Inc. This message was produced and distributed by Amazon Web Services Inc., 410 Terry Ave. North, Seattle, WA 98109-5210
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment