Example of a bare-minimum terraform script to setup an API Gateway endpoint that takes records and puts them into an SQS queue.
Start by creating the SQS queue.
resource "aws_sqs_queue" "queue" {
name = "my-sqs-queue"
delay_seconds = 0 // how long to delay delivery of records
max_message_size = 262144 // = 256KiB, which is the limit set by AWS
message_retention_seconds = 86400 // = 1 day in seconds
receive_wait_time_seconds = 10 // how long to wait for a record to stream in when ReceiveMessage is called
}Now we need to define an IAM role so API Gateway has the necessary permissions to SendMessage to our SQS queue. These perms will give API Gateway the ability to create and write to CloudWatch logs, as well as the ability to put, read, and list data to the SQS queue.
resource "aws_iam_role" "api" {
name = "my-api-role"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "apigateway.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
}
resource "aws_iam_policy" "api" {
name = "my-api-perms"
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams",
"logs:PutLogEvents",
"logs:GetLogEvents",
"logs:FilterLogEvents"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"sqs:GetQueueUrl",
"sqs:ChangeMessageVisibility",
"sqs:ListDeadLetterSourceQueues",
"sqs:SendMessageBatch",
"sqs:PurgeQueue",
"sqs:ReceiveMessage",
"sqs:SendMessage",
"sqs:GetQueueAttributes",
"sqs:CreateQueue",
"sqs:ListQueueTags",
"sqs:ChangeMessageVisibilityBatch",
"sqs:SetQueueAttributes"
],
"Resource": "${aws_sqs_queue.queue.arn}"
},
{
"Effect": "Allow",
"Action": "sqs:ListQueues",
"Resource": "*"
}
]
}
EOF
}
resource "aws_iam_role_policy_attachment" "api" {
role = "${aws_iam_role.api.name}"
policy_arn = "${aws_iam_policy.api.arn}"
}Now define an API Gateway REST API. We will host the POST method at the root / of the API.
resource "aws_api_gateway_rest_api" "api" {
name = "my-sqs-api"
description = "POST records to SQS queue"
}Create a validation template to validate the POST request body. You can leave this out, but it comes in handy for transforming and customizing service responses, in addition to validating the actual request payload.
This template defines our payload should be of Content-Type: application/json and the json needs to contain an object with two keys; id, and docs, and docs should be an array of objects greater than or equal to 1. See the AWS docs on creating request validators with OpenAPI.
resource "aws_api_gateway_request_validator" "api" {
rest_api_id = "${aws_api_gateway_rest_api.api.id}"
name = "payload-validator"
validate_request_body = true
}
resource "aws_api_gateway_model" "api" {
rest_api_id = "${aws_api_gateway_rest_api.api.id}"
name = "PayloadValidator"
description = "validate the json body content conforms to the below spec"
content_type = "application/json"
schema = <<EOF
{
"$schema": "http://json-schema.org/draft-04/schema#",
"type": "object",
"required": [ "id", "docs"],
"properties": {
"id": { "type": "string" },
"docs": {
"minItems": 1,
"type": "array",
"items": {
"type": "object"
}
}
}
}
EOF
}Create an API Gateway POST method for the API. The request does not require the request_models or request_validator_id key, but as mentioned above it's useful for both validating the message body format and formatting our service responses, which will be demonstrated shortly.
resource "aws_api_gateway_method" "api" {
rest_api_id = "${aws_api_gateway_rest_api.api.id}"
resource_id = "${aws_api_gateway_rest_api.api.root_resource_id}"
api_key_required = false
http_method = "POST"
authorization = "NONE"
request_validator_id = "${aws_api_gateway_request_validator.api.id}"
request_models = {
"application/json" = "${aws_api_gateway_model.api.name}"
}
}Now we can define the API gateway integration that will forward records into the SQS queue. The request_templates defined is required in order to take the message body content and send it on to S3 (assuming the content type to be handled is json), as well as the request_parameters defined, since the SQS endpoint requires the data to be form encoded.
resource "aws_api_gateway_integration" "api" {
rest_api_id = "${aws_api_gateway_rest_api.api.id}"
resource_id = "${aws_api_gateway_rest_api.api.root_resource_id}"
http_method = "POST"
type = "AWS"
integration_http_method = "POST"
passthrough_behavior = "NEVER"
credentials = "${aws_iam_role.api.arn}"
uri = "arn:aws:apigateway:${var.region}:sqs:path/${aws_sqs_queue.queue.name}"
request_parameters = {
"integration.request.header.Content-Type" = "'application/x-www-form-urlencoded'"
}
request_templates = {
"application/json" = "Action=SendMessage&MessageBody=$input.body"
}
}We should define a basic 200 handler for successful requests with a custom response message. Layer on more responses as needed.
Notice the response_templates value below, which is what the service will return as a 200 status code and message. The selection_pattern is nothing more than a regex pattern to match any 2XX status codes that come back from SQS, which will then return a 200 and the json message {"message": "great success!"} to the client from the API Gateway request.
resource "aws_api_gateway_integration_response" "200" {
rest_api_id = "${aws_api_gateway_rest_api.api.id}"
resource_id = "${aws_api_gateway_rest_api.api.root_resource_id}"
http_method = "${aws_api_gateway_method.api.http_method}"
status_code = "${aws_api_gateway_method_response.200.status_code}"
selection_pattern = "^2[0-9][0-9]" // regex pattern for any 200 message that comes back from SQS
response_templates = {
"application/json" = "{\"message\": \"great success!\"}"
}
depends_on = ["aws_api_gateway_integration.api"]
}
resource "aws_api_gateway_method_response" "200" {
rest_api_id = "${aws_api_gateway_rest_api.api.id}"
resource_id = "${aws_api_gateway_rest_api.api.root_resource_id}"
http_method = "${aws_api_gateway_method.api.http_method}"
status_code = 200
response_models = {
"application/json" = "Empty"
}
}And lastly, create the API Gateway deployment.
resource "aws_api_gateway_deployment" "api" {
rest_api_id = "${aws_api_gateway_rest_api.api.id}"
stage_name = "main"
depends_on = [
"aws_api_gateway_integration.api",
]
}With everything defined we are good to run.
terraform init
terraform applyWhich will output something like below.
Apply complete! Resources: 12 added, 0 changed, 0 destroyed.
Outputs:
test_cURL = curl -X POST -H 'Content-Type: application/json' -d '{"id":"test", "docs":[{"key":"value"}]}' https://4h8qo0bdd1.execute-api.us-east-1.amazonaws.com/main/Run it yourself and give the outputted test cURL command a try to see that the record ended up in the SQS queue. You can view all the example terraform code together below.
You are so amazing, thank you for sharing this! Spent a few hours figuring out how to Terraform API Gateway → SQS (my use case is to deploy a webhook to listen for events from a SaaS), I realise the most important part is still the
uriinaws_api_gateway_integrationby looking at your example.My mistake was to include the ARN of the SQS directly in
aws_api_gateway_integrationwhen in fact, it should be in the formatarn:aws:apigateway:${var.region}:sqs:path/${aws_sqs_queue.queue.name}.