An Amazon Linux 2 Squid web proxy with a SASL-authenticated Postfix Implicit TLS for SMTP Submission relay to Amazon SES built with Packer and Terraform

README.md

Squid and Postfix SES Relay

A Squid transparent proxy server with Postfix configured as a SASL-authenticated SMTP relay to Amazon Simple Email Service (SES).

How do you use this module?

• See the root README for instructions on using Terraform modules.
• See variables.tf for all the variables you can set on this module.
• See outputs.tf for all outputs you can get from this module in a terraform_remote_state data source.

Warnings

Let's Encrypt limits 5 new certificates per week per FQDN, and 50 certificates per week per registered domain.

See Rate Limits - Let's Encrypt - Free SSL/TLS Certificates to learn more.

Provide squid_backup_timestamp when provisioning to avoid this.

Core concepts

Squid

Squid is a caching and forwarding HTTP web proxy. It has a wide variety of uses, including speeding up a web server by caching repeated requests, caching web, DNS and other computer network lookups for a group of people sharing network resources, and aiding security by filtering traffic.

https://en.wikipedia.org/wiki/Squid_(software)

Postfix

Postfix is a free and open-source mail transfer agent that routes and delivers electronic mail.

https://en.wikipedia.org/wiki/Postfix_(software)

Postfix is configured to provide SMTP via Implicit TLS (establish TLS connection before sending anything) on TCP/465. Why not explicit TLS (connect in plaintext, then upgrade using STARTTLS) on TCP/587? Many sites have published articles suggesting that TCP/465 is an incorrect port for SMTP unsupported by the IETF and should be discouraged. As of January 2018, they are incorrect as the IETF has issued RFC 8314 which clearly advises otherwise:

To encourage more widespread use of TLS and to also encourage greater consistency regarding how TLS is used, this specification now recommends the use of Implicit TLS for POP, IMAP, SMTP Submission, and all other protocols used between an MUA and an MSP.

https://tools.ietf.org/html/rfc8314#section-3

Building Squid and Postfix AMI

The ami variable requires a pre-build AMI with Squid and Postfix configured. Run Packer to build the AMI:

cd packer
# get actual from Github OAuth token from your password manager
export GITHUB_OAUTH_TOKEN='**********************' 
export ssh_grunt_iam_role_arn="arn:aws:iam::$(aws-vault exec security -- aws sts get-caller-identity | jq '.Account' --raw-output):role/allow-ssh-iam-access-from-other-accounts"
function create-squid-ami {
    environment="$1"
    export squid_bucket="squid-allowlist-sbi-$environment"
    export PACKER_VPC_ID="$(aws-vault exec "$environment" -- aws ec2 describe-vpcs --filters 'Name=tag:Name,Values=mgmt' --query 'Vpcs[].VpcId' --output text)"
    export PACKER_SUBNET_ID="$(aws-vault exec "$environment" -- aws ec2 describe-subnets --filters 'Name=tag:Name,Values=mgmt-public-0' --query 'Subnets[].SubnetId' --output text)"
    aws-vault exec "$environment" -- packer build squid-server.json
}

To build it for dev run the following:

create-squid-ami dev

Verify

To ensure everything is configured properly on the Squid server:

1. Run "iptables -t nat -L", you should see the proper nat statements.

   target     prot opt source               destination
   REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:http redir ports 3129
   REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:https redir ports 3130
   
   Chain INPUT (policy ACCEPT)
   target     prot opt source               destination
   
   Chain OUTPUT (policy ACCEPT)
   target     prot opt source               destination
   
   Chain POSTROUTING (policy ACCEPT)
   target     prot opt source               destination
   

2. Verify Allow List is updating.

   Go to cat /etc/squid/allowlist.txt. Should only have one entry, ".amazonaws.com". Create a text file called allowlist.txt with a new allowlist and upload to your squid bucket with the following entries.

   .amazonaws.com
   .example.com
   

   To do this, upload from the console. You MUST choose "AWS KMS master-key" for encryption and choose "cmk-dev". In one minute you should see the list automatically populate the /etc/squid/allowlist.txt file!

3. Test Squid Proxy:

   SSH to instance using keypair. Sudo su. Run "systemctl status squid" and ensure it is actively running.

   tail -f  /var/log/squid/access.log
   

   From another server that is using the squid server as the Subnet Default Gateway, use the following curl commands to verify:

   curl 'http://www.example.com/'
   curl 'https://www.example.com/'
   curl 'http://s3.amazonaws.com/'
   curl 'https://s3.amazonaws.com/'
   

Troubleshooting

/var/log/squid/access.log

These sections explain how to interpret /var/log/squid/access.log output.

There are four possible types of results. Below is a list of the 4 types and what each will look like in the squid log.

Successful HTTP

1580355117.679 72 10.1.2.30 TCP_MISS/301 315 GET http://s3.amazonaws.com/ - ORIGINAL_DST/52.217.37.6 text/html

Successful HTTPS

1580355347.533 519 10.1.2.30 TCP_TUNNEL/200 1533059 CONNECT s3.amazonaws.com - ORIGINAL_DST/52.217.37.6 - 1580355347.533 519 10.1.2.30 TCP_TUNNEL/200 1533059 CONNECT 52.217.37.6:443 - ORIGINAL_DST/52.217.37.6

Access Denied HTTP

1580355105.737 48 10.1.2.30 TCP_DENIED/403 3831 GET http://badssl.com/ - HIER_NONE/- text/html

Access Denied HTTPS

1580355433.795 86 10.1.2.30 TAG_NONE/200 0 CONNECT badssl.com - HIER_NONE/- - 1580355433.795 86 10.1.2.30 TAG_NONE/200 0 CONNECT 104.154.89.105:443 - HIER_NONE/- -

Notice that the Access Denied HTTPS is identifiable by "TAG_NONE" and "HIER_NONE"

Allowing All Traffic

This expains to allow all HTTP and HTTPS traffic through Squid if required.

Replace the following two lines in /etc/squid/squid.conf:

To allow all HTTP, replace:

acl allowed_http_sites dstdomain "/etc/squid/allowlist.txt"

with:

acl allowed_http_sites dstdom_regex .*

To allow all HTTPS, replace:

acl allowed_https_sites ssl::server_name "/etc/squid/allowlist.txt"

with:

acl allowed_https_sites ssl::server_name_regex .*

Then run "squid -k parse" and "squid -k reconfigure" to load the new file without having to restart squid server.

main.tf

# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# CREATE A WEB PROXY SERVICE USING SQUID TO FILTER EXTERNAL WEB ACCESS
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

terraform {
  required_version = ">= 0.12"
  required_providers {
    aws = ">= 2.35.0"
  }
}

# ------------------------------------------------------------------------------
# CREATE A SQUID SERVER
# ------------------------------------------------------------------------------

# The proxy instance with interface specified separately to make it easier to
# associate with a route table

resource "aws_network_interface" "squid_eni" {
  subnet_id = var.subnet_ids[0]

  # Important to disable this check to allow traffic not addressed to the
  # proxy to be received
  source_dest_check = false
  security_groups = [
    aws_security_group.squid.id,
  ]
}

resource "aws_eip" "squid" {
  instance = aws_instance.squid.id
  vpc      = true
}

resource "aws_instance" "squid" {
  ami                  = var.ami
  instance_type        = var.instance_type
  iam_instance_profile = aws_iam_instance_profile.squid.name
  key_name             = var.keypair_name
  user_data            = local.user_data

  network_interface {
    network_interface_id = aws_network_interface.squid_eni.id
    device_index         = 0
  }

  root_block_device {
    volume_size = var.volume_size
  }

  tags = {
    Name = var.server_name
  }
}

# ------------------------------------------------------------------------------
# ALLOW ACCESS TO S3
# ------------------------------------------------------------------------------

resource "aws_iam_instance_profile" "squid" {
  name = var.server_name
  role = aws_iam_role.squid_instance.name
}

resource "aws_iam_role" "squid_instance" {
  name               = var.server_name
  assume_role_policy = data.aws_iam_policy_document.allow_ec2_assume_role.json
}

data "aws_iam_policy_document" "allow_ec2_assume_role" {
  version = "2012-10-17"

  statement {
    sid    = "AllowEC2"
    effect = "Allow"

    actions = [
      "sts:AssumeRole",
    ]

    principals {
      type = "Service"
      identifiers = [
        "ec2.amazonaws.com",
      ]
    }
  }
}

resource "aws_iam_role_policy" "allow_letsencrypt" {
  name   = "allow-letsencrypt"
  role   = aws_iam_role.squid_instance.name
  policy = data.aws_iam_policy_document.allow_letsencrypt.json
}

data "aws_iam_policy_document" "allow_letsencrypt" {
  statement {
    sid    = "AllowLetsEncryptRoute53Write"
    effect = "Allow"
    actions = [
      "route53:ChangeResourceRecordSets",
    ]
    resources = [
      "arn:aws:route53:::hostedzone/${aws_route53_zone.postfix.zone_id}",
    ]
  }

  statement {
    sid    = "AllowLetsEncryptRoute53Read"
    effect = "Allow"
    actions = [
      "route53:ListHostedZones",
      "route53:GetChange",
    ]
    resources = [
      "*",
    ]
  }
}

resource "aws_iam_role_policy" "allow_read_smtp_secret" {
  name   = "allow-read-smtp-secret"
  role   = aws_iam_role.squid_instance.name
  policy = data.aws_iam_policy_document.allow_read_smtp_secret.json
}

data "aws_iam_policy_document" "allow_read_smtp_secret" {
  statement {
    sid    = "AllowGetSmtpSecret"
    effect = "Allow"
    actions = [
      "secretsmanager:GetSecretValue",
    ]
    resources = [
      var.smtp_secret_arn
    ]
  }
}

resource "aws_iam_role_policy" "allow_config_downloads" {
  name   = "allow-config-downloads"
  role   = aws_iam_role.squid_instance.name
  policy = data.aws_iam_policy_document.allow_config_downloads.json
}

data "aws_iam_policy_document" "allow_config_downloads" {
  statement {
    sid    = "AllowListConfigFiles"
    effect = "Allow"
    actions = [
      "s3:ListBucket",
    ]
    resources = [
      "arn:aws:s3:::${var.squid_bucket_name}",
    ]
  }

  statement {
    sid    = "AllowDownloadConfigFiles"
    effect = "Allow"
    actions = [
      "s3:GetObject",
    ]
    resources = [
      "arn:aws:s3:::${var.squid_bucket_name}/*",
    ]
  }

  statement {
    sid    = "AllowDecryptConfigFiles"
    effect = "Allow"
    actions = [
      "kms:Decrypt",
      "kms:ReEncrypt*",
      "kms:GenerateDataKey*",
      "kms:DescribeKey",
    ]
    resources = [
      var.squid_bucket_kms_key_arn,
    ]
  }
}

# ------------------------------------------------------------------------------
# CREATE SUBDOMAIN FOR LETS ENCRYPT VALIDATION
# ------------------------------------------------------------------------------

resource "aws_route53_zone" "postfix" {
  name    = var.domain_name
  comment = "Delegated subdomain for Postfix."
}

resource "aws_route53_record" "postfix_subdomain" {
  name    = replace(var.domain_name, var.parent_public_domain_name, "")
  type    = "NS"
  zone_id = var.parent_public_hosted_zone_id

  ttl     = "300"
  records = aws_route53_zone.postfix.name_servers
}

resource "aws_route53_record" "postfix_internal" {
  name    = replace(var.domain_name, var.parent_public_domain_name, "")
  type    = "A"
  zone_id = var.client_hosted_zone_id

  ttl     = "300"
  records = aws_network_interface.squid_eni.private_ips
}

# ------------------------------------------------------------------------------
# CONFIGURE SECURITY GROUPS FOR SQUID SERVER
# ------------------------------------------------------------------------------

locals {
  proxy_ports = {
    http  = 80
    https = 443
    smtps = 465
  }
}

resource "aws_security_group" "squid" {
  name   = var.server_name
  vpc_id = var.vpc_id
}

resource "aws_security_group_rule" "squid_from_openvpn_server" {
  description              = "Allow SSH from OpenVPN clients"
  type                     = "ingress"
  protocol                 = "tcp"
  from_port                = 22
  to_port                  = 22
  source_security_group_id = var.openvpn_security_group_id
  security_group_id        = aws_security_group.squid.id
}

resource "aws_security_group_rule" "inbound_proxy" {
  for_each          = local.proxy_ports
  description       = "Allow ${each.key} from proxy clients"
  type              = "ingress"
  protocol          = "tcp"
  from_port         = each.value
  to_port           = each.value
  security_group_id = aws_security_group.squid.id
  cidr_blocks       = var.client_cidr_blocks
}

# ------------------------------------------------------------------------------
# ALLOW ACCESS TO ENTIRE INTERNET
# ------------------------------------------------------------------------------

resource "aws_security_group_rule" "squid_to_entire_internet" {
  description       = "Allow all traffic to the Internet (Squid is the firewall)"
  type              = "egress"
  protocol          = "all"
  from_port         = 0
  to_port           = 0
  security_group_id = aws_security_group.squid.id
  cidr_blocks = [
    "0.0.0.0/0",
  ]
}

resource "aws_route" "private_subnet_default_gateway_squid" {
  for_each               = { for route_table_id in var.route_table_ids : route_table_id => route_table_id }
  route_table_id         = each.value
  destination_cidr_block = "0.0.0.0/0"
  network_interface_id   = aws_network_interface.squid_eni.id
}

# ------------------------------------------------------------------------------
# Get IAM-SSH working
# ------------------------------------------------------------------------------

module "iam_policies" {
  source = "git::git@github.com:gruntwork-io/module-security.git//modules/iam-policies?ref=v0.31.0"

  aws_account_id = var.aws_account_id

  # We can't use MFA with ssh-iam, since it's an automated app
  trust_policy_should_require_mfa = false
  iam_policy_should_require_mfa   = false

  # Since our IAM users are defined in a separate AWS account, we need to give ssh-iam permission to make API calls to
  # that account.
  allow_access_to_other_account_arns = {
    group1 = [
      var.external_account_ssh_iam_role_arn,
    ]
  }
}

resource "aws_iam_role_policy" "ssh_iam_permissions" {
  name   = "ssh-iam-permissions"
  role   = aws_iam_role.squid_instance.name
  policy = module.iam_policies.allow_access_to_other_accounts["group1"]
}

# ------------------------------------------------------------------------------
# Build user-data
# ------------------------------------------------------------------------------

locals {
  user_data = templatefile("${path.module}/user-data/user-data.sh", {
    aws_region                        = var.aws_region
    domain_name                       = var.domain_name
    external_account_ssh_iam_role_arn = var.external_account_ssh_iam_role_arn
    letsencrypt_bucket                = var.letsencrypt_bucket
    letsencrypt_subscriber_email      = var.letsencrypt_subscriber_email
    ses_smtp_address                  = var.ses_smtp_address
    postfix_mydomain                  = var.postfix_mydomain
    postfix_relay_domains             = var.postfix_relay_domains
    smtp_secret_name                  = var.smtp_secret_name
  })
}

outputs.tf

output "instance_id" {
  value = aws_instance.squid.id
}

output "eni_id" {
  value = aws_network_interface.squid_eni.id
}

output "iam_role_name" {
  value = aws_iam_role.squid_instance.name
}

output "iam_instance_profile_name" {
  value = aws_iam_instance_profile.squid.name
}
output "squid_sg" {
  value = aws_security_group.squid.id
}

squid-server.json

{
  "variables": {
    "aws_region": "ca-central-1",
    "vpc_id": "{{env `PACKER_VPC_ID`}}",
    "subnet_id": "{{env `PACKER_SUBNET_ID`}}",
    "ssh_grunt_iam_role_arn": "{{env `ssh_grunt_iam_role_arn`}}",
    "github_auth_token": "{{env `GITHUB_OAUTH_TOKEN`}}",
    "squid_bucket": "{{env `squid_bucket`}}"
  },
  "builders": [
    {
      "ami_name": "squid-server-{{isotime | clean_resource_name}}",
      "ami_description": "Squid + Postfix server built on Amazon Linux 2 {{isotime}}.",
      "instance_type": "t3.small",
      "region": "{{user `aws_region`}}",
      "vpc_id": "{{user `vpc_id`}}",
      "subnet_id": "{{user `subnet_id`}}",
      "associate_public_ip_address": "true",
      "type": "amazon-ebs",
      "source_ami_filter": {
        "filters": {
          "virtualization-type": "hvm",
          "architecture": "x86_64",
          "name": "amzn2-ami-hvm-*-x86_64-ebs",
          "root-device-type": "ebs"
        },
        "owners": [
          "amazon"
        ],
        "most_recent": true
      },
      "ssh_username": "ec2-user",
      "encrypt_boot": true
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "inline": [
        "echo 'OS packages... updating'",
        "sudo yum update --assumeyes",
        "echo 'OS packages... updated'"
      ]
    },
    {
      "type": "shell",
      "inline": [
        "echo 'Gruntwork installer... installing'",
        "curl -Ls https://raw.githubusercontent.com/gruntwork-io/gruntwork-installer/master/bootstrap-gruntwork-installer.sh | bash /dev/stdin --version v0.0.28",
        "echo 'Gruntwork installer... installed'"
      ],
      "environment_vars": [
        "GITHUB_OAUTH_TOKEN={{user `github_auth_token`}}"
      ]
    },
    {
      "type": "shell",
      "inline": [
        "gruntwork-install --module-name 'bash-commons'                                   --repo 'https://github.com/gruntwork-io/bash-commons'          --tag  'v0.1.2'",
        "gruntwork-install --module-name 'logs/cloudwatch-log-aggregation-scripts'        --repo 'https://github.com/gruntwork-io/module-aws-monitoring' --tag 'v0.21.1' --module-param aws-region={{user `aws_region`}}",
        "gruntwork-install --module-name 'metrics/cloudwatch-memory-disk-metrics-scripts' --repo 'https://github.com/gruntwork-io/module-aws-monitoring' --tag 'v0.21.1'",
        "gruntwork-install --module-name 'logs/syslog'                                    --repo 'https://github.com/gruntwork-io/module-aws-monitoring' --tag 'v0.21.1'",
        "gruntwork-install --module-name 'auto-update'                                    --repo 'https://github.com/gruntwork-io/module-security'       --tag 'v0.30.0'",
        "gruntwork-install --binary-name 'ssh-grunt'                                      --repo 'https://github.com/gruntwork-io/module-security'       --tag 'v0.28.7'",
        "gruntwork-install --binary-name 'gruntkms'                                       --repo 'https://github.com/gruntwork-io/gruntkms'              --tag  'v0.0.8'",
        "sudo /usr/local/bin/ssh-grunt iam install --iam-group ssh-iam-users --iam-group-sudo ssh-iam-sudo-users --role-arn '{{user `ssh_grunt_iam_role_arn`}}'"
      ],
      "environment_vars": [
        "GITHUB_OAUTH_TOKEN={{user `github_auth_token`}}"
      ]
    },
    {
      "type": "shell",
      "script": "./squid-server.sh",
      "environment_vars": [
        "squid_bucket={{user `squid_bucket`}}"
      ]
    }
  ]
}

squid-server.sh

#!/bin/bash
set -Eeuxo pipefail

# Required environment Variables
declare -r squid_bucket="${squid_bucket}"

echo ------------------------------------------------------------------------
echo Installing and configuring the necessary dependencies
echo ------------------------------------------------------------------------
sudo yum update -y
# - policycoreutils-python is needed for certbot
# - jq is needed to parse JSON-formatted Secrets Manager secrets, to avoid
#   death by 1,000 $0.40/month paper cuts (that's the fee per secret)
sudo yum install -y policycoreutils-python jq

# Install certbot on Amazon Linux 2.
# Modified to do Route53 challenges since the server is on a private network.
# SOURCE: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/SSL-on-amazon-linux-2.html#letsencrypt
wget -r --no-parent -A 'epel-release-*.rpm' http://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/
sudo rpm -Uvh dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-*.rpm
sudo yum-config-manager --enable epel*
sudo yum install -y certbot python2-certbot-dns-route53 python2-pip

# BUG: certbot will fail due to conflicting boto3 packages, so fix it
# SOURCE: https://unix.stackexchange.com/a/456362
sudo pip uninstall --yes botocore boto3 || true
sudo pip install boto3

echo ------------------------------------------------------------------------
echo Installing Squid and Dovecot
echo ------------------------------------------------------------------------
sudo yum install --assumeyes squid dovecot

# Create original allowlist for Squid
# Note: put a proper allowlist in your squid-config S3 bucket (this is bad)
cat | sudo tee /etc/squid/allowlist.txt >/dev/null <<EOF
.*
EOF

# Create certificates for SSL peek
sudo mkdir /etc/squid/ssl && cd /etc/squid/ssl
sudo openssl genrsa -out squid.key 2048
sudo openssl req -new -key squid.key -out squid.csr -subj "/C=XX/ST=XX/L=squid/O=squid/CN=squid"
sudo openssl x509 -req -days 3650 -in squid.csr -signkey squid.key -out squid.crt
sudo cat squid.key squid.crt | sudo tee squid.pem >/dev/null

# Create squid.conf file
cat | sudo tee /etc/squid/squid.conf >/dev/null <<EOF
visible_hostname squid

acl https_logs port 443
logformat https %ts.%03tu %6tr %>a %Ss/%03>Hs %<st %rm %ssl::>sni %[un %Sh/%<a %mt
access_log stdio:/var/log/squid/access.log  logformat=https https_logs
access_log daemon:/var/log/squid/access.log all

http_port 3128

#Handling HTTP requests
http_port 3129 intercept
acl allowed_http_sites dstdom_regex "/etc/squid/allowlist.txt"
http_access allow allowed_http_sites

#Handling HTTPS requests
https_port 3130 cert=/etc/squid/ssl/squid.pem ssl-bump intercept

acl SSL_port port 443
acl allowed_https_sites ssl::server_name_regex "/etc/squid/allowlist.txt"

acl step1 at_step SslBump1
acl step2 at_step SslBump2
acl step3 at_step SslBump3

# at step 1 we're peeking at client TLS-request in order to find the SNI
ssl_bump peek step1 all
# here we're peeking at server certificate
ssl_bump peek step2 allowed_https_sites
ssl_bump terminate step2 !allowed_https_sites
# here we're splicing connections which match the allowlist
ssl_bump splice step3 allowed_https_sites
# finally we're terminating all other SSL connections
ssl_bump terminate

http_access allow SSL_port
http_access deny all
EOF

# Allow Windows Updates
# Microsoft uses cert trusted only by Windows for Microsoft reasons
tmp_cert=$(mktemp)
curl -L 'http://go.microsoft.com/fwlink/?linkid=747875&clcid=0x409' > "$tmp_cert"
sudo mv "$tmp_cert" '/etc/pki/ca-trust/source/anchors/Microsoft_Root_Certificate_Authority_2011.cer'
sudo update-ca-trust enable
sudo update-ca-trust extract

# Configure NAT port forwarding from HTTP and HTTPS to Squid ports
sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 3129
sudo iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-port 3130
sudo iptables-save | sudo tee /etc/sysconfig/iptables >/dev/null

# Set up allowlist updater
sudo mkdir /etc/squid/old/
cat | sudo tee /etc/squid/squid-conf-refresh >/dev/null <<EOF
cp /etc/squid/* /etc/squid/old/
aws s3 sync s3://"${squid_bucket}" /etc/squid
/usr/sbin/squid -k parse && /usr/sbin/squid -k reconfigure || (cp /etc/squid/old/* /etc/squid/; exit 1)
EOF
sudo chmod +x /etc/squid/squid-conf-refresh

# On boot: load NAT port forwarding
# Every minute: Refresh configuration
# Daily: rotate squid log files
# Use workaround for appending to cron: https://stackoverflow.com/a/13355743
set +o pipefail
(
  sudo crontab -l || true
  echo -n "
@reboot /etc/squid/nat > /var/log/squid/nat.log 2>&1
* * * * * /etc/squid/squid-conf-refresh
0 0 * * * /usr/sbin/squid -k rotate
"
) | sort | uniq | sudo crontab -
set -o pipefail

cat | sudo tee /etc/dovecot/conf.d/10-auth.conf >/dev/null <<EOF
auth_mechanisms = plain login
!include auth-system.conf.ext
EOF

cat | sudo tee /etc/dovecot/conf.d/10-master.conf >/dev/null <<EOF
service auth {
  unix_listener /var/spool/postfix/private/auth  {
    mode = 0666
    user = postfix
    group = postfix
  }
}
EOF

# Create dummy credentials for SMTP relay client
# user-data must overwrite /etc/postfix/sasl_passwd with real content
# and run `sudo postmap hash:/etc/postfix/sasl_passwd`
cat | sudo tee /etc/postfix/sasl_passwd >/dev/null <<EOF
[email-smtp.eu-west-1.amazonaws.com]:587 SMTPUSERNAME:SMTPPASSWORD
EOF
sudo postmap hash:/etc/postfix/sasl_passwd

# Configure SMTP server for internal clients
# user-data will need to set ses-relay-user's password
# ex: `echo 'dont-use-this-password' | sudo passwd "ses-relay-user" --stdin`
# user-data will need to set smtpd_tls_cert_file and smtpd_tls_key_file
# ```
# sudo postconf 'smtpd_tls_cert_file=/etc/letsencrypt/live/yourhostname.com/fullchain.pem
# sudo postconf 'smtpd_tls_key_file=/etc/letsencrypt/live/youthostname.com/privkey.pem
# ```
sudo postconf \
  'inet_interfaces = all' \
  'smtpd_sender_login_maps = hash:/etc/postfix/sasl_passwd' \
  'smtpd_recipient_restrictions = permit_sasl_authenticated, reject' \
  'smtpd_relay_restrictions = permit_sasl_authenticated, reject' \
  'smtpd_sasl_type = dovecot' \
  'smtpd_sasl_path = private/auth' \
  'smtpd_sasl_auth_enable = yes' \
  'smtpd_sasl_security_options = noanonymous' \
  'smtpd_sasl_local_domain= $myhostname' \
  'smtpd_sasl_authenticated_header = yes' \
  'smtpd_tls_wrappermode=yes' \
  'smtpd_tls_security_level = encrypt' \
  'smtpd_tls_loglevel = 1' \
  'smtpd_tls_received_header=yes' \
  'broken_sasl_auth_clients=yes'

# Configure SMTP client for relaying
# user-data must supply **relayhost** parameter
# ex: `sudo postconf "relayhost = [email-smtp.us-west-2.amazonaws.com]:587"`
sudo postconf \
  "smtp_sasl_auth_enable = yes" \
  "smtp_sasl_security_options = noanonymous" \
  "smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd" \
  "smtp_use_tls = yes" \
  "smtp_tls_security_level = secure" \
  "smtp_tls_note_starttls_offer = yes" \
  'smtp_tls_CAfile = /etc/ssl/certs/ca-bundle.crt'

# Disable plain SMTP and enable Implicit TLS for SMTP Submisssion
# https://tools.ietf.org/html/rfc8314#section-3.3
sudo sed -in-place=.bak --regexp-extended \
  's/^smtp /\#smtp / ; s/^\#smtps /smtps /' /etc/postfix/master.cf

sudo systemctl enable squid.service
sudo systemctl enable dovecot

user-data.sh

#!/bin/bash
set -Eeuxo pipefail

# Template Variables
declare -r aws_region="${aws_region}"
declare -r domain_name="${domain_name}"
declare -r external_account_ssh_iam_role_arn="${external_account_ssh_iam_role_arn}"
declare -r letsencrypt_bucket="${letsencrypt_bucket}"
declare -r letsencrypt_subscriber_email="${letsencrypt_subscriber_email}"
declare -r ses_smtp_address="${ses_smtp_address}"
declare -r postfix_mydomain="${postfix_mydomain}"
declare -r postfix_relay_domains="${postfix_relay_domains}"
declare -r smtp_secret_name="${smtp_secret_name}"

function main() {
  echo "$(date) - user-data - Started"

  set_hostname
  sync_ssh_iam_users
  update_squid_allowlist
  install_updates_since_ami
  get_signed_tls_certs
  configure_postfix_ses_relay

  echo "$(date) - user-data - Finished"
}

function set_hostname() {
  hostnamectl set-hostname "$domain_name"
}

function update_squid_allowlist() {
  /etc/squid/squid-conf-refresh
}

function install_updates_since_ami() {
  yum update --assumeyes
}

function get_signed_tls_certs() {
  # Accept agreement, provide cert email
  certbot certonly --non-interactive -m "$letsencrypt_subscriber_email" \
    --dns-route53 -d "$domain_name" --agree-tos
}

function configure_postfix_ses_relay() {
  # Load secret from AWS Secrets Manager
  local -r smtp_secret_json="$(
    aws secretsmanager get-secret-value --region "$aws_region" \
        --secret-id "$smtp_secret_name" --query 'SecretString' --output text
  )"
  # {
  #   "ses_smtp_username": "...",
  #   "ses_smtp_password": "...",
  #   "smtp_relay_username": "...",
  #   "smtp_relay_password": "..."
  # }
  local -r ses_smtp_username="$(
    jq --raw-output '.ses_smtp_username' <<< "$smtp_secret_json"
  )"
  local -r ses_smtp_password="$(
    jq --raw-output '.ses_smtp_password' <<< "$smtp_secret_json"
  )"
  local -r smtp_relay_username="$(
    jq --raw-output '.smtp_relay_username' <<< "$smtp_secret_json"
  )"
  local -r smtp_relay_password="$(
    jq --raw-output '.smtp_relay_password' <<< "$smtp_secret_json"
  )"

  # Set up credentials for relaying to SES
  echo "[$ses_smtp_address]:587 $ses_smtp_username:$ses_smtp_password" > /etc/postfix/sasl_passwd
  postmap hash:/etc/postfix/sasl_passwd

  # Set up user for internal SMTP clients
  useradd "$smtp_relay_username"
  echo "$smtp_relay_password" | passwd "$smtp_relay_username" --stdin

  # Use this domain name as the hostname
  postconf "myhostname = $domain_name"

  # Relay all email through SES
  # NOTE: even though Implicit TLS on TCP/465 is preferred we use
  #       TCP/587 because the Postfix 2.x SMTP client does not support
  #       Implicit TLS for SMTP Submission
  # Once we have Postfix 3.0+ the SMTP client can use Implicit TLS
  # `smtp_tls_wrappermode = yes`
  postconf "relayhost = [$ses_smtp_address]:587"

  # Only MAIL FROM (sender) addresses with this domain
  postconf "mydomain = $postfix_mydomain"

  # Only RCPT-TO (recipient) addresses with this domain
  postconf "relay_domains = $postfix_relay_domains"

  # Set up TLS for SMTP server
  postconf "smtpd_tls_cert_file=/etc/letsencrypt/live/$domain_name/fullchain.pem"
  postconf "smtpd_tls_key_file=/etc/letsencrypt/live/$domain_name/privkey.pem"

  systemctl restart postfix
  systemctl restart dovecot
}

function sync_ssh_iam_users() {
  /usr/local/bin/ssh-grunt iam sync-users \
      --iam-group ssh-iam-users \
      --iam-group-sudo ssh-iam-sudo-users \
      --role-arn "${external_account_ssh_iam_role_arn}"
}

main

variables.tf

# ------------------------------------------------------------------------------
# REQUIRED PARAMETERS
# These must be passed in.
# ------------------------------------------------------------------------------

variable "squid_bucket_name" {
  description = "Name of the S3 bucket where the Squid domain allow list is stored."
  type        = string
}

variable "squid_bucket_kms_key_arn" {
  description = "ARN of the KMS Key ID that encrypts objects in the S3 bucket where the Squid domain allow list is stored."
  type        = string
}

variable "vpc_id" {
  description = "ID of the VPC in which the Squid server will be deployed."
  type        = string
}

variable "subnet_ids" {
  description = "IDs of the subnets in which the Squid server can be deployed."
  type        = list(string)
}

variable "client_cidr_blocks" {
  description = "CIDRs of the proxy clients. (ex: [\"10.0.0.0/8\"])"
  type        = list(string)
}

variable "openvpn_security_group_id" {
  description = "Security group ID for the OpenVPN server."
  type        = string
}

variable "route_table_ids" {
  description = "IDs of route tables for which Squid will become the default route."
  type        = list(string)
}

variable "external_account_ssh_iam_role_arn" {
  description = "Since our IAM users are defined in a separate AWS account, this variable is used to specify the ARN of an IAM role that allows ssh-iam to retrieve IAM group and public SSH key info from that account."
  type        = string
}

variable "ami" {
  description = "The AMI to run on the Squid Server. This should be built from the Packer template under packer/squid-server.json."
  type        = string
}

variable "keypair_name" {
  description = "The name of the Key Pair that can be used to SSH to the Data Intake server."
  type        = string
}

variable "server_name" {
  description = "The name of the new Squid server."
  type        = string
}

variable "domain_name" {
  description = "The domain name of the new Postfix server."
  type        = string
}

variable "parent_public_domain_name" {
  description = "The domain name of the parent public DNS zone."
  type        = string
}

variable "parent_public_hosted_zone_id" {
  description = "The hosted zone ID of the parent public DNS zone."
  type        = string
}

variable "client_hosted_zone_id" {
  description = "The hosted zone ID of the private DNS zone internal clients will use (should be the same zone as parent_public_hosted_zone_id or should be a private hosted zone with the same domain name)."
  type        = string
}

variable "letsencrypt_bucket" {
  description = "Name of the bucket that stores Let's Encrypt."
  type        = string
}

variable "letsencrypt_subscriber_email" {
  description = "Email address for the Let's Encrypt subscription and notifications."
  type        = string
}

variable "ses_smtp_address" {
  description = "Hostname of the SES SMTP endpoint. (ex: email-smtp.eu-west-1.amazonaws.com)"
  type        = string
}

variable "postfix_mydomain" {
  description = "Allowed domain name for MAIL FROM sender addresses (ex: example.com)"
  type        = string
}

variable "postfix_relay_domains" {
  description = "Allowed domain name for MAIL FROM sender addresses (ex: example.com)"
  type        = string
}

variable "smtp_secret_name" {
  description = "Name of the AWS Secrets Manager secret with the SMTP details (ex: ses/dev-smtp)"
  # This AWS Secrets Manager secret referred to here must take this form:
  # {
  #   "ses_smtp_username": "...",
  #   "ses_smtp_password": "...",
  #   "smtp_relay_username": "...",
  #   "smtp_relay_password": "..."
  # }
  type = string
}

variable "smtp_secret_arn" {
  description = "ARN of the AWS Secrets Manager secret with the SMTP details (ex: arn:aws:secretsmanager:ca-central-1:446693630893:secret:ses/dev-smtp-PFXzIa)"
  # This AWS Secrets Manager secret referred to here must take this form:
  # {
  #   "ses_smtp_username": "...",
  #   "ses_smtp_password": "...",
  #   "smtp_relay_username": "...",
  #   "smtp_relay_password": "..."
  # }
  type = string
}

# ------------------------------------------------------------------------------
# OPTIONAL PARAMETERS
# ------------------------------------------------------------------------------

variable "volume_size" {
  description = "The size of the root volume."
  type        = number
  default     = 300
}

variable "instance_type" {
  description = "The instance type used."
  type        = string
  default     = "t2.micro"
}

# ------------------------------------------------------------------------------
# STANDARD MODULE PARAMETERS
# ------------------------------------------------------------------------------

variable "aws_region" {
  description = "The AWS region in which all resources will be created."
  type        = string
}

variable "aws_account_id" {
  description = "The ID of the AWS Account in which to create resources."
  type        = string
}