How-to websockets through ELB + Nginx reverse proxy

elb-nginx-websockets.md

Websockets-through-ELB-to-Nginx-to-another-ELB recap

We need to setup an ELB with SSL termination in front of a Nginx reverse proxy to some already working ELB + project system. IOW: SSL-ELB -> Nginx reverse proxy -> ELB -> project. We'll assume the last ELB + project bit works properly.

Front ELB configuration

• Configure the listeners to use TCP (80) instead of HTTP to allow the websockets (non-HTTP) to pass through. For HTTPS with SSL termination, simply use SSL (443) -> TCP (80)

• Configure the health checks to use TCP (80) instead of HTTP as well otherwise Nginx won't be happy about it (no PROXY headers), which will lead the ELB to think the instance is out of order since the health check fails. In such case the Nginx logs would typically look like this (note the ELB-HealthChecker/1.0 user-agent):

  2018/01/16 14:00:47 [error] 703#703: *41055 broken header: "GET / HTTP/1.1
  host: ip.add.ress.here
  User-Agent: ELB-HealthChecker/1.0
  Accept: */*
  Connection: keep-alive
  
  " while reading PROXY protocol, client: ip.add.ress.here, server: 0.0.0.0:80
  

• Enable ProxyProtocol on the ELB. So far this can only be accomplished with the AWS CLI tool. See https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-proxy-protocol.html for detailed instructions, but here's a quick cheatsheet:

  # List the ELB's properties, including policies
  aws elb describe-load-balancers --load-balancer-name my-loadbalancer
  
  # Create a ProxyProtocol policy for the ELB
  aws elb create-load-balancer-policy --load-balancer-name my-loadbalancer --policy-name my-ProxyProtocol-policy --policy-type-name ProxyProtocolPolicyType --policy-attributes AttributeName=ProxyProtocol,AttributeValue=true
  
  # Add the new ProxyProtocol policy to the ELB. Beware, this operation SETS the list of policies for the ELB so you need to include previous existing policies if any
  aws elb set-load-balancer-policies-for-backend-server --load-balancer-name my-loadbalancer --instance-port 80 --policy-names my-ProxyProtocol-policy my-existing-policy

Nginx reverse proxy configuration

• Tell Nginx to expect all connections on this port will use proxy protocol. If proxy protocol is enabled on the ELB but not in Nginx, you'll typically get 400 Bad Request errors.

  server {
     listen 80 proxy_protocol;
     ...
  }
  

• Configure Nginx to deal with websockets through ProxyProtocol:

  location /socket.io/ {
     proxy_pass http://target-host-for-websockets/socket.io/;
     proxy_http_version 1.1;
     proxy_set_header Host $host;
     proxy_set_header Upgrade $http_upgrade;
     proxy_set_header Connection "Upgrade";
     proxy_set_header X-Real-IP $remote_addr;
     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  }