Skip to content

Instantly share code, notes, and snippets.

@Likeyn
Last active September 9, 2019 23:26
Show Gist options
  • Save Likeyn/63da37b23424ad814c6732747a32aad8 to your computer and use it in GitHub Desktop.
Save Likeyn/63da37b23424ad814c6732747a32aad8 to your computer and use it in GitHub Desktop.
How-to websockets through ELB + Nginx reverse proxy

Websockets-through-ELB-to-Nginx-to-another-ELB recap

We need to setup an ELB with SSL termination in front of a Nginx reverse proxy to some already working ELB + project system. IOW: SSL-ELB -> Nginx reverse proxy -> ELB -> project. We'll assume the last ELB + project bit works properly.

Front ELB configuration

  • Configure the listeners to use TCP (80) instead of HTTP to allow the websockets (non-HTTP) to pass through. For HTTPS with SSL termination, simply use SSL (443) -> TCP (80)

  • Configure the health checks to use TCP (80) instead of HTTP as well otherwise Nginx won't be happy about it (no PROXY headers), which will lead the ELB to think the instance is out of order since the health check fails. In such case the Nginx logs would typically look like this (note the ELB-HealthChecker/1.0 user-agent):

    2018/01/16 14:00:47 [error] 703#703: *41055 broken header: "GET / HTTP/1.1
    host: ip.add.ress.here
    User-Agent: ELB-HealthChecker/1.0
    Accept: */*
    Connection: keep-alive
    
    " while reading PROXY protocol, client: ip.add.ress.here, server: 0.0.0.0:80
    
  • Enable ProxyProtocol on the ELB. So far this can only be accomplished with the AWS CLI tool. See https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-proxy-protocol.html for detailed instructions, but here's a quick cheatsheet:

    # List the ELB's properties, including policies
    aws elb describe-load-balancers --load-balancer-name my-loadbalancer
    
    # Create a ProxyProtocol policy for the ELB
    aws elb create-load-balancer-policy --load-balancer-name my-loadbalancer --policy-name my-ProxyProtocol-policy --policy-type-name ProxyProtocolPolicyType --policy-attributes AttributeName=ProxyProtocol,AttributeValue=true
    
    # Add the new ProxyProtocol policy to the ELB. Beware, this operation SETS the list of policies for the ELB so you need to include previous existing policies if any
    aws elb set-load-balancer-policies-for-backend-server --load-balancer-name my-loadbalancer --instance-port 80 --policy-names my-ProxyProtocol-policy my-existing-policy

Nginx reverse proxy configuration

  • Tell Nginx to expect all connections on this port will use proxy protocol. If proxy protocol is enabled on the ELB but not in Nginx, you'll typically get 400 Bad Request errors.
    server {
       listen 80 proxy_protocol;
       ...
    }
    
  • Configure Nginx to deal with websockets through ProxyProtocol:
    location /socket.io/ {
       proxy_pass http://target-host-for-websockets/socket.io/;
       proxy_http_version 1.1;
       proxy_set_header Host $host;
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "Upgrade";
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment