Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
nginx TLS SNI routing, based on subdomain pattern

Nginx TLS SNI routing, based on subdomain pattern

Nginx can be configured to route to a backend, based on the server's domain name, which is included in the SSL/TLS handshake (Server Name Indication, SNI).
This works for http upstream servers, but also for other protocols, that can be secured with TLS.

prerequisites

  • at least nginx 1.15.9 to use variables in ssl_certificate and ssl_certificate_key.
  • check nginx -V for the following:
    ...
    TLS SNI support enabled
    ...
    --with-stream_ssl_module 
    --with-stream_ssl_preread_module

It works well with the nginx:1.15.9-alpine docker image.

non terminating, TLS pass through

Pass the TLS stream to an upstream server, based on the domain name from TLS SNI field. This does not terminate TLS.
The upstream server can serve HTTPS or other TLS secured TCP responses.

stream {  

  map $ssl_preread_server_name $targetBackend {
    ab.mydomain.com  upstream1.example.com:443;
    xy.mydomain.com  upstream2.example.com:443;
  }   
 
  server {
    listen 443; 
        
    proxy_connect_timeout 1s;
    proxy_timeout 3s;
    resolver 1.1.1.1;
    
    proxy_pass $targetBackend;       
    ssl_preread on;
  }
}

terminating TLS, forward TCP

Terminate TLS and forward the plain TCP to the upstream server.

stream {  

  map $ssl_server_name $targetBackend {
    ab.mydomain.com  upstream1.example.com:443;
    xy.mydomain.com  upstream2.example.com:443;
  }

  map $ssl_server_name $targetCert {
    ab.mydomain.com /certs/server-cert1.pem;
    xy.mydomain.com /certs/server-cert2.pem;
  }

  map $ssl_server_name $targetCertKey {
    ab.mydomain.com /certs/server-key1.pem;
    xy.mydomain.com /certs/server-key2.pem;
  }
  
  server {
    listen 443 ssl; 
    ssl_protocols       TLSv1.2;
    ssl_certificate     $targetCert;
    ssl_certificate_key $targetCertKey;
        
    proxy_connect_timeout 1s;
    proxy_timeout 3s;
    resolver 1.1.1.1;
      
    proxy_pass $targetBackend;
  } 
}

Choose upstream based on domain pattern

The domain name can be matched by a regex pattern, and extracted to variables. See regex_names.
This can be used to choose a backend/upstream based on the pattern of a (sub)domain. This is inspired by robszumski/k8s-service-proxy.

The following configuration extracts a subdomain into variables and uses them to create the upstream server name.

stream {  

  map $ssl_preread_server_name $targetBackend {
    ~^(?<app>.+)-(?<namespace>.+).mydomain.com$ $app-public.$namespace.example.com:8080;
  }
  ...
}

Your Nginx should be reachable over the wildcard subdomain *.mydomain.com.
A request to shop-staging.mydomain.com will be forwarded to shop-public.staging.example.com:8080.

K8s service exposing by pattern

In Kubernetes, you can use this to expose all services with a specific name pattern.
This configuration exposes all service which names end with -public.
A request to shop-staging-9999.mydomain.com will be forwarded to shop-public in the namespace staging on port 9999.
You will also need to update the resolver, see below.

stream {  

  map $ssl_preread_server_name $targetBackend {
    ~^(?<service>.+)-(?<namespace>.+)-(?<port>.+).mydomain.com$ $service-public.$namespace.svc.cluster.local:$port;
  }
  
  server {
    ...
    resolver kube-dns.kube-system.svc.cluster.local;
    ...
  }
}
@razorRun

This comment has been minimized.

Copy link

razorRun commented Aug 16, 2019

Do we need to have an SSL certificate on ab.mydomain.com in this case?

@kekru

This comment has been minimized.

Copy link
Owner Author

kekru commented Aug 16, 2019

Hi razorRun, yes you need a certificate on the nginx, when using "terminating TLS, forward TCP".

If you use "non terminating, TLS pass through", then you will need the certificate on the backend server, but not on the nginx.

In both cases the certificate must match ab.mydomain.com

@razorRun

This comment has been minimized.

Copy link

razorRun commented Aug 17, 2019

Kekru Thanks, mate for the reply

I have a small clarification if you don't mind
What I want to do is
vpn1.app.com ─┬─► nginx at 10.0.0.1 ─┬─► vpn1 at another-server-1
vpn2.app.com ─┤ ├─► vpn2 at another-server-2
wildcard.app.com ─┘ ─► Y

I have a wildcard(all the subdomains) pointed to a nginx server

The thing is I want to do a dynamic mapping and am not in control of another-server-X. If a client asks for a subdomain(not fixed) I will have to have a lookup table and map subdomain to the expected external server.

So do I have to add a SSL cert to my nginx server. will it give the SNI? I am getting a blank "-" at the moment.

Any help will be really handy.
Thanks in advance

Current Config

stream {
log_format basic '$remote_addr [$time_local] '
'$ssl_preread_server_name'
'$protocol $status $bytes_sent $bytes_received '
'$session_time';

access_log  /var/log/nginx/access.log basic;
error_log  /var/log/nginx/error.log debug;

#I will dynamically update the map section
map $ssl_preread_server_name $targetBackend {
sample.mydomain.com 32.23.232.32:3431;
xy.mydomain.com 44.23.342.32:3431;
}
server {
listen 80;
proxy_pass $targetBackend;
ssl_preread on;
proxy_connect_timeout 1s;
proxy_timeout 3s;
resolver 1.1.1.1;

}

}

@kekru

This comment has been minimized.

Copy link
Owner Author

kekru commented Aug 19, 2019

Hi razorRun,

  1. Does "vpn1 at another-server-1" have a certificate? If no, then you need a certificate at your nginx.
    Same for "vpn2 at another-server-2"
  2. The SNI comes from the client, during the TLS handshake between client and nginx.
  3. It is not guaranteed, that all clients send an SNI, but most HTTPS clients should do, including all browsers.
  4. Which protocol are you using, between client and nginx? HTTPS or another? If another, be sure, that it is build on top of TCP+TLS. Otherwise it won't work.
  5. Your current config listens on port 80. If your protocol is HTTPS, be sure to explicitly write it in your browser's address line, or your browser will send HTTP by default. Then you don't have an SNI. SNI is only for TLS secured connections.
@paravz

This comment has been minimized.

Copy link

paravz commented Jan 13, 2020

"terminating TLS, forward TCP" can be extended by adding a ssl_preread listener with stream ssl listeners as backends.

This way each stream tls listener can have unique ssl configuration, not just parametrized ssl certs. This makes it possible to enable mutual tls for some clients based on available ssl_preread variables http://nginx.org/en/docs/stream/ngx_stream_ssl_preread_module.html#variables, ie $ssl_preread_server_name

@totayma

This comment has been minimized.

Copy link

totayma commented Feb 2, 2020

how can use this for proxy google.com ? for example run dns server and make A record google.com >>> 1.2.3.4 and this server(1.2.3.4) run nginx proxy to origin google.com 172.217.23.142

@kekru

This comment has been minimized.

Copy link
Owner Author

kekru commented Feb 3, 2020

@totayma Should work, I think. Only problem will be, that you don't have a valid TLS cert for google.com

@rayray221

This comment has been minimized.

Copy link

rayray221 commented Apr 18, 2020

@kekru is it possible to have multiple configurations for terminating TLS on the same reverse proxy? I currently have a https upstream server defined and working, terminating at the proxy. I would like to add another server to the configuration, that would terminate on the backend server. I have tried adding a stream directive with a similar configuration as the above to my config, but nginx seems to ignore it and routes it to my default site anyway. Is this even possible? Any help is greatly appreciated!

@kekru

This comment has been minimized.

Copy link
Owner Author

kekru commented May 11, 2020

Hi @rayray221,
sorry for the late answer. Did you already find a solution?

Do I understand correctly? You have one nginx. And you want to configure at the same time:

  • non terminating, TLS pass through for hello.example.com
  • terminating TLS, forward TCP for world.example.com

I'm not a super nginx expert, but I think it will work, but not on the same port.
I think you need something like this:

stream {  
   
   ... # content see above, in the artice
   ...

   server { # non terminating, TLS pass through
    listen 443; 
        
    proxy_connect_timeout 1s;
    proxy_timeout 3s;
    resolver 1.1.1.1;
    
    proxy_pass $targetBackend; 
    ssl_preread on;
  }
  
  server { # terminating TLS, forward TCP
    # here another port
    listen 8443 ssl; 
    ssl_protocols       TLSv1.2;
    ssl_certificate     $targetCert;
    ssl_certificate_key $targetCertKey;
        
    proxy_connect_timeout 1s;
    proxy_timeout 3s;
    resolver 1.1.1.1;
      
    proxy_pass $targetBackend;
  } 
}

As you see, I choose two different ports (443 and 8443)
But, I'm not sure, I did not test this

@rayray221

This comment has been minimized.

Copy link

rayray221 commented May 11, 2020

Thanks @kekru. I ended up terminating both on the reverse proxy. To be honest, the reason I wanted to have multiple terminations was more out of laziness than need. I had an existing Nginx web server that was terminating already, and was adding an additional website under the same IP. The existing one has a bit of a complicated setup and I didn't want to spend the time troubleshooting it. All in all, I did get this working and it is all terminating at my reverse proxy which honestly is the desired configuration anyway.

Appreciate your response!

@LeonDragon

This comment has been minimized.

Copy link

LeonDragon commented Jun 24, 2020

I am a newbie in this part. But I have a naive question: This configuration is anything related to the DNS server? or just for the webserver (i.e., VPS) only. Let say, I am not configuring anything in the DNS server (such as subdomain) and my domain is www.example.com. The server is configured to "extract a subdomain into variables" as your code.

  • When a client request a content at abc.example.com. What is the DNS server behavior? the DNS server will pass through to the designate webserver with the metadata [abc.example.com] for the server, and the server will know which part of the website, the client wants to connect?

Thank you in advance!

@kekru

This comment has been minimized.

Copy link
Owner Author

kekru commented Jun 24, 2020

Hi @LeonDragon,
your DNS server must return the IP of your webserver, wenn abc.example.com is requested.

For example, if your webserver's IP is 11.22.33.44:

  • You can use a wildcard A record in your DNS: A *.example.com 11.22.33.44 (I prefer this)
  • Or a simple A record: A abc.example.com 11.22.33.44

Then your webserver gets the domain name abc.example.com in the HTTP Host header.
It can extract parts of the domain and knows which parts of the website it should show, or which other server it should delegate to

@LeonDragon

This comment has been minimized.

Copy link

LeonDragon commented Jun 25, 2020

Thank @kekru, I understand now :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.