Whitelists in haproxy (the right way)

haproxy_whitelists.md

tldr; Don't just test a whitelist based on an initial pass/fail. An update to that whitelist or addition of a parameter to a use_backend statement alone can cause a routing mess.

I don't normally say things like "the right way" but in this case attention to detail is usually always the right way. We had two use_backend statements in haproxy shown below where when an IP address wasn’t in the whitelist it would be routed straight to production. The proposed fix for this meant that traffic in the whitelist would always be routed to production. Which is the opposite of what I believe was intended in both cases.

  use_backend b1 if host-site worldpay_callback worldpay_whitelist worldpay_env_dev worldpay_auth
  use_backend b2 if host-site worldpay_callback worldpay_whitelist worldpay_env_prd worldpay_auth

This works, you can put whitelist evaluation in a use_backend statement but if it's nested inside a larger scope and the logic falls through it's going to bite you. Troubleshooting this particular issue is easy, we can make each request unique based on the auth token to distinguish our test traffic, we could add a header or something but why bother. You could use this trick to troubleshoot anything you want to be honest with urlp(). We’re not trying to make it work, just trying to make it route. If we recreate the same use_backend statements but this time with haproxy's interpolation string syntax {} we can hardcode the auth token. The n.n.n.n/32 here is just the boxes ip. Note all this testing is done on the secondary standby node in the pair but you could run both the original (above) and test (below) statements in parallel on the primary node and observe live traffic in the log.

  use_backend b1 if { src 127.0.0.1/32 n.n.n.n/32 } { hdr(host) -i site.com } { path_beg -i /api/wp_callback } { urlp(env) dev } {  urlp(auth) 1234 }
  use_backend b2 if { src 127.0.0.1/32 n.n.n.n/32  } { hdr(host) -i site.com } { path_beg -i /api/wp_callback } { urlp(env) prd } { urlp(auth) 1234 }

Now anything coming from the local box with an env param in the url should end up at the correct environment respectively. However for dev if I remove the boxes ip n.n.n.n/32 and use that as my src what happens? Well it routes to prd, it isn’t denied and goes to the wrong backend. We'd expect a NOSRV 403 forbidden if a deny rule was hit.

  use_backend b1 if { src 127.0.0.1/32  } { hdr(host) -i site.com } { path_beg -i /api/wp_callback } { urlp(env) dev } {  urlp(auth) 1234 }
  use_backend b2 if { src 127.0.0.1/32 n.n.n.n/32  } { hdr(host) -i site.com } { path_beg -i /api/wp_callback } { urlp(env) prd } { urlp(auth) 1234 }

What if I comment out the prd rule? Well it still routes to prd so it's not that rule that's doing it. This is a simple technique to show that another rule is catching and routing the traffic.

  use_backend b1 if { src 127.0.0.1/32  } { hdr(host) -i site.com } { path_beg -i /api/wp_callback } { urlp(env) dev } {  urlp(auth) 1234 }
  #use_backend b2 if { src 127.0.0.1/32 n.n.n.n/32  } { hdr(host) -i site.com } { path_beg -i /api/wp_callback } { urlp(env) prd } { urlp(auth) 1234 }

Long story short having a whitelist in a use_backend statement doesn’t imply deny when the rule isn’t matched. The use_backend statement creates a scope in which the request will be evaluated if it meets the criterion however if for example the ip isn’t in the whitelist it will fall back to the proxy configs global scope.

acl acl-api path_bag -i ^/api.* 

AND

default_backend b2 

Both of these “global scope” rules would be matched against the request when the whitelist evaluation was a miss.

Get your whitelists dynamically, set any test ips statically and for the love of god add a deny statements but test them properly.

# haproxy.cfg
  # dynamic list generated when ansible provisions the config 
  acl worldpay_whitelist src -f /etc/haproxy/acls/whitelists/worldpay/worldpay_prefix.lst
  # static list required for testing
  acl worldpay_reservation_payment_callback_whitelist src 127.0.0.1/32 n.n.n.n/32
  # will deny if caller is not in permitted whitelist
  http-request deny if worldpay_callback !worldpay_whitelist
  # will deny if caller is in permitted whitelist but the auth token is wrong
  http-request deny if worldpay_callback !worldpay_auth
  use_backend b1 if host-site worldpay_callback worldpay_env_dev
  use_backend b2 if host-site worldpay_callback worldpay_env_prd

Now evaluation in the use_backend is done based on the url and the env. The reason is because if you only setup the whitelist and you do that "the right way" if the requests is valid and doesn't match deny, you'll end up with the same routing issue where the new, valid request is again only being routed to production.

This can be rolled out with ansible. Whois can be slow to update but the list is only dynamic at provision time.

---
- name: Ensure worldplay/worldpay_prefix.lst is created
  ansible.builtin.file:
    path: /etc/haproxy/acls/whitelists/worldpay/worldpay_prefix.lst
    state: touch
    owner: haproxy
    group: haproxy
    mode: 0644

- name: Fetch worldpay prefix list
  ansible.builtin.shell: /usr/bin/whois -h whois.radb.net -- '-i origin AS15768' | grep ^route  | grep -v route6 | awk '{print$2}'
  register: worldpay_prefix_list
  check_mode: no
  become: no
  delegate_to: 127.0.0.1

- name: Populate worldpay/worldpay_prefix.lst with latest worldpay prefix list
  ansible.builtin.blockinfile:
    path: /etc/haproxy/acls/whitelists/worldpay/worldpay_prefix.lst
    state: present
    block: "{{ worldpay_prefix_list.stdout }}"