Skip to content

Instantly share code, notes, and snippets.

@csabapalfi

csabapalfi/proxies.md

Forked from aslushnikov/proxies.md
Created June 10, 2020 15:55
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save csabapalfi/19fb9f2a6df97ca3e8b67ad3159573ea to your computer and use it in GitHub Desktop.
Save csabapalfi/19fb9f2a6df97ca3e8b67ad3159573ea to your computer and use it in GitHub Desktop.
Download ZIP
Raw
proxies.md

As we were preparing Playwright 1.1.0, I learned quite a lot about Web Proxies. Below is the summary of my findings.

TIL: WEB Proxies

Reverse Web Proxy vs Forwarding Web Proxy

There are two groups of proxies that drive internet:

  • reverse proxy: many clients, few destination websites. NGINX is a popular reverse-proxy.
  • forwarding proxy: few clients, many destination websites. Squid is a popular forwarding proxy.

Forwarding Web Proxies

Forwarding Web proxies could be divided into:

  • SOCKS Proxy: low-level protocol, wraps connection and forwards packets
  • HTTP Proxy: parses HTTP requests, parses the request URI and forwards there. Clients connect to HTTP proxy over HTTP and communicate in plain text.
  • Secure Web Proxy: HTTP proxy to which clients connect over HTTPS and thus communication is encrypted.

SOCKS and HTTP proxies are very popular around the web and are widely supported.

Secure Web Proxies are somewhat less popular:

  • Squid supports secure web proxies, but requires a custom-compiled binary. I failed to make it work.
  • Chromium supports Secure Web Proxies with the --proxy-server=https://<proxy>:<port> flag
  • Firefox seems to be capable of using Secure Web Proxies. However, I didn't find UI to configure one.
  • MacOS supports configuring global Secure Web Proxy in its network settings.

Proxy Authentication

There are some proxies that serve anonymous traffic, but most proxies require user authentication.

  • SOCKS Proxy: browsers don't support username/password authentication for SOCKS proxies, so these are usually protected by IP allowlist.
  • HTTP Proxy: these support username/password authentication using HTTP header. Note: since HTTP proxy talks in plain text, the credentials might be intercepted.
  • Secure Web Proxy: these support username/password authentication using HTTP header. All communication including credentials is encrypted.

NOTE: I highly recommend to protect all your proxies. My anonymous full-access HTTP proxy was scraped in less then a day and used to serve some foreign traffic.

Browsers and Proxies

First of all: when we talk about browser proxies, we talk about forwarding proxies.

Browsers often let us specify multiple different proxies, e.g. one proxy for its HTTP traffic, and then another proxy for its HTTPS traffic, and so on. These usually are named as "HTTP proxy" and "HTTPS proxy".

Here's a Firefox 76 UI for proxy configuration:

image

IMPORTANT: "HTTPS Proxy" has nothing to do with "Secure Web Proxy". The confusion usually comes from the fact that server address for Secure Web Proxy actually starts with 'https://', and I for one fall short of misunderstanding what it is.

No matter what proxy you set for the browser - HTTP, Socks, or Secure Web Proxy, browsers can use it to fetch all types of web content, including HTTPS websites. Here's how:

  • SOCKS Proxy: nothing interesting here, SSL happens above the proxying layer.
  • HTTP Proxy: browsers use HTTP tunneling to support HTTPS-over-HTTP.
  • HTTPS Proxy: browser does a TLS handshake with the proxy, and then uses HTTP tunnelling to do HTTPS-over-HTTP.

Setting Up Forwarding Proxies on Ubuntu

The goal here is to have a simple, fowarding proxy for personal use that's protected by username/password authentication.

NOTE: I initially had a non-protected proxy, but it leaked and was used by someone in less then a day.

Pre-requisites:

  • ubuntu 18.04 machine with public IP
  • a domain name that points to the VM
  • a pair of <cert, key> files generated for this domain. The easiest would be to run certbot to get them.

1. Setting Up HTTP Proxy

  1. Install squid
  2. Move /etc/squid/squid.conf to /etc/squid/squid.conf.backup. Squid config is notoriously huge, so we'll start from scratch.
  3. Put the following into /etc/squid/squid.conf:
acl SSL_ports port 443
acl Safe_ports port 80		# http
acl Safe_ports port 21		# ftp
acl Safe_ports port 443		# https
acl Safe_ports port 70		# gopher
acl Safe_ports port 210		# wais
acl Safe_ports port 1025-65535	# unregistered ports
acl Safe_ports port 280		# http-mgmt
acl Safe_ports port 488		# gss-http
acl Safe_ports port 591		# filemaker
acl Safe_ports port 777		# multiling http
acl CONNECT method CONNECT

http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager

auth_param basic program /usr/lib/squid/basic_ncsa_auth /etc/squid/passwords
auth_param basic realm proxy
acl authenticated proxy_auth REQUIRED

http_access allow authenticated

always_direct allow all

http_port 3128
  1. Enctrypt a username/password into a file (the htpasswd might require installation some apache-utils).
sudo htpasswd -c /etc/squid/passwords username_you_like
  1. Enable squid service so that it auto-starts once system reboots: sudo systemctl enable squid
  2. Launch squid: sudo systemctl start squid

2.Setting Up Secure Web Proxy

Once you have a simple HTTP proxy, you can wrap it with stunnel to get a Secure Web Proxy.

NOTE: I originally tried to put squid behind NGINX, but this didn't work out since NGINX doesn't support HTTP tunnelling.

  1. Install stunnel
  2. Add config to /etc/stunnel/stunnel.conf:
[squid]
accept = 3129
connect = 127.0.0.1:3128
cert = <path to cert>
key = <path to key>
  1. Edit /etc/default/stunnel4 to enable service:
ENABLED=1
  1. Run stunnel: sudo systemctl start stunnel4.service

NOTE: stunnel v4 crashes for me on Ubuntu 18.04 frequently. This seems to be this bug: https://bugs.launchpad.net/ubuntu/+source/stunnel4/+bug/1847275

3. Setting up Socks proxy

I followed these steps and they worked pretty good for me.

The summary;

  1. Install dante server: apt install dante-server
  2. Backup default config: mv /etc/danted.conf /etc/danted.conf.backup
  3. Put the following in the config /etc/danted.conf
logoutput: /var/log/socks.log
internal: eth0 port = 1080
external: eth0
clientmethod: none

socksmethod: none

client pass {
        from: 11.22.333.333/32 to: 0.0.0.0/0
        log: error connect disconnect
}

client block {
        from: 0.0.0.0/0 to: 0.0.0.0/0
        log: connect error
}

socks pass {
        from: 0.0.0.0/0 to: 0.0.0.0/0
        log: error connect disconnect
}

socks block {
        from: 0.0.0.0/0 to: 0.0.0.0/0
        log: connect error
}

NOTE: the 11.22.333.333 IP address is the IP that has access to the socks. For multiple IPs, copy the client pass block.

  1. enable danted systemctl enable danted
  2. run danted systemctl start danted
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment