Caching PyPi packages locally with nginx

nginx.conf

user www-data;
worker_processes 4;

error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

events {
  worker_connections 1024;
}

http {
  include /etc/nginx/mime.types;

  access_log off;

  default_type application/octet-stream;

  sendfile on;
  tcp_nodelay on;
  tcp_nopush off;

  reset_timedout_connection on;

  server_tokens off;

  # Cache 100G worth of packages for up to 1 month
  proxy_cache_path /var/lib/nginx/pypi levels=1:2 keys_zone=pypi:16m inactive=1M max_size=100G;

  # Multiple server definitions makes nginx retry on errors
  upstream pypi {
    server pypi.python.org:443;
    server pypi.python.org:443;
    keepalive 16;
  }

  gzip on;
  gzip_types application/json text/css text/javascript;
  gzip_proxied any;
  gzip_vary on;

  server {
    listen 80 default_server;
    server_name pypi.example.com;

    root /var/www;

    proxy_cache pypi;
    proxy_cache_key $uri;
    proxy_cache_lock on;
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;

    proxy_http_version 1.1;
    proxy_set_header Host pypi.python.org;
    proxy_set_header Connection "";
    proxy_set_header Accept-Encoding "";

    # Rewrite any http redirects to use relative to proxy
    proxy_redirect ~https?://pypi.python.org(.*) $1;

    location / {
      # Replace any reference to actual pypi w/ caching proxy
      sub_filter 'https://pypi.python.org' $scheme://$host;
      sub_filter_once off;
      proxy_pass https://pypi;
      proxy_cache off;
    }

    location ^~ /simple {
      # Make sure URI ends with /
      rewrite ^(.*[^/])$ $1/ break;

      add_header X-Cache2 $upstream_cache_status;

      proxy_cache_valid any 5m;

      proxy_pass https://pypi;
    }

    location ^~ /packages {
      add_header X-Cache2 $upstream_cache_status;

      proxy_cache_valid any 1M;

      proxy_pass https://pypi;
    }
  }
}

You just saved my life with your config snippet dude !
Thanks a lot !!

As of 2022 this nginx won't work as is. It requires some changes / additions:

• All references to pypi.python.org (proxy_set_header Host pypi.org, upstream, etc) should be changed to the current primary domain pypi.org, otherwise you just will get 301 and pip just thinks there is no package with that name
• proxy_ssl_server_name should be set to 'on, as pypi.org` requires SNI.

This needs several changes to work in 2023, in addition to the things ecerulm mentions.

Here's what works for me (includes tests to prove it works for pip): https://github.com/hauntsaninja/nginx_pypi_cache