Skip to content

Instantly share code, notes, and snippets.

@dctrwatson
Last active April 28, 2024 10:26
Show Gist options
  • Save dctrwatson/5785638 to your computer and use it in GitHub Desktop.
Save dctrwatson/5785638 to your computer and use it in GitHub Desktop.
Caching PyPi packages locally with nginx
user www-data;
worker_processes 4;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
access_log off;
default_type application/octet-stream;
sendfile on;
tcp_nodelay on;
tcp_nopush off;
reset_timedout_connection on;
server_tokens off;
# Cache 100G worth of packages for up to 1 month
proxy_cache_path /var/lib/nginx/pypi levels=1:2 keys_zone=pypi:16m inactive=1M max_size=100G;
# Multiple server definitions makes nginx retry on errors
upstream pypi {
server pypi.python.org:443;
server pypi.python.org:443;
keepalive 16;
}
gzip on;
gzip_types application/json text/css text/javascript;
gzip_proxied any;
gzip_vary on;
server {
listen 80 default_server;
server_name pypi.example.com;
root /var/www;
proxy_cache pypi;
proxy_cache_key $uri;
proxy_cache_lock on;
proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
proxy_http_version 1.1;
proxy_set_header Host pypi.python.org;
proxy_set_header Connection "";
proxy_set_header Accept-Encoding "";
# Rewrite any http redirects to use relative to proxy
proxy_redirect ~https?://pypi.python.org(.*) $1;
location / {
# Replace any reference to actual pypi w/ caching proxy
sub_filter 'https://pypi.python.org' $scheme://$host;
sub_filter_once off;
proxy_pass https://pypi;
proxy_cache off;
}
location ^~ /simple {
# Make sure URI ends with /
rewrite ^(.*[^/])$ $1/ break;
add_header X-Cache2 $upstream_cache_status;
proxy_cache_valid any 5m;
proxy_pass https://pypi;
}
location ^~ /packages {
add_header X-Cache2 $upstream_cache_status;
proxy_cache_valid any 1M;
proxy_pass https://pypi;
}
}
}
@cedef
Copy link

cedef commented Nov 9, 2018

You just saved my life with your config snippet dude !
Thanks a lot !!

@ecerulm
Copy link

ecerulm commented May 19, 2022

As of 2022 this nginx won't work as is. It requires some changes / additions:

  • All references to pypi.python.org (proxy_set_header Host pypi.org, upstream, etc) should be changed to the current primary domain pypi.org, otherwise you just will get 301 and pip just thinks there is no package with that name
  • proxy_ssl_server_name should be set to 'on, as pypi.org` requires SNI.

@hauntsaninja
Copy link

This needs several changes to work in 2023, in addition to the things ecerulm mentions.

Here's what works for me (includes tests to prove it works for pip): https://github.com/hauntsaninja/nginx_pypi_cache

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment