Skip to content

Instantly share code, notes, and snippets.

@xaviershay
Created January 10, 2013 06:11
Show Gist options
  • Save xaviershay/4499828 to your computer and use it in GitHub Desktop.
Save xaviershay/4499828 to your computer and use it in GitHub Desktop.
Proof-of-concept squid proxy settings for proxying http://rubygems.org/ Tested on one project on my local machine both cold, and warm with internet turned off.
#!/usr/bin/env ruby
# url rewriter for rubygems squid proxy
STDOUT.sync = true
while line = gets
url = line.split(' ')[0]
# Cargo-culted this conditional, not sure if it is necessary
response = if url
# Need to find a better way to cache this redirect, rather than hard-coding
# it.
if url.include?("http://rubygems.org/gems/")
"302:" + url.gsub(
"http://rubygems.org/",
"http://production.cf.rubygems.org/"
)
else
""
end
else
""
end
puts response
end
##########################################################################
#
# Proof-of-concept squid proxy settings for proxying http://rubygems.org/
# Caches everything so is resilient to network outage or upstream outage.
# Works with bundler, gives a significant speed up when cache is warm (100%+)
#
##########################################################################
# Disable default patterns that prevent caching of query params
# refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
# refresh_pattern . 0 20% 4320
# Cache calls to resolve dependency list for at least 15 minutes, at max a day
# TODO: What is the actual behaviour of this? Desired is refresh after 15, but
# continue to work for a day if upstream is unavailable.
refresh_pattern -i /api/v1/dependencies 900 100% 86400
# "Cache" redirects so that the proxy is completely isolated from the internet
# when it is warm.
url_rewrite_program rewrite.rb
# 1 Gb disk cache. Probably larger is better.
cache_dir ufs /usr/local/var/cache/squid 1024 16 256
# BELOW THIS LINE IS DEFAULT SQUID CONFIG
# Recommended minimum configuration:
#
# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
#
# Recommended minimum Access Permission configuration:
#
# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager
# Deny requests to certain unsafe ports
http_access deny !Safe_ports
# Deny CONNECT to other than secure SSL ports
http_access deny CONNECT !SSL_ports
# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
#http_access deny to_localhost
#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#
# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
http_access allow localnet
http_access allow localhost
# And finally deny all other access to this proxy
http_access deny all
# Squid normally listens to port 3128
http_port 3128
# Leave coredumps in the first cache dir
coredump_dir /usr/local/var/cache/squid
@elskwid
Copy link

elskwid commented Jan 10, 2013

@xaviershay, what about the call that bundler makes for dependency resolution? I wonder if bundle install --full-index would use a cached gems index in that case? That might work.

@wolfeidau
Copy link

I would be interested to see how successful this is, have you run a report on using the logs produced ?

http://cord.de/tools/squid/calamaris/ is what i use to gather information about HITS/MISSES and how much is "saved".

@kevinelliott
Copy link

I want a rubygems caching proxy to work. It seems like with the heavy use of RVM and various ruby interpreters, installing gems faster would be a significant gain.

@dpritchett
Copy link

I know I'm a year late but playing with Docker has got me really hoping for a way to cache gem downloads. Anyone ever come up with a good solution?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment