Create a gist now

Instantly share code, notes, and snippets.

@fnichol /README.md
Last active Jul 24, 2017

What would you like to do?
Auto-enable Local HTTP Caching in Test Kitchen

Auto-enable Local HTTP Caching in Test Kitchen

Note: total experiment and hack, looks nasty, could be awesome:

Setup

  • Drop the kitchen.local.yml into $HOME/.kitchen/config.yml
  • Install polipo (with Mac: brew install polipo, with Ubuntu: apt-get install polipo)
  • Drop polipo-start and polipo-console somewhere useful (perhaps $HOME/bin?)

Run with HTTP Caching Proxy

Simply start up polipo in the foreground in a terminal/tmux/screen session with: ~/bin/polipo-start. In your Test Kitchen project verify that caching is enabled by running kitchen diagnose and look for the http_proxy and chef_omnibus_url config options, like in this abbreviated example output:

---
timestamp: 2015-03-06 19:27:56 UTC
kitchen_version: 1.3.1
instances:
  default-ubuntu-1204:
    state_file: {}
    driver:
      http_proxy: http://10.0.10.20:8123
      https_proxy: http://10.0.10.20:8123
      provision_command: env http_proxy=http://10.0.10.20:8123 bash -c 'curl -L
        http://www.chef.io/chef/install.sh | bash'
    provisioner:
      chef_omnibus_url: http://www.chef.io/chef/install.sh
<%
require 'socket'
# @return [String] public IP address of workstation used for egress traffic
def local_ip
@local_ip ||= begin
# turn off reverse DNS resolution temporarily
orig, Socket.do_not_reverse_lookup = Socket.do_not_reverse_lookup, true
UDPSocket.open do |s|
s.connect '64.233.187.99', 1 # a google IP, does not hit network
s.addr.last
end
ensure
Socket.do_not_reverse_lookup = orig
end
end
# @return [Integer] default polipo listening port
def local_port ; 8123 ; end
# @return [String] the polipo proxy URL
def http_proxy_url ; "http://#{local_ip}:#{local_port}" ; end
# @return [TrueClass,FalseClass] whether or not the polipo port is listening
def proxy_running?
socket = TCPSocket.new(local_ip, local_port)
true
rescue SocketError, Errno::ECONNREFUSED,
Errno::EHOSTUNREACH, Errno::ENETUNREACH, IOError
false
rescue Errno::EPERM, Errno::ETIMEDOUT
false
ensure
socket && socket.close
end
%>
---
<% if proxy_running? %>
driver:
http_proxy: <%= http_proxy_url %>
https_proxy: <%= http_proxy_url %>
# used in kitchen-docker to pre-install/cache a Chef Ommnibus package in a
# layer before booting containers
provision_command: "env http_proxy=<%= http_proxy_url %> bash -c 'curl -L http://www.chef.io/chef/install.sh | bash'"
provisioner:
# installs from an http:// source to enable better caching through
# http_proxy, and yes, this is a bit evil
chef_omnibus_url: http://www.chef.io/chef/install.sh
<% end %>
#!/usr/bin/env bash
set -e
[ -n "$DEBUG" ] && set -x
if ! command -v lynx >/dev/null ; then
echo "$(basename $0) - lynx command not found, please install and retry"
exit 10
fi
exec lynx "http://127.0.0.1:8123"
#!/usr/bin/env bash
set -e
[ -n "$DEBUG" ] && set -x
if ! command -v polipo >/dev/null ; then
echo "$(basename $0) - polipo command not found, please install and retry"
exit 10
fi
mkdir -p "$HOME/.polipo-cache"
exec polipo \
proxyAddress='0.0.0.0' \
disableIndexing='false' \
disableServersList='false' \
allowedClients='0.0.0.0/0' \
diskCacheRoot='~/.polipo-cache'

Awesome! Must try this.

esciara commented Apr 16, 2014

With reference to test-kitchen/kitchen-vagrant#90: remote_file is not always cached with spike as is. Need to add the following to .kitchen.local.yml :

provisioner:
  chef_omnibus_url: http://www.getchef.com/chef/install.sh
  solo_rb:
    http_proxy: <%= http_proxy_url %>
    https_proxy: <%= http_proxy_url %>

Even with this, it seems that some apt calls are not cached. (see issue link for more details)

Thanks so much!

I had an issue with Polipo 1.0.4.1 from homebrew on OSX 10.9.2., random seg faults. The 1.1.0 works much better so I put in a PR to homebrew.

slyness commented Oct 6, 2014

kitchen list --debug is depreciated. Use kitchen diagnose to examine your configuration.

pmocek commented Oct 24, 2014

I suggest using https://www.getchef.com instead of http://www.getchef.com.

Owner

fnichol commented Mar 6, 2015

@slyness You're totally right, updated the README, thanks!

Owner

fnichol commented Mar 6, 2015

@pmocek Yes, you're right about highly preferring the HTTPS URL over the HTTP one, but I'm majorly cheating here in this testing context to get faster cached packages. Easily something you might want to change in your version. I've updated the YAML with a comment that I hope is slightly clearer.

estahn commented Mar 7, 2015

@fnichol Works like a charm, cheers. I had huge issues with Polipo. First segfaults with the MacPorts version and then proxy issues with the latest release from GitHub. apt-get update would only return Ignore ... or 404. Looks like the proxy implementation might be faulty in some ways.

I'm using SquidMan now and it works without any problems so far.

estahn commented Mar 18, 2015

@fnichol Just took your snippet and used it for vagrant-proxyconf. Works great.

tmatilai/vagrant-proxyconf#112

The only chef_omnibus_url that still works with a HTTP proxy cache is http://www.opscode.com/chef/install.sh I'm afraid. The rest (http://www.getchef.com/chef/install.sh, http://www.chef.io/chef/install.sh) send the request to omnitruck-direct.chef.io over HTTPS and the S3 URL is over HTTPS as well.

Tested with Ubuntu 15.04:

root@9c04906b5ba8:~# curl -L http://www.opscode.com/chef/install.sh -o /tmp/install.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 19970  100 19970    0     0  39383      0 --:--:-- --:--:-- --:--:-- 39466
root@9c04906b5ba8:~# bash /tmp/install.sh -v 12.5.1
Getting information for chef stable 12.5.1 for ubuntu...
downloading http://omnitruck-direct.chef.io/stable/chef/metadata?v=12.5.1&p=ubuntu&pv=15.04&m=x86_64
  to file /tmp/install.sh.105/metadata.txt
trying curl...
url http://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/14.04/x86_64/chef_12.5.1-1_amd64.deb
md5 d8fec2da288e94a7e2d649803a9d70f4
sha256  656a4c4a8fd64d74d1d970fb0d07076d6f1d8230d37d751f2c3698a52d82c070
version 12.5.1downloaded metadata file looks valid...
downloading http://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/14.04/x86_64/chef_12.5.1-1_amd64.deb
  to file /tmp/install.sh.105/chef_12.5.1-1_amd64.deb
trying curl...
Comparing checksum with sha256sum...
Installing chef 12.5.1
installing with dpkg...
(Reading database ... 26118 files and directories currently installed.)
Preparing to unpack .../chef_12.5.1-1_amd64.deb ...
Unpacking chef (12.5.1-1) over (12.5.1-1) ...
Setting up chef (12.5.1-1) ...
Thank you for installing Chef!

root@9c04906b5ba8:~# curl -L http://www.chef.io/chef/install.sh -o /tmp/install.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 19957  100 19957    0     0  42416      0 --:--:-- --:--:-- --:--:-- 42371
root@9c04906b5ba8:~# bash /tmp/install.sh -v 12.5.1
Getting information for chef stable 12.5.1 for ubuntu...
downloading http://www.chef.io/stable/chef/metadata?v=12.5.1&p=ubuntu&pv=15.04&m=x86_64
  to file /tmp/install.sh.12/metadata.txt
trying curl...
url https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/14.04/x86_64/chef_12.5.1-1_amd64.deb
md5 d8fec2da288e94a7e2d649803a9d70f4
sha256  656a4c4a8fd64d74d1d970fb0d07076d6f1d8230d37d751f2c3698a52d82c070
version 12.5.1downloaded metadata file looks valid...
downloading https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/14.04/x86_64/chef_12.5.1-1_amd64.deb
  to file /tmp/install.sh.12/chef_12.5.1-1_amd64.deb
trying curl...
Comparing checksum with sha256sum...
Installing chef 12.5.1
installing with dpkg...
(Reading database ... 26118 files and directories currently installed.)
Preparing to unpack .../chef_12.5.1-1_amd64.deb ...
Unpacking chef (12.5.1-1) over (12.5.1-1) ...
Setting up chef (12.5.1-1) ...
Thank you for installing Chef!

root@9c04906b5ba8:~# curl -L http://www.getchef.com/chef/install.sh -o /tmp/install.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 19971  100 19971    0     0  91830      0 --:--:-- --:--:-- --:--:-- 91830
root@9c04906b5ba8:~# bash /tmp/install.sh -v 12.5.1
Getting information for chef stable 12.5.1 for ubuntu...
downloading https://omnitruck-direct.chef.io/stable/chef/metadata?v=12.5.1&p=ubuntu&pv=15.04&m=x86_64
  to file /tmp/install.sh.198/metadata.txt
trying curl...
url https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/14.04/x86_64/chef_12.5.1-1_amd64.deb
md5 d8fec2da288e94a7e2d649803a9d70f4
sha256  656a4c4a8fd64d74d1d970fb0d07076d6f1d8230d37d751f2c3698a52d82c070
version 12.5.1downloaded metadata file looks valid...
downloading https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/14.04/x86_64/chef_12.5.1-1_amd64.deb
  to file /tmp/install.sh.198/chef_12.5.1-1_amd64.deb
trying curl...
Comparing checksum with sha256sum...
Installing chef 12.5.1
installing with dpkg...
(Reading database ... 26118 files and directories currently installed.)
Preparing to unpack .../chef_12.5.1-1_amd64.deb ...
Unpacking chef (12.5.1-1) over (12.5.1-1) ...
Setting up chef (12.5.1-1) ...
Thank you for installing Chef!

Would be great if the proxy_running can exit in a quick way when polipo is not running otherwise it crawls every kitchen command.

eherot commented Dec 5, 2016

Noticing that you catch Errno::ETIMEDOUT here but without doing anything to enforce a meaningful timeout on TCPSocket.new. By default it's 75 seconds which makes most kitchen commands unusable in the event that this gets triggered.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment