public
Last active

A script to validate your local gem cache against the public s3 repositories. If you find mismatches that contain both local and remote values, please post them in comments.

  • Download Gist
validate_local_cache.sh
Shell
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
#!/usr/bin/env sh
 
if ! which md5sum > /dev/null; then
echo Install md5sum
exit 1
fi
 
if ! which curl > /dev/null; then
echo Install curl
exit 1
fi
 
home=$(gem env GEM_HOME)
cache=$home/cache
 
echo This will take a while...
 
for gem in $cache/*.gem; do
gemfile=$(basename $gem)
 
local=$(md5sum $gem | awk '{print $1}')
remote=$(curl -s -D - -X HEAD -H 'Connection:close' http://production.cf.rubygems.org/gems/$gemfile | grep 'ETag' | cut -d '"' -f 2)
 
if [[ ! $local = $remote ]]; then
echo $gemfile mismatch. local: $local, remote: $remote
fi
done
 
echo All done.
validate_local_cache_openssl.sh
Shell
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
#!/usr/bin/env sh
 
if ! which openssl > /dev/null; then
echo Install openssl
exit 1
fi
 
if ! which curl > /dev/null; then
echo Install curl
exit 1
fi
 
home=$(gem env GEM_HOME)
cache=$home/cache
 
echo This will take a while...
 
for gem in $cache/*.gem; do
gemfile=$(basename $gem)
 
local=$(openssl md5 $gem | awk '{print $2}')
remote=$(curl -s -D - -X HEAD -H 'Connection:close' http://production.cf.rubygems.org/gems/$gemfile | grep 'ETag' | cut -d '"' -f 2)
 
if [[ ! $local = $remote ]]; then
echo $gemfile mismatch. local: $local, remote: $remote
fi
done
 
echo All done.
validate_locate.sh
Shell
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
#!/usr/bin/env sh
 
if ! which openssl > /dev/null; then
echo Install openssl
exit 1
fi
 
if ! which curl > /dev/null; then
echo Install curl
exit 1
fi
 
echo This will take a while...
 
for gem in $(locate \*.gem); do
gemfile=$(basename $gem)
 
local=$(openssl md5 $gem | awk '{print $2}')
remote=$(curl -s -D - -X HEAD -H 'Connection:close' http://production.cf.rubygems.org/gems/$gemfile | grep 'ETag' | cut -d '"' -f 2)
 
if [[ ! $local = $remote ]]; then
echo $gemfile mismatch. local: $local, remote: $remote
fi
done
 
echo All done.

If you're on OSX and you don't have md5sum, and you use homebrew, do this:

brew install md5sha1sum

Alternatively, use the openssl variant..

Since we're all in the mood for learning :)

Nice tip drnic, but I will just mention you need edit the script to use md5 -r instead of md5sum, since an alias will not work.

If you have a valid locatedb, then validate_locate.sh is probably best, as it'll cover other gemsets etc.

@raggi Thank you for posting this.

Thanks very much for posting this. You've put my mind at ease.

:disappointed:./validate_local_cache.sh: 24: ./validate_local_cache.sh: [[: not found

(Ubuntu 12.10 x86_64, zsh 5.0.0 (x86_64-unknown-linux-gnu))

[edit]
Running this with bash ./validate_local_cache.sh makes it happen - perhaps you should amend the she-bang to #!/usr/bin/env bash
[/edit]

rack-perftools_profiler-0.4.1.gem mismatch. local: 17bc074dcb1f57a0e4b12739397e428b, remote: 4be04a110f5e7fb42a8d7c1ef2766936
daemon-kit-0.2.0.gem mismatch. local: e48293321ef54695a33ebbcd849d607d, remote: 2ad8fee42336f519e96f7c6329493251
awesome_bot_factory-0.0.1.gem mismatch. local: b9f7ab361ac31527f9c3a4f560062c9e, remote: d1eeb86996c50f93be882fa7cf66e55e

sahi-1.0.0.gem mismatch. local: c32f744846c751d51f3a4e296432daa8, remote: faff6ce473c47a4bcb5e0c9b5a4def11

robin@robin-harvey-workstation:gem-checker$ locate *.gem | grep sahi | xargs ls -lh
-rw-rw-r-- 1 robin robin 11K Jul 19 2011 /home/robin/sahi/ruby/sahi-1.0.0.gem
-rw-rw-r-- 1 robin robin 11K Jul 19 2011 /home/robin/sahi/ruby/sahi-1.0.1.gem

fission-0.4.0.gem mismatch. local: 0f010d636fbbdc036004710b91e3c2b1, remote: d34073ffa3435be30a5f92856c76ed33

Here's an OSX ready script, that also has some hints for folks using bundler. Also OSX md5 works fine, but awk must look at $4 not $1. Anyhoo: https://gist.github.com/4683645

Much appreciated!

Please sign in to comment on this gist.

Something went wrong with that request. Please try again.