Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
A script to validate your local gem cache against the public s3 repositories. If you find mismatches that contain both local and remote values, please post them in comments.
#!/usr/bin/env sh
if ! which md5sum > /dev/null; then
echo Install md5sum
exit 1
fi
if ! which curl > /dev/null; then
echo Install curl
exit 1
fi
home=$(gem env GEM_HOME)
cache=$home/cache
echo This will take a while...
for gem in $cache/*.gem; do
gemfile=$(basename $gem)
local=$(md5sum $gem | awk '{print $1}')
remote=$(curl -s -D - -X HEAD -H 'Connection:close' http://production.cf.rubygems.org/gems/$gemfile | grep 'ETag' | cut -d '"' -f 2)
if [[ ! $local = $remote ]]; then
echo $gemfile mismatch. local: $local, remote: $remote
fi
done
echo All done.
#!/usr/bin/env sh
if ! which openssl > /dev/null; then
echo Install openssl
exit 1
fi
if ! which curl > /dev/null; then
echo Install curl
exit 1
fi
home=$(gem env GEM_HOME)
cache=$home/cache
echo This will take a while...
for gem in $cache/*.gem; do
gemfile=$(basename $gem)
local=$(openssl md5 $gem | awk '{print $2}')
remote=$(curl -s -D - -X HEAD -H 'Connection:close' http://production.cf.rubygems.org/gems/$gemfile | grep 'ETag' | cut -d '"' -f 2)
if [[ ! $local = $remote ]]; then
echo $gemfile mismatch. local: $local, remote: $remote
fi
done
echo All done.
#!/usr/bin/env sh
if ! which openssl > /dev/null; then
echo Install openssl
exit 1
fi
if ! which curl > /dev/null; then
echo Install curl
exit 1
fi
echo This will take a while...
for gem in $(locate \*.gem); do
gemfile=$(basename $gem)
local=$(openssl md5 $gem | awk '{print $2}')
remote=$(curl -s -D - -X HEAD -H 'Connection:close' http://production.cf.rubygems.org/gems/$gemfile | grep 'ETag' | cut -d '"' -f 2)
if [[ ! $local = $remote ]]; then
echo $gemfile mismatch. local: $local, remote: $remote
fi
done
echo All done.
@raggi

This comment has been minimized.

Copy link
Owner Author

raggi commented Jan 30, 2013

If you're on OSX and you don't have md5sum, and you use homebrew, do this:

brew install md5sha1sum
@raggi

This comment has been minimized.

Copy link
Owner Author

raggi commented Jan 30, 2013

Alternatively, use the openssl variant..

@drnic

This comment has been minimized.

@drnic

This comment has been minimized.

Copy link

drnic commented Jan 30, 2013

Since we're all in the mood for learning :)

@mtjhax

This comment has been minimized.

Copy link

mtjhax commented Jan 30, 2013

Nice tip drnic, but I will just mention you need edit the script to use md5 -r instead of md5sum, since an alias will not work.

@raggi

This comment has been minimized.

Copy link
Owner Author

raggi commented Jan 30, 2013

If you have a valid locatedb, then validate_locate.sh is probably best, as it'll cover other gemsets etc.

@jfelchner

This comment has been minimized.

Copy link

jfelchner commented Jan 31, 2013

@raggi Thank you for posting this.

@robinjam

This comment has been minimized.

Copy link

robinjam commented Jan 31, 2013

Thanks very much for posting this. You've put my mind at ease.

@coffeeaddict

This comment has been minimized.

Copy link

coffeeaddict commented Jan 31, 2013

😞 ./validate_local_cache.sh: 24: ./validate_local_cache.sh: [[: not found

(Ubuntu 12.10 x86_64, zsh 5.0.0 (x86_64-unknown-linux-gnu))

[edit]
Running this with bash ./validate_local_cache.sh makes it happen - perhaps you should amend the she-bang to #!/usr/bin/env bash
[/edit]

@kunik

This comment has been minimized.

Copy link

kunik commented Jan 31, 2013

rack-perftools_profiler-0.4.1.gem mismatch. local: 17bc074dcb1f57a0e4b12739397e428b, remote: 4be04a110f5e7fb42a8d7c1ef2766936
@jcfischer

This comment has been minimized.

Copy link

jcfischer commented Jan 31, 2013

daemon-kit-0.2.0.gem mismatch. local: e48293321ef54695a33ebbcd849d607d, remote: 2ad8fee42336f519e96f7c6329493251
@Holek

This comment has been minimized.

Copy link

Holek commented Jan 31, 2013

awesome_bot_factory-0.0.1.gem mismatch. local: b9f7ab361ac31527f9c3a4f560062c9e, remote: d1eeb86996c50f93be882fa7cf66e55e
@BraveSirRobin

This comment has been minimized.

Copy link

BraveSirRobin commented Jan 31, 2013

sahi-1.0.0.gem mismatch. local: c32f744846c751d51f3a4e296432daa8, remote: faff6ce473c47a4bcb5e0c9b5a4def11

robin@robin-harvey-workstation:gem-checker$ locate *.gem | grep sahi | xargs ls -lh
-rw-rw-r-- 1 robin robin 11K Jul 19 2011 /home/robin/sahi/ruby/sahi-1.0.0.gem
-rw-rw-r-- 1 robin robin 11K Jul 19 2011 /home/robin/sahi/ruby/sahi-1.0.1.gem

@jasonroelofs

This comment has been minimized.

Copy link

jasonroelofs commented Jan 31, 2013

fission-0.4.0.gem mismatch. local: 0f010d636fbbdc036004710b91e3c2b1, remote: d34073ffa3435be30a5f92856c76ed33

@blasterpal

This comment has been minimized.

Copy link

blasterpal commented Jan 31, 2013

Here's an OSX ready script, that also has some hints for folks using bundler. Also OSX md5 works fine, but awk must look at $4 not $1. Anyhoo: https://gist.github.com/4683645

@lwdallas

This comment has been minimized.

Copy link

lwdallas commented Jan 31, 2013

Much appreciated!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.