Skip to content

Instantly share code, notes, and snippets.

@raggi

raggi/.gitignore Secret

Last active December 11, 2015 23:18
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save raggi/4a4e005b575d71c92a24 to your computer and use it in GitHub Desktop.
Save raggi/4a4e005b575d71c92a24 to your computer and use it in GitHub Desktop.
basic rubygems validator
*.list
all
top1m
latest
rubyforge
rubygems-shas
rubygems-md5
rubyforge-md5
*.log
#!/usr/bin/env sh
set -e
mkdir -p all
cd all
if [[ ! -f specs.4.8.gz ]]; then
curl -s -o specs.4.8.gz http://production.cf.rubygems.org/specs.4.8.gz
fi
if [[ ! -f specs.4.8 ]]; then
gzip -d specs.4.8
fi
if [[ ! -f specs.list ]]; then
ruby -e "Marshal.load(File.read('specs.4.8')).map { |n, v, p| puts [n, v, p == 'ruby' ? nil : p].compact.join('-') }" > specs.list
fi
cat specs.list | xargs -P 20 -I% -n 1 ../validate_from_url.sh http://production.cf.rubygems.org/gems/%.gem
#!/usr/bin/env sh
set -e
sum=$1
gem=$(basename $2)
dir=${gem/.gem/}
echo $gem
cd latest/$dir
if [[ "" = $((echo $1 $2; sha512sum ./$gem | awk '{print $1 " " $2}') | uniq -u) ]]; then
echo $gem >> ../../valid_checksums.list
else
echo $gem > ../../invalid_checksums.list
fi
#!/usr/bin/env sh
set -e
cat rubygems-shas | xargs -n 2 -P 20 ./validate_checksum.sh
#!/usr/bin/env sh
set -e
url=$1
gem=$(basename $1)
dir=${gem/.gem/}
echo $gem
mkdir -p $dir
cd $dir
if [[ ! -f $gem ]]; then
curl -s -o $gem $url
fi
if [[ ! -f metadata.gz ]]; then
tar xf $gem
fi
if $(zgrep '!ruby' metadata.gz | grep -v '!ruby/object:Gem::'); then
echo $gem >> ../../suspects.list
else
echo $gem >> ../../clean.list
fi
#!/usr/bin/env sh
url=https://gist.github.com/raw/4675540/72b8146c46412721c49d7e3d1036004622f7907c/gistfile1.txt
mkdir -p latest
cd latest
curl -s $url | xargs -P 20 -n 1 ../validate_from_url.sh
#!/usr/bin/env sh
if ! which openssl > /dev/null; then
echo Install openssl
exit 1
fi
if ! which curl > /dev/null; then
echo Install curl
exit 1
fi
echo This will take a while...
for gem in $(locate \*.gem); do
gemfile=$(basename $gem)
local=$(openssl md5 $gem | awk '{print $2}')
remote=$(curl -s -D - -X HEAD -H 'Connection:close' http://production.cf.rubygems.org/gems/$gemfile | grep 'ETag' | cut -d '"' -f 2)
if [[ ! $local = $remote ]]; then
echo $gemfile mismatch. local: $local, remote: $remote
fi
done
echo All done.
#!/usr/bin/env sh
sum=$1
gem=$(basename $2)
echo $gem
if [[ $sum = $(curl -s -D - -X HEAD -H 'Connection:close' http://production.cf.rubygems.org/gems/$gem | grep ETag | cut -d '"' -f 2) ]]; then
echo $gem >> valid_md5.list
else
echo $gem >> invalid_md5.list
fi
#!/usr/bin/env sh
cat rubygems-md5 | xargs -P 100 -n 2 ./validate_md5s3.sh
rf = open('rubyforge-md5')
rf_sums = rf.each_line.map { |l| l.split(/\s+/) }
rf_sums.map! { |sum, name| [sum, File.basename(name)] }
cf = open('rubygems-md5')
cf_sums = cf.each_line.map { |l| l.split(/\s+/) }
cf_sums.map! { |sum, name| [sum, File.basename(name)] }
rf_sums.each do |sum, name|
cs, cn = cf_sums.find { |csum, cname| cname == name }
next unless cs && cn
next if cs == sum
puts name
end
#!/usr/bin/env sh
url=https://gist.github.com/raw/4675642/398f4d86c44381b4557050aa05018b7e15fe3b10/gistfile1.txt
mkdir -p top1m
cd top1m
curl -s $url | xargs -P 20 -n 1 ../validate_from_url.sh
@leckylao
Copy link

leckylao commented Feb 1, 2013

One stupid question: How do I get the following files?
rubygems-shas
rubygems-md5
rubyforge-md5

@leckylao
Copy link

leckylao commented Feb 1, 2013

Found it.

.gem file checksums
Unverified list (SHA512): https://gist.github.com/4685276 This list is updated as new checksums are created. Check the Revisions for the last update
RubyGems (MD5): http://cl.ly/MY8P
RubyGems (SHA512): http://cl.ly/MYie
RubyForge (MD5): https://gist.github.com/69c89387c6c3bfcf8719
AT&T Mirror (SHA512): https://gist.github.com/4679063
does not include prereleases or the last 2 months

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment