-
-
Save raggi/4a4e005b575d71c92a24 to your computer and use it in GitHub Desktop.
basic rubygems validator
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
*.list | |
all | |
top1m | |
latest | |
rubyforge | |
rubygems-shas | |
rubygems-md5 | |
rubyforge-md5 | |
*.log |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env sh | |
set -e | |
mkdir -p all | |
cd all | |
if [[ ! -f specs.4.8.gz ]]; then | |
curl -s -o specs.4.8.gz http://production.cf.rubygems.org/specs.4.8.gz | |
fi | |
if [[ ! -f specs.4.8 ]]; then | |
gzip -d specs.4.8 | |
fi | |
if [[ ! -f specs.list ]]; then | |
ruby -e "Marshal.load(File.read('specs.4.8')).map { |n, v, p| puts [n, v, p == 'ruby' ? nil : p].compact.join('-') }" > specs.list | |
fi | |
cat specs.list | xargs -P 20 -I% -n 1 ../validate_from_url.sh http://production.cf.rubygems.org/gems/%.gem |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env sh | |
set -e | |
sum=$1 | |
gem=$(basename $2) | |
dir=${gem/.gem/} | |
echo $gem | |
cd latest/$dir | |
if [[ "" = $((echo $1 $2; sha512sum ./$gem | awk '{print $1 " " $2}') | uniq -u) ]]; then | |
echo $gem >> ../../valid_checksums.list | |
else | |
echo $gem > ../../invalid_checksums.list | |
fi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env sh | |
set -e | |
cat rubygems-shas | xargs -n 2 -P 20 ./validate_checksum.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env sh | |
set -e | |
url=$1 | |
gem=$(basename $1) | |
dir=${gem/.gem/} | |
echo $gem | |
mkdir -p $dir | |
cd $dir | |
if [[ ! -f $gem ]]; then | |
curl -s -o $gem $url | |
fi | |
if [[ ! -f metadata.gz ]]; then | |
tar xf $gem | |
fi | |
if $(zgrep '!ruby' metadata.gz | grep -v '!ruby/object:Gem::'); then | |
echo $gem >> ../../suspects.list | |
else | |
echo $gem >> ../../clean.list | |
fi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env sh | |
url=https://gist.github.com/raw/4675540/72b8146c46412721c49d7e3d1036004622f7907c/gistfile1.txt | |
mkdir -p latest | |
cd latest | |
curl -s $url | xargs -P 20 -n 1 ../validate_from_url.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env sh | |
if ! which openssl > /dev/null; then | |
echo Install openssl | |
exit 1 | |
fi | |
if ! which curl > /dev/null; then | |
echo Install curl | |
exit 1 | |
fi | |
echo This will take a while... | |
for gem in $(locate \*.gem); do | |
gemfile=$(basename $gem) | |
local=$(openssl md5 $gem | awk '{print $2}') | |
remote=$(curl -s -D - -X HEAD -H 'Connection:close' http://production.cf.rubygems.org/gems/$gemfile | grep 'ETag' | cut -d '"' -f 2) | |
if [[ ! $local = $remote ]]; then | |
echo $gemfile mismatch. local: $local, remote: $remote | |
fi | |
done | |
echo All done. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env sh | |
sum=$1 | |
gem=$(basename $2) | |
echo $gem | |
if [[ $sum = $(curl -s -D - -X HEAD -H 'Connection:close' http://production.cf.rubygems.org/gems/$gem | grep ETag | cut -d '"' -f 2) ]]; then | |
echo $gem >> valid_md5.list | |
else | |
echo $gem >> invalid_md5.list | |
fi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env sh | |
cat rubygems-md5 | xargs -P 100 -n 2 ./validate_md5s3.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rf = open('rubyforge-md5') | |
rf_sums = rf.each_line.map { |l| l.split(/\s+/) } | |
rf_sums.map! { |sum, name| [sum, File.basename(name)] } | |
cf = open('rubygems-md5') | |
cf_sums = cf.each_line.map { |l| l.split(/\s+/) } | |
cf_sums.map! { |sum, name| [sum, File.basename(name)] } | |
rf_sums.each do |sum, name| | |
cs, cn = cf_sums.find { |csum, cname| cname == name } | |
next unless cs && cn | |
next if cs == sum | |
puts name | |
end | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env sh | |
url=https://gist.github.com/raw/4675642/398f4d86c44381b4557050aa05018b7e15fe3b10/gistfile1.txt | |
mkdir -p top1m | |
cd top1m | |
curl -s $url | xargs -P 20 -n 1 ../validate_from_url.sh |
Found it.
.gem file checksums
Unverified list (SHA512): https://gist.github.com/4685276 This list is updated as new checksums are created. Check the Revisions for the last update
RubyGems (MD5): http://cl.ly/MY8P
RubyGems (SHA512): http://cl.ly/MYie
RubyForge (MD5): https://gist.github.com/69c89387c6c3bfcf8719
AT&T Mirror (SHA512): https://gist.github.com/4679063
does not include prereleases or the last 2 months
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
One stupid question: How do I get the following files?
rubygems-shas
rubygems-md5
rubyforge-md5