Skip to content

Instantly share code, notes, and snippets.

@eregon

eregon/commands.txt Secret

Last active February 15, 2021 18:58
Show Gist options
  • Save eregon/fc24f74752322d88a9de78c7b0e76284 to your computer and use it in GitHub Desktop.
Save eregon/fc24f74752322d88a9de78c7b0e76284 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
commands.txt
rubyspec:
6c466d455bbd03906e4080be52b0fbc7c842fbad
mspec:
6cb1f10b1490f239e2d6267b1747d67011b58c36
Commands:
# CRuby
cd rubyspec
chruby 2.6.6
ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
mspec run -f stats -V .
# TruffleRuby
cd truffleruby
commit: 63ac8611de88ba57befa9f8224b4c71980d0b6ce
ruby-build truffleruby-dev ~/.rubies/truffleruby-dev
chruby truffleruby-dev
truffleruby 20.2.0-dev-aecb8d32, like ruby 2.6.6, GraalVM CE Native [x86_64-linux]
mspec run --config spec/truffleruby.mspec --excl-tag fails -f stats -V --timeout 180 -t ruby spec/ruby
# JRuby
cd jruby
commit: f03792ddeb98d675ce289653262e8ee966594aa6
Also accounted for 2 security fixes not yet merged in master when writing (fixing 6 security specs):
https://github.com/jruby/jruby/pull/6305
https://github.com/jruby/jruby/pull/6306
ruby-install jruby head --url https://github.com/ruby/jruby-dev-builder/releases/latest/download/jruby-head-ubuntu-18.04.tar.gz
chruby jruby-head
jruby 9.3.0.0-SNAPSHOT (2.6.5) 2020-06-25 8cf2992770 OpenJDK 64-Bit Server VM 25.252-b09-jvmci-20.1-b02 on 1.8.0_252-b09 +jit [linux-x86_64]
mspec run --config spec/jruby.mspec --excl-tag fails -f stats -V --timeout 180 -t ruby spec/ruby
# Opal
git clone https://github.com/opal/opal.git
commit: df758db79ed9da8e9cc73e1d42ce5a1410556f24
cd opal
bin/setup
bin/opal -e 'puts RUBY_DESCRIPTION'
opal 1.0.0 (2019-05-12 revision 0)
bundle exec rake mspec_ruby_chrome
1520 files, 13577 examples, 53972 expectations, 0 failures, 0 errors, 2105 tagged
bundle exec rake mspec_ruby_chrome RUBYSPECS=true PATTERN="spec/ruby/language/**/*_spec.rb"
57 files, 1740 examples, 2823 expectations, 0 failures, 0 errors, 87 tagged
bundle exec rake mspec_ruby_chrome RUBYSPECS=true PATTERN="spec/ruby/core/**/*_spec.rb"
1113 files, 10735 examples, 48774 expectations, 0 failures, 0 errors, 1787 tagged
bundle exec rake mspec_ruby_chrome RUBYSPECS=true PATTERN="spec/ruby/library/**/*_spec.rb"
287 files, 713 examples, 1442 expectations, 0 failures, 0 errors, 213 tagged
# Artichoke
git clone https://github.com/artichoke/artichoke.git
commit: 507fd36a516b1117de4303802e580ed4826c2ed7
cd artichoke
dnf install cargo clang llvm-devel
See https://github.com/artichoke/artichoke/blob/trunk/BUILD.md
cargo build --release
./target/release/artichoke --version
artichoke 0.1.0-pre.0 (2020-06-28 revision 3265) [x86_64-unknown-linux-gnu]
cargo run -q -p spec-runner -- spec-runner/enforced-specs.yaml
Passed 1694, skipped 214, not implemented 0, failed 0 specs.
cargo run -q -p spec-runner -- spec-runner/core-specs.yaml
Passed 1367, skipped 202, not implemented 0, failed 0 specs.
cargo run -q -p spec-runner -- spec-runner/library-specs.yaml
Passed 327, skipped 12, not implemented 0, failed 0 specs.
# To generate the table
ruby tool/analyze_per_file.rb --html --mri-totals results-ruby-2.6.6.yml results-truffleruby-20.2.0-dev-aecb8d32.yml results-jruby-9.3.0.0-SNAPSHOT.yml results-opal-dev.yml results-artichoke-dev.yml > ../blog/_includes/spec_report_table.html
Raw
results-artichoke-dev.yml
---
core/all_spec.rb:
:examples: 1367
:errors: 0
:failures: 0
:tagged: 0
library/all_spec.rb:
:examples: 327
:errors: 0
:failures: 0
:tagged: 0
Raw
results-opal-dev.yml
---
language/all_spec.rb:
:examples: 1740
:errors: 0
:failures: 0
:tagged: 87
core/all_spec.rb:
:examples: 10735
:errors: 0
:failures: 0
:tagged: 1787
library/all_spec.rb:
:examples: 713
:errors: 0
:failures: 0
:tagged: 213
Raw
text_summary.txt
$ ruby tool/analyze_per_file.rb --mri-totals results-ruby-2.6.6.yml results-truffleruby-20.2.0-dev-aecb8d32.yml results-jruby-9.3.0.0-SNAPSHOT.yml results-opal-dev.yml results-artichoke-dev.yml
ruby-2.6.6
command_line 141/ 141 = 100.00%
core 20841/20841 = 100.00%
language 2367/ 2367 = 100.00%
library 6891/ 6891 = 100.00%
capi 1325/ 1325 = 100.00%
security 40/ 40 = 100.00%
total 31605/31605 = 100.00%
total_no_capi 30280/30280 = 100.00%
truffleruby-20.2.0-dev-aecb8d32
command_line 124/ 141 = 87.94%
core 20175/20841 = 96.80%
language 2316/ 2367 = 97.85%
library 6727/ 6891 = 97.62%
capi 1290/ 1325 = 97.36%
security 40/ 40 = 100.00%
total 30672/31605 = 97.05%
total_no_capi 29382/30280 = 97.03%
jruby-9.3.0.0-SNAPSHOT
command_line 104/ 141 = 73.76%
core 19510/20841 = 93.61%
language 2300/ 2367 = 97.17%
library 5924/ 6891 = 85.97%
capi 0/ 1325 = 0.00%
security 37/ 40 = 92.50%
total 27875/31605 = 88.20%
total_no_capi 27875/30280 = 92.06%
opal-dev
language 1653/ 2367 = 69.84%
core 8948/20841 = 42.93%
library 500/ 6891 = 7.26%
command_line 0/ 141 = 0.00%
capi 0/ 1325 = 0.00%
security 0/ 40 = 0.00%
total 11101/31605 = 35.12%
total_no_capi 11101/30280 = 36.66%
artichoke-dev
core 1367/20841 = 6.56%
library 327/ 6891 = 4.75%
command_line 0/ 141 = 0.00%
language 0/ 2367 = 0.00%
capi 0/ 1325 = 0.00%
security 0/ 40 = 0.00%
total 1694/31605 = 5.36%
total_no_capi 1694/30280 = 5.59%
$ ruby tool/analyze_per_file.rb results-ruby-2.6.6.yml results-truffleruby-20.2.0-dev-aecb8d32.yml results-jruby-9.3.0.0-SNAPSHOT.yml results-opal-dev.yml results-artichoke-dev.yml
ruby-2.6.6
command_line 141/ 141 = 100.00%
core 20841/20841 = 100.00%
language 2367/ 2367 = 100.00%
library 6891/ 6891 = 100.00%
capi 1325/ 1325 = 100.00%
security 40/ 40 = 100.00%
total 31605/31605 = 100.00%
total_no_capi 30280/30280 = 100.00%
truffleruby-20.2.0-dev-aecb8d32
command_line 124/ 141 = 87.94%
core 20175/20835 = 96.83%
language 2316/ 2367 = 97.85%
library 6727/ 6891 = 97.62%
capi 1290/ 1322 = 97.58%
security 40/ 40 = 100.00%
total 30672/31596 = 97.08%
total_no_capi 29382/30274 = 97.05%
jruby-9.3.0.0-SNAPSHOT
command_line 104/ 141 = 73.76%
core 19510/20834 = 93.65%
language 2300/ 2367 = 97.17%
library 5924/ 6826 = 86.79%
capi 0/ 1325 = 0.00%
security 37/ 40 = 92.50%
total 27875/31533 = 88.40%
total_no_capi 27875/30208 = 92.28%
opal-dev
language 1653/ 1740 = 95.00%
core 8948/10735 = 83.35%
library 500/ 713 = 70.13%
command_line 0/ 141 = 0.00%
capi 0/ 1325 = 0.00%
security 0/ 40 = 0.00%
total 11101/14694 = 75.55%
total_no_capi 11101/13369 = 83.04%
artichoke-dev
core 1367/ 1367 = 100.00%
library 327/ 327 = 100.00%
command_line 0/ 141 = 0.00%
language 0/ 2367 = 0.00%
capi 0/ 1325 = 0.00%
security 0/ 40 = 0.00%
total 1694/ 5567 = 30.43%
total_no_capi 1694/ 4242 = 39.93%
Raw
z_analyze_per_file.rb
require 'yaml'
KINDS = [:examples, :errors, :failures, :tagged]
LEVELS = 1 # 2
DETAILS = ARGV.delete('--details')
USE_MRI_TOTALS = ARGV.delete('--mri-totals')
HTML = ARGV.delete('--html')
longest_group = "total_no_capi".size
processed = ARGV.map { |results_file|
ruby = results_file[/^results-(.+)\.yml$/, 1] or raise results_file
data = YAML.load_file(results_file)
data = data.transform_keys { |file| file.gsub(%r{^spec/ruby/}, '') }
data = data.transform_keys { |file| file.gsub(%r{^optional/capi/}, 'capi/') }
groups = data.group_by do |file, details|
group = file.split('/')[0, LEVELS].join('/')
if group.end_with?('_spec.rb')
group = file.split('/')[0, LEVELS - 1].join('/')
end
group
end
longest_group = [longest_group, groups.keys.max_by(&:size).size].max
summary = groups.each_pair.map { |group, files|
totals = KINDS.map { |kind|
[kind, files.sum { |file, details| details.fetch(kind) }]
}.to_h
[group, totals]
}.to_h
[ruby, summary]
}.to_h
mri_summary = processed.find { |ruby,| ruby.start_with?('ruby-') }[1]
# Fix up data
processed.each_pair do |ruby, summary|
mri_summary.each_pair do |group,|
unless summary.include?(group)
STDERR.puts "#{ruby} has no #{group} specs"
summary[group] = KINDS.map { |kind| [kind, 0] }.to_h
end
end
summary.each_pair do |group, totals|
if totals[:examples] == 0
STDERR.puts "Correcting examples for #{group} for #{ruby}"
totals[:examples] = mri_summary[group][:examples]
totals[:errors] = totals[:examples]
end
end
end
if USE_MRI_TOTALS
processed.each_pair do |ruby, summary|
summary.each_pair do |group, totals|
mri = mri_summary[group]
diff = mri[:examples] - totals[:examples]
if diff > 0
totals[:examples] += diff
totals[:errors] += diff
elsif diff < 0
raise total.inspect
end
end
end
end
# Compute totals
processed.each_pair do |ruby, summary|
summary["total"] = KINDS.map { |kind| [kind, summary.sum { |_, totals| totals[kind] }] }.to_h
summary_no_capi = summary.reject { |key, _| key == 'capi' or key == 'total' }.to_h
summary["total_no_capi"] = KINDS.map { |kind| [kind, summary_no_capi.sum { |_, totals| totals[kind] }] }.to_h
summary.each_value do |totals|
totals[:passing] = totals[:examples] - totals[:errors] - totals[:failures] - totals[:tagged]
end
end
if HTML
# https://medium.com/@pppped/how-to-code-a-responsive-circular-percentage-chart-with-svg-and-css-3632f8cd7705
circle = -> percents {
text = percents == 100.0 ? "100" : "%.2f" % percents
<<-SVG
<svg viewBox="0 0 36 36" class="circular-chart">
<path class="circle-bg"
d="M18 2.0845
a 15.9155 15.9155 0 0 1 0 31.831
a 15.9155 15.9155 0 0 1 0 -31.831"/>
<path class="circle"
stroke-dasharray="#{percents}, 100"
d="M18 2.0845
a 15.9155 15.9155 0 0 1 0 31.831
a 15.9155 15.9155 0 0 1 0 -31.831"/>
<text x="18" y="20.35" class="percentage">#{text}%</text>
</svg>
SVG
}
puts %q{<table style="width: 100%">}
puts "<colgroup>"
group_width = 16
puts %Q{<col span="1" style="width: #{group_width}%;">}
processed.each_key do
puts %Q{<col span="1" style="width: #{((100 - group_width) / processed.size)}%">}
end
puts "</colgroup>"
puts "<thead>"
puts "<th>Group</th>"
processed.each_key do |ruby|
# ruby_name = ruby.sub(/-/, ' ').sub(/^ruby/, 'cruby').capitalize.sub('ruby', 'Ruby')
ruby_name = ruby.sub(/-/, ' ').sub(/^ruby/, 'cruby').split.first.capitalize.sub('ruby', 'Ruby')
puts %Q{<th style="text-align: center">#{ruby_name}</th>}
end
puts "</thead>"
groups = mri_summary.keys.sort_by { |group|
%w[command_line language core library security total_no_capi capi total].index(group) || raise(group)
}
puts "<tbody>"
groups.each do |group|
# next unless group == 'total'
case group
when 'total'
group_name = '<b>TOTAL</b>'
when 'total_no_capi'
group_name = '<b>TOTAL without C-API specs</b>'
else
group_name = group.gsub('_', '-').capitalize
group_name = group_name.sub('Capi', 'C-API').sub('Library', 'Standard Library').sub('Core', 'Core Library')
link = group == 'capi' ? 'optional/capi' : group
group_name = %Q{<a href="https://github.com/ruby/spec/tree/master/#{link}">#{group_name}</a>}
# group_name += " specs"
end
puts group.start_with?('total') ? '<tr style="border-top: 2px solid black">' : '<tr>'
puts "<td>#{group_name}<br/>\n
#{mri_summary[group][:examples]} specs</td>"
processed.each_pair do |ruby, summary|
totals = summary[group]
ratio = totals[:passing].to_f / totals[:examples]
puts '<td style="text-align: center">'
puts circle[ratio * 100]
puts %Q{<span style="font-size: 95%">#{totals[:passing]} passing</span>} if group.start_with?('total')
puts "</td>"
end
puts "</tr>"
end
puts "</tbody>"
puts "</table>"
else
processed.each_pair do |ruby, summary|
puts nil, ruby
if DETAILS
summary.each_pair do |group, totals|
puts "#{group.ljust(longest_group)} #{totals}"
end
puts
end
summary.each_pair do |group, totals|
f = "%5d"
ratio = totals[:passing].to_f / totals[:examples]
puts "#{group.ljust(longest_group)} #{f % totals[:passing]}/#{f % totals[:examples]} = #{"%6.2f" % (ratio * 100)}%"
end
end
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment