Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?
package Test::TAP::Unify;
use strict;
use warnings;
use Carp;
our $VERSION = '0.01';
=head1 NAME
Test::TAP::Unify - Create a minimal set of TODO code for a set of test results
rm log.test-*
os=`uname -o`
for p in 5.10.1 5.10.1-nt 5.12.1 5.12.1d 5.12.1d-nt 5.13.3 5.13.3d 5.13.3-nt
perl$p Makefile.PL
make test >log.test-$os-$p
use Test::TAP::Unify;
# take features from test filenames
my $features = {debugging => qr/\dd/,
threads => qr/-nt/,
os => qr/log.test-(\w+)-/,
version => qr/(5\.[\d\.]+/)};
my $t = Test::TAP::Unify->new(filename => $features);
my $u = $t->parse(<log.test-*>);
# prints the complete test matrix
print $u->matrix;
# prints @TODO lists as code snippet
print $u->todo->code('PUSH', 'index'); # as PUSH'ed lists of test indices
# take features from test filenames and test comments also (the module name)
my $m = Test::TAP::Unify->new
( filenames => {debugging => qr/\dd/,
threads => qr/-nt/,
os => qr/log.modules-(\w+)-/,
version => qr/(5\.[\d\.]+/)},
comment => {module => qr/(pass|fail) (\S+)/})
print $m->matrix; # all results
print $m->todo->code('IF', 'module'); # as IF checked lists for module lists
You hate failing tests, you get a set of test results per perl version
by yourself, and also gathered by cpantesters. So you can mark all failing
tests as TODO check tickets, and release a new version, where all tests pass.
Great! Error free software.
Really, for more complicated XS modules with some randomly failing tests
you need to keep TODO lists uptodate. Just to see regressions.
This modules was written to unify test results of the perl compiler L<B::C>, you
will get tired to mark TODO tests manually after each feature change or added
platform or perl version you are testing against. It will give you insights
which feature might cause the damage. You can e.g. mark tests with any feature name
and check them with this module.
Test::TAP::Unify reads the output of make test (TAP results), detects features
by various methods, e.g. the filename where make test is logged to, or the test comments
and tries to find the minmal set of failing tests per feature. The shortest set of
feature-fail combination. Since you have an arbitrary long list of features -
typically: version, threads, os, debugging, cc - producing this set is not
The sets can be printed as matrix, or as code, lists of @TODO with added feature logic.
Each test is indexed by number 1..plan.
Sample output for todo->code('PUSH', 'index')
sub is_todo {
my @TODO;
@TODO = (15,39,44);
@TODO = (39) if !$ITHREADS;
@TODO = (15,41..45) if $version < 5.007;
@TODO = (39,41,44) if $version >= 5.010;
@TODO = (15,29,39,44) if $version >= 5.010 and !$ITHREADS;
push @TODO, (27) if $version >= 5.012 and $ITHREADS;
push @TODO, (6,8..10,16,21,23,24,26,30,31,35) if $version >= 5.013002;
push @TODO, (15,25,42..43) if $version >= 5.013 and $ITHREADS;
return @TODO;
Sample output for todo->code('IF', 'module')
sub is_todo {
my ($module, $version, $name, $threads) = @_;
foreach (qw(ExtUtils::MakeMaker LWP Attribute::Handlers MooseX::Types)) {
return 'generally' if $_ eq $module;
if ($] < 5.007) {
foreach(qw( ExtUtils::CBuilder Sub::Name)) {
return '< 5.007' if $_ eq $module;
if ($] < 5.010) {
foreach(qw(B::Hooks::EndOfScope)) {
return '< 5.010' if $_ eq $module;
if ($threads) {
File::Temp ExtUtils::Install
Test::Tester Attribute::Handlers
Test::Deep FCGI B::Hooks::EndOfScope Digest::SHA1
namespace::clean DateTime::Locale DateTime
)) {
return 'with threads' if $_ eq $module;
if ($] >= 5.010 and $] < 5.012) {
Class::Accessor Class::MOP
)) {
return '5.10 with threads' if $_ eq $module;
} else {
if ($] >= 5.010 and $] < 5.012) {
IO ExtUtils::Install Test::Tester Test::Deep Path::Class
)) {
return '5.10 without threads' if $_ eq $module;
=head1 SEE ALSO
=head1 AUTHOR
Reini Urban E<lt>rurban@cpan.atE<gt>
Copyright (C) 2010 by Reini Urban
This library is free software; you can redistribute it and/or modify
it under the same terms as Perl itself, either Perl version 5.10.1 or,
at your option, any later version of Perl 5 you may have available.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment