distify.markdown

SCN-App-Distify

A perl distribution builder for ServerCentral perl projects.

To build a module named SCN-Module-Foo ...

cd SCN-Module-Foo
distify SCN-Module-Foo

The result is several additional files, the most important of which is the SCN-Module-Foo-VERSION.tar.gz tarball.

Building this project itself is as easy as

cd SCN-App-Distify
perl script/distify SCN-App-Distify

distify.pl

#!perl

use strict;
use warnings;

use Archive::Tar;
use CPAN::Meta;
use ExtUtils::Manifest qw/mkmanifest maniread/;
use Module::CPANfile;
use Module::Metadata;
use Module::Build::Tiny ();
use Time::Piece;

my ($dist, $path, $module) = @ARGV;

die "distribution name is required\n"
  unless $dist;

my $version = gmtime->strftime('%Y%m%d.%H%M%S');
my $re = $module ? qr/\Q$module\E/ : qr/([^\s;]+)/;
$re = qr/package\s++$re\K\s*;/;

unless ($path) {
  $path = $dist;
  $path =~ s[-][/]g;
  $path = "lib/$path.pm";
}

die "Cannot find file $path\n"
  unless -f $path;

# find main module and inject
{
  my $found = 0;
  local @ARGV = ($path);
  local $^I = '';
  while (<>) {
    if (!$found && s/$re/ $version;/) {
      $module = $1 unless $module;
      $found = 1;
    }
    print;
  }

  die "Main module name was not found in $path\n"
    unless $module;

  die "Module version was not written into $path\n"
    unless $found;
}

my $prereqs = Module::CPANfile->load('cpanfile')->prereq_specs;
$prereqs->{configure}{requires}{'Module::Build::Tiny'} //= $Module::Build::Tiny::VERSION;
my $provides = -d 'lib' ? Module::Metadata->provides(version => 2, dir => 'lib') : {};

my $meta = CPAN::Meta->create({
  name           => $dist,
  author         => ['ServerCentral'],
  version        => $version,
  abstract       => "Generated abstract for $module",
  dynamic_config => 0,
  license        => [ 'restricted' ],
  prereqs        => $prereqs,
  provides       => $provides,
  release_status => 'stable',
});
$meta->save('META.json');

maybe_spurt('Build.PL', <<'FILE');
use Module::Build::Tiny;
Build_PL();
FILE

maybe_spurt('MANIFEST.SKIP', <<'FILE');
#!include_default
^debian/
^local/
^[^/]*\.tar\.gz$
^[^/]*.conf$
FILE

$ExtUtils::Manifest::Quiet = 1;
$ExtUtils::Manifest::Verbose = 0;
mkmanifest();
my $files = maniread();

my $arch = Archive::Tar->new;
$arch->add_files(sort keys %$files);
my $file = "$dist-$version.tar.gz";
$arch->write($file, &Archive::Tar::COMPRESS_GZIP, $dist);

print "$file\n";

sub maybe_spurt {
  my ($file, $text) = @_;
  return if -f $file;
  open my $fh, '>', $file or die "Cannot open $file for writing\n";
  print $fh $text;
}

perldeps.markdown

This simple tool wraps cpm and carton in order to test and install dependent modules quickly (and locally) while still writing a cpanfile.snapshot. It installs the modules declared in the current working directory's cpanfile, it installs modules into ./local/ and writes cpanfile.snapshot to the working directory. This is useful during development to install necessary dependencies while still allowing the later build processes to repeat the build using the snapshot.

It resolves modules from our darkpan and falls back to the cpan metadb. This combination allows installing our local modules while still allowing pinning of modules via the cpanfile all the way back into backpanned modules.

Usage is simply $ perldeps. Any additional arguments are passed to cpm. For example $ perldeps -w 8 to use more concurrency in building.

Note that the cpanfile.snapshot should be committed to the project (and updated as needed) while the local/ directory should not be (and probably should be gitignored).

For the time being (until I figure out how) dependencies that are satisfied will not be updated. In order to update modules (both installed and in the snapshot) either update the cpanfile to require a newer version, install the newer version via another install tool into local/ and then rerun this one to get the snapshot, or simply remove the local directory and reinstall everything.

perldeps.pl

#!/usr/bin/env perl

use strict;
use warnings;

use App::cpm::CLI;
use File::Spec;
use Carton::CPANfile;
use Carton::Snapshot;

my $install_path = 'local';
my $darkpan = 'https://darkpan.servercentral.com/combined';

# call the cli library directly rather than shelling out
# equivalent to running `$ cpm install ...`
App::cpm::CLI->new->run(
  'install',
  '-v',
  '--color',
  '--mirror', $darkpan,
  '-L', $install_path,
  '--exclude-vendor', # undocumented option that is needed for fully self-contained builds
  '-r', "02packages,$darkpan",
  '-r', 'metadb',
  '--test', # run module tests
  # features that carton snapshots by default
  '--with-configure', '--with-build', '--with-runtime', '--with-test', '--with-develop',
  @ARGV, # any additional cpm options can be passed as switches to this script
) && die "Installation failed\n";

my $cpanfile_path = File::Spec->rel2abs('cpanfile');

my $cpanfile = Carton::CPANfile->new(path => $cpanfile_path);
$cpanfile->load;

my $snapshot = Carton::Snapshot->new(path => "$cpanfile_path.snapshot");

$snapshot->find_installs($install_path, $cpanfile->requirements);
$snapshot->save;

These are tools I built at $work to facilitate our CI/CD workflow. CI builds a project into a cpan-style dist with distify. Then CI pushes it into an opan/darkpan. I also needed a module installer that both could use the darkpan with upstream cpan and metadb (forget what the problems were here with cpm/carton) and generate a cpanfile.snapshot.

In our dockerfiles we have:

RUN cpm install -v -r snapshot --mirror=https://darkpan.servercentral.com/combined -g -w 8