chizmw/pinto2opan.md

## pinto2opan.md

      
    Raw
  

              pinto2opan.md
            
          
    Overview

I'm currently using Pinto at $orkplace.
Pinto is becoming increasingly painful to use.
More and more often we're seeing:
The repository is currently in use -- please try again later (Timed out waiting for blocking lock)

There's added fun when Pinto fails because backpan is down.
Speaking to MST on IRC it looks like opan might be a production suitable alternative.
OPAN

Dockerfile

FROM perl:5.28.2

RUN cpanm App::opan

RUN apt-get update && apt-get install -y apt-utils tree

WORKDIR /tmp

RUN useradd --create-home --user-group opan

USER        opan
WORKDIR     /home/opan

RUN opan init
RUN opan fetch

ENV OPAN_AUTH_TOKENS        something987654
ENV OPAN_RECURRING_PULL     1

ENTRYPOINT  ["opan"]
CMD ["prefork", "-l", "http://0.0.0.0:8030"]

docker-compose.yml

version: "3.7"
services:
    opan-server:
        build:              .
        image:              docker.zoopla.com/zpg-opan
        environment:
            - OPAN_AUTH_TOKENS=something987654
        ports:
            - 8030:8030
        volumes:
            - "pans:/home/opan/pans"
    opan-tree:
        image:              docker.zoopla.com/zpg-opan
        entrypoint:         ["tree"]
        command:            ["/home/opan/pans"]
        volumes:
            - "pans:/home/opan/pans"

volumes:
    pans:

Extracting from Pinto

I'm lucky; we backup our Pinto data to an S3 bucket.
I only wanted to seed with the most recent version of each (internal) module.
bin/find-latest-of-each-dist

This is a quick script I wrote to point at my directory of dist tarballs to give me a list of the latest version of each thing there:
#!/usr/bin/env perl
use strict;
use warnings;

use Cwd 'abs_path';
use Dist::Metadata;
use File::Find::Rule;
use version;

my $dir = $ARGV[0] || die "need a directory to scan\n";

my $abs_dir = abs_path($dir) || die "$dir: can't resolve to absolute path\n";

if ( ! -d $abs_dir ) {
    die "$abs_dir: directory not found\n";
}

warn "Processing $abs_dir...\n";

my @files = File::Find::Rule->file()
                            ->name( '*.tar.gz' )
                            ->in( $abs_dir );

my $results;

my $count = 0;

foreach my $file (@files) {
    $count++;
    my $dist = Dist::Metadata->new(file => $file);
    my $description = sprintf "Dist %s (%s)", $dist->name, $dist->version;

    # something new
    if (!exists $results->{$dist->name}) {
        warn "first time:     adding: $description\n";
        $results->{$dist->name} = $dist;
    }
    # compare with existing
    else {
        my $stored_dist = $results->{$dist->name};

        # what we have stored is older than the current tarball
        if ( version->parse($stored_dist->version) < version->parse($dist->version) ) {
            warn 'found newer:   stored version (' . $stored_dist->version . ') older than ' . $description . "\n";
            $results->{$dist->name} = $dist;
        }

        # we're already storing the newer of the two
        else {
            warn 'ignoring:      ' . $description . "\n";
        }
    }

    #last if $count > 25;
}

foreach my $dist (sort keys %{$results}) {
    say $results->{$dist}->{file};
}

bin/seed-data

#!/bin/bash
set -euo pipefail

function opan-upload() {
    curl -H 'Authorization:Basic base64zzzauthzzzstringssazzxx==' -F "dist=@${1}" https://opan.wherever:8030/upload;
}

aws s3 sync s3://the-s3-bucket/authors/id/Z/ZP/          pinto-authors-id/

# find the latest version of each normal ZPG Thing
# it's a bit slow, but shouldn't need running very often
# we'll only run it if we don't have the file already
if [ ! -f tarball.list ]; then
    bin/find-latest-of-each-dist pinto-authors-id/ZPGPACKAGE/ |tee tarball.list
fi

# we can then loop through them and add them
for F in $(cat tarball.list); do
    opan-upload $F
done