alicefuzier

Pipeline merging

What should the pipeline be doing?

A few cases to remind ourselves what the behaviour of the whole pipeline should be when merging things:

• A:1 -> B:1 if A:1 is the canonical results in:

  • A:1 merged with B:1
  • B:1 redirected to A:1

• A:1 -> B:1 -> C:1 if A:1 is the canonical results in:

alicefuzier

import json
from elasticsearch import Elasticsearch

es = Elasticsearch(['https://user:password@host:443/'])

page = es.search(
  index = 'indexname',
  doc_type = 'work',
  scroll = '2m',
  size = 1000,

alicefuzier

import boto3
from tenacity import retry

client = boto3.client('s3')
paginator = client.get_paginator('list_objects_v2')

@retry
def delete_objects(client, objects):
    response = client.delete_objects(Bucket='wellcomecollection-tmp-wiretap',Delete=objects)
    errors = response.get('Errors')

alicefuzier

import boto3
from multiprocessing import Pool
from tenacity import retry
from tenacity.wait import wait_random
from tqdm import tqdm


@retry(wait=wait_random(min=1, max=300))
def update_data(table_name, id):
    try:

alicefuzier

Keybase proof

I hereby claim:

• I am alicefuzier on github.
• I am alicefuzier (https://keybase.io/alicefuzier) on keybase.
• I have a public key whose fingerprint is 9622 7CE9 6B5F 4431 A8F4 83D1 E40F D96E 9A49 1B2D

To claim this, I am signing this object:

alicefuzier

import akka.actor.ActorSystem
import akka.stream.ActorMaterializer
import akka.stream.scaladsl._
import scala.concurrent.Future
import akka.stream._
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global

implicit val system = ActorSystem("MyActorSystem")
implicit val materializer = ActorMaterializer()

alicefuzier

import cats.syntax.either._
import io.circe._
import io.circe.optics.JsonPath._
import io.circe.parser._

val json = parse(s"""
      {
        "items": [
          {
            "identifiers": [