SDPS Enduro + Archivematica

readme.md

SDPS Enduro + Archivematica

Instructions to set up and configure the Enduro SDPS Kubernetes cluster and the Archivematica Docker Compose environment locally.

Archivematica

Follow the documentation to set up a local instance of Archivematica:

https://github.com/artefactual/archivematica/tree/qa/1.x/hack

SSH KEY

If you don't have one, create an Ed25519 SSH key. E.g.:

ssh-keygen -t ed25519 -C "whatever"

This key will be used to authenticate between Enduro and the SFTP server, so you can delete it from your system after configuring the environments.

SFTP service

We'll use sftpgo to set up an SFTP server in the AM Docker Compose env. Modify AM's Docker Compose file to include a new internal volume and the sftpgo service, mounting that volume in the archivematica-storage-service and the sftpgo services:

diff --git a/hack/docker-compose.yml b/hack/docker-compose.yml
index ddad624c..6c9db96c 100644
--- a/hack/docker-compose.yml
+++ b/hack/docker-compose.yml
@@ -9,6 +9,7 @@ volumes:
   mysql_data:
   elasticsearch_data:
   archivematica_storage_service_staging_data:
+  sftp_dir:
 
   # External named volumes.
   # These are intended to be accessible beyond the docker host (e.g. via NFS).
@@ -231,5 +232,17 @@ services:
       - "archivematica_pipeline_data:/var/archivematica/sharedDirectory:rw"
       - "archivematica_storage_service_staging_data:/var/archivematica/storage_service:rw"
       - "archivematica_storage_service_location_data:/home:rw"
+      - sftp_dir:/home/enduro_transfers
     links:
       - "mysql"
+
+  sftpgo:
+    image: drakkan/sftpgo:v2.5.5-alpine-slim
+    environment:
+      SFTPGO_LOADDATA_FROM: /etc/sftpgo/initial-data.json
+    ports:
+      - 12380:8080
+      - 12322:2022
+    volumes:
+      - ./sftpgo-data.json:/etc/sftpgo/initial-data.json
+      - sftp_dir:/home/enduro_transfers

Create the sftpgo-data.json file used for the other volume in the sftp service, alongside the docker-compose.yml file:

{
  "users": [
    {
      "id": 1,
      "status": 1,
      "username": "archivematica",
      "password": "$2a$10$3Y.Qp.8BlNLCI6t0yKnbJuzLcKBgukuQl4Zg9C1jA6D0Xr3FhDoTu",
      "public_keys": [
        "ssh-ed25519 ... whatever"
      ],
      "has_password": true,
      "home_dir": "/home",
      "permissions": { "/": ["*"] }
    }
  ],
  "admins": [
    {
      "id": 1,
      "status": 1,
      "username": "admin",
      "password": "$2a$10$3OttaDDy1VIJbMTu80zNP.jx5UQQw2gwtN4QZXIcBIW8870j4koDi",
      "permissions": ["*"]
    }
  ]
}

Replace the public key placeholder with the Ed25519 public key. And start the volume and service:

docker compose up -d

Enduro

Follow the documentation to set up the Enduro Kubernetes cluster:

https://github.com/artefactual-sdps/enduro/blob/main/docs/src/dev-manual/devel.md

Make sure you set am as the preservation system, and check the documentation to work with Archivematica:

https://github.com/artefactual-sdps/enduro/blob/main/docs/src/dev-manual/archivematica.md

We need to create three files inside the repository:

hack/kube/overlays/dev-am/.am.secret

address=http://host.k3d.internal:62080
user=test
api_key=test
sftp_host=host.k3d.internal
sftp_port=12322
sftp_user=archivematica
sftp_remote_dir=/enduro_transfers
sftp_private_key_passphrase=
amss_url=http://host.k3d.internal:62081
amss_user=test
amss_api_key=test
amss_location_id=e0ed8b2a-8ae2-4546-b5d8-f0090919df04

We are using K3d's internal host to connect with the host from within the cluster. If you use Minikube, change that to host.minikube.internal.

hack/kube/overlays/dev-am/.id_ed25519.secret

-----BEGIN OPENSSH PRIVATE KEY-----
...
-----END OPENSSH PRIVATE KEY-----

Copy the Ed25519 private key here.

hack/kube/overlays/dev-am/.known_hosts.secret

To get the SFTP server keys and create the known_hosts file we'll first add an entry to /etc/hosts to match the host name used inside the cluster:

127.0.0.1 host.k3d.internal

And use ssh-keyscan to generate the secret file directly in the repository, make sure to set the right path to you local Enduro folder:

ssh-keyscan -H -p 12322 host.k3d.internal > /path/to/enduro/hack/kube/overlays/dev-am/.known_hosts.secret

Finally, in Enduro's Tilt UI, restart the (Tiltfile) and the enduro-am resources, in that order.

Warning Each time you recreate the sftpgo service in AM you'll need to regenerate the .known_hosts.secret file and restart Enduro's resources.