stevenwilliamson

Triton Disaster Recovery

Useful context

According to the docs and from investigation Triton stores all important persistent state in the manatee cluster, which is a cluster of postgresql instances.

The manatee cluster itself depends upon the binder service operating which provides zookeeper and DNS services that manatee makes use of.

Most services store the state in manatee via Moray. Moray provides a key value store API that is backed by manatee. Restoring binder, manatee and moray is the main part of a restore. All other services depend on these services and are almost completely stateless.

Depending on how images are configured in the event a complete headnode loss images will not be restored as they are only stored locally.

stevenwilliamson

root@d424fe5583e1:/# gpgv --keyring /etc/apt/trusted.gpg.d/debian-archive-buster-security-automatic.gpg --keyring /etc/apt/trusted.gpg.d/debian-archive-stretch-security-automatic.gpg InRelease
gpgv: Signature made Mon Jul 13 11:26:13 2020 UTC
gpgv:                using RSA key 379483D8B60160B155B372DDAA8E81B4331F7F50
gpgv: BAD signature from "Debian Security Archive Automatic Signing Key (9/stretch) <ftpmaster@debian.org>"

stevenwilliamson

root@d424fe5583e1:/# apt -o Debug::Acquire::gpgv=true update
Get:1 http://deb.debian.org/debian buster InRelease [121 kB]
Get:2 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
0% [1 InRelease 121 kB/121 kB 100%]inside VerifyGetSigners
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
0% [Working]Preparing to exec:  /usr/bin/apt-key --quiet --readonly verify --status-fd 3 /tmp/apt.sig.4gsSlo /tmp/apt.data.ZNKjLd
gpgv exited with status 1
Summary:
  Good:
  Valid:

stevenwilliamson

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Origin: Debian
Label: Debian-Security
Suite: stable
Version: 10
Codename: buster
Date: Mon, 13 Jul 2020 11:26:12 UTC
Valid-Until: Mon, 20 Jul 2020 11:26:12 UTC

stevenwilliamson

root@6e7d433a2c61:/# apt-get update
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://deb.debian.org/debian buster InRelease [121 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Err:1 http://security.debian.org/debian-security buster/updates InRelease
  At least one invalid signature was encountered.
Err:2 http://deb.debian.org/debian buster InRelease
  At least one invalid signature was encountered.
Err:3 http://deb.debian.org/debian buster-updates InRelease
  At least one invalid signature was encountered.

stevenwilliamson

Keybase proof

I hereby claim:

• I am stevenwilliamson on github.
• I am stevenwilliamson (https://keybase.io/stevenwilliamson) on keybase.
• I have a public key ASDjXoRS99PTXAQrjcvsjTDFMAcSWfRmKt2VK_JKN98G6go

To claim this, I am signing this object:

stevenwilliamson

--<root@2016q4-builder>-(/data/chroot/dev-2016Q4-x86_64)-</data/pkgsrc/lang/erlang>--
-> bmake show-options
Any of the following general options may be selected:
        dtrace   Enable DTrace support.
        erlang-hipe
        java     Enable Java support.
At most one of the following odbc options may be selected:
        iodbc    Enable ODBC support via iODBC.
        unixodbc         Enable ODBC support via unixODBC.

stevenwilliamson

#!/bin/bash

cd /data/pkgsrc
/opt/local/bin/git remote set-url origin git://github.com/joyent/pkgsrc-legacy.git && git pull
/opt/local/bin/git clone -b 2016Q4 http://github.com/fac/pkgsrc-fa.git fa
cd /data/pkgsrc/fa
/opt/local/bin/git remote rename origin http
/opt/local/bin/git remote add origin git@github.com/fac/pkgsrc-fa

stevenwilliamson

#!/bin/bash

set -uo pipefail

ZONE_NAME=2016Q4-builder
IMAGE=0d649af0-e6ed-11e6-8689-7fb3356bad96
PACKAGE=r4-d30-c4
ZONE_EXIST=false
ZONE_IP=$(triton instance ip $ZONE_NAME 2>/dev/null)

stevenwilliamson

Tested on image "e74a9cd0-f2d0-11e6-8b69-b3acf2ef87f7"

# Create /etc/X11/xorg.conf

```
Section "Monitor"
    Identifier             "Monitor0"
EndSection