stevenwilliamson

Triton Disaster Recovery

Useful context

According to the docs and from investigation Triton stores all important persistent state in the manatee cluster, which is a cluster of postgresql instances.

The manatee cluster itself depends upon the binder service operating which provides zookeeper and DNS services that manatee makes use of.

Most services store the state in manatee via Moray. Moray provides a key value store API that is backed by manatee. Restoring binder, manatee and moray is the main part of a restore. All other services depend on these services and are almost completely stateless.

Depending on how images are configured in the event a complete headnode loss images will not be restored as they are only stored locally.

bahamat

[root@headnode (barovia) ~]# cat /opt/custom/bin/update-all 
#!/bin/bash

set -o errexit
set -o xtrace

imgadm vacuum -f

sdcadm update --all --just-images -y
sdcadm self-update --allow-major-update --latest

bahamat

No Compute Resources Available

This error is somewhat ambiguious because there is seemingly no indication which resource is exhausted.

Triton chooses compute nodes with a subcomponent called sdc-designation (also refered to as , DAPI). The full DAPI log for a provision job is embedded in the CNAPI log file and can be extracted with the workflow job uuid.

Note: The script below is no longer necessary because cnapi now includes /opt/smartdc/cnapi/bin/alloc-reasons.sh which does the same thing.

#!/bin/bash

bahamat

Use this one instead. https://gist.github.com/bahamat/b1f6511ebdaea510a0f9391c75e2362c

phobos182

#!/usr/bin/env ruby
#
# OpenTSDB TCP handler
# ===
#
# This handler sends metrics to a OpenTSDB server via
# TCP socket.
#
# Compatible checks should generate output in the format:
#   metric.path.one timestamp value host=fqdn\n

deverton

{
    "template": "logstash-*",
    "settings" : {
        "number_of_shards" : 1,
        "number_of_replicas" : 0,
        "index" : {
            "query" : { "default_field" : "@message" },
            "store" : { "compress" : { "stored" : true, "tv": true } }
        }
    },

colonD

#!/bin/sh

# Do elasticsearch optimize on logstash previous day index
# if $1 = all then optimize all indicies

esindex="/opt/elasticsearch/data/elasticsearch/nodes/0/indices"
# Grab yesterday's values
D=`date +%d -d yesterday`
M=`date +%m -d yesterday`
Y=`date +%Y -d yesterday`