zmc

import logging
import os

logging.basicConfig()
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)

def get_file_contents(path, default=''):
    contents = default
    if not os.path.exists(path):

zmc

A few notes on the RHEVpocalypse

This document is very far from exhaustive; I just wanted to write a few things down before an extended PTO.

Things that were easier than expected

Rebuilding certain services in OpenShift without access to original deployments

• This wouldn't have been feasible if David hadn't deployed it as a POC.
• It will be even easier "next time" if we build a strong habit of keeping services and their configuration in version control.

Making the OpenShift cluster healthy (it wasn't very fuctional when the outage hit)

zmc

---
apiVersion: v1
kind: Namespace
metadata:
  name: fio-test

zmc

go_gc_duration_seconds{quantile="0"}
go_gc_duration_seconds{quantile="0.25"}
go_gc_duration_seconds{quantile="0.5"}
go_gc_duration_seconds{quantile="0.75"}
go_gc_duration_seconds{quantile="1"}
go_gc_duration_seconds_sum
go_gc_duration_seconds_count
go_goroutines
go_info{version="go1.10.1"}
go_memstats_alloc_bytes

zmc

Debugging gevent hangs with manhole

If you have a hung process, find its PID and skip this. Otherwise, let's create a dummy hang:

$ cd teuthology
$ curl -O https://gist.githubusercontent.com/zmc/55b697822f332cadf297dfb98b2dcecd/raw/46efb4276f3e98224ae725e8f27c4fb2047c35f8/gevent_debug.py
$ source ./virtualenv/bin/activate
$ python
>>> import os; print os.getpid()

zmc

import gc
import gevent
import traceback


def find_greenlets():
    greenlets = list()
    objects = gc.get_objects()
    for obj in objects:
        if isinstance(obj, gevent.greenlet.greenlet):

zmc

16:20:54 zack@teuthology ceph master $ cat /tmp/old.sh
#!/bin/sh -x
set -e


# Reformat version if needed to match RPM version and release
if expr index $(git describe --always) '-' > /dev/null ; then
    desc=$(git describe --always | sed 's/^v//')
    RPM_VER=$(echo $desc | cut -d'-' -f1)

zmc

TASK: [ceph-osd | debug combined_parted_results.results] ********************** 
ok: [target078012.ovh.sepia.ceph.com] => {
    "var": {
        "combined_parted_results.results": [
            {
                "changed": false,
                "cmd": "parted --script /dev/vda print | egrep -sq '^ 1.*ceph'",
                "delta": "0:00:00.011337",
                "end": "2016-02-22 16:36:08.487354",
                "failed": false,

zmc

Introduction

We recently set up a new lab for Ceph testing and decided to document the parts of the process that are most relevant to teuthology. This is the result.

We started by setting aside two of the test machines: one as the 'teuthology node', and another as the 'paddles/pulpito node'. These would be used to orchestrate automated testing and to store and serve the results on our intranet.

paddles/pulpito node

zmc

#!/bin/bash

# A simple script used by Red Hat to start teuthology-worker processes.

ARCHIVE=$HOME/archive
WORKER_LOGS=$ARCHIVE/worker_logs

function start_workers_for_tube {
    echo "Starting $2 workers for $1"
    for i in `seq 1 $2`