lusis

Random idea

I'm a big fan of the Ops School idea. I've struggled for years about how to "train up" someone coming into this field. So much of our skill set is forged in the fire of outages and troubleshooting.

One of the things that is both good and bad about system administration and operations is that we tend to see patterns. It's good in that we immediately see things that stand out. The downside is we tend to superimpose that pattern recognition in inappropriate ways.

We had an interesting issue happen yesterday at the dayjob. I won't go into exactly what it was here but I had an idea based on some graphs I was looking at. It's part social experiment but it's also part problem solving skills.

Given the following image with no context, what do you think happened? What are some of the key indicator points that jump out and what pattern do they call to mind?

_(Since it may not be clear simply from an image resolution perspective, there are 4 me

lusis

syntax = "proto3"
package = "com.myorg.ptypes";

message User {
  int64 id = 1;
  UserStatus status = 2;
}

message Group {
  int64 id = 1;

lusis

This is a copy/paste from an internal wiki on how we should use docker

This guide is to serve as an outline of internal best practices to using Docker. The idea is to give enough information to allow engineers to create containers for new stack components while minimizing the cleanup required to make them production ready.

Concepts

Before we get to the practices, it's important to understand WHY we are doing this. The key concepts we're concerned about with broad Docker usage are provenance, determinism, repeatability and auditability.

Provenance

Provenance refers to knowing WHERE something comes from. Generally with the end use of software, this is easy:

• Maven artifacts are pulled from central

lusis

Current metrics at startup

root@node1:~# curl -s http://localhost:9569/metrics  | grep peer
# HELP peer_route_state shows the current peer state for a given route
# TYPE peer_route_state gauge
peer_route_state{family="ipv4 unicast",peer_asn="64496",peer_ip="192.168.1.2",route="192.168.88.2/32",self_asn="64496",self_ip="192.168.1.184"} 1
# HELP peer_state shows the current peer state
# TYPE peer_state gauge
peer_state{peer_asn="64496",peer_ip="192.168.1.2",self_asn="64496",self_ip="192.168.1.184"} 1

lusis

# place in /etc/yum/pluginconf.d/artifactory.conf
[main]
enabled=1

lusis

<!-- the env variables are controlled by Chef and passed in via -D on the java command-line -->
<!-- This is using the appender here: https://github.com/t0xa/gelfj -->
<appender name="graylog2" class="org.graylog2.log.GelfAppender">
    <param name="graylogHost" value="${graylog.server}"/>
    <param name="originHost" value="${graylog.origin}"/>
    <param name="extractStacktrace" value="true"/>
    <param name="addExtendedInformation" value="true"/>
    <!-- The _web part is because a given app has multiple components -->
    <!-- This app might have a _web as well as an _batch component -->
    <param name="facility" value="${graylog.facility}_web"/>

lusis

For a moment, I'm going to throw away my automation and configuration management hat. I'll let you know when I put it back on. Also, let's ignore that we're talking about Riak specifically for a moment. Also also, let's ignore any (for a brief moment) the proper role of a package manager.

What's your target market?

If you're writing server software, you have two target markets. The system administrator/operations team and the developer.

Why do you want autostart?

The main reason you want autostart is to get people up and running quickly. To do this, you need to ship safe and sane defaults. This means something like a default configuration that listens only on localhost.

lusis

Grab the jar

http://semicomplete.com/files/logstash/logstash-1.1.0beta6-monolithic.jar

start the sender

java -jar logstash-1.1.0beta6-monolithic.jar agent -c logstash-sender.conf

start the receiver

java -jar logstash-1.1.0beta6-monolithic.jar agent -c logstash-receiver.conf

In sender terminal

lusis

Warning: Like the shitty OSS person I am, I forgot to update this. This particular iteration is exploitable in a really basic way that I missed. I will update this with the updated version hopefully soon

Reasoning

While authing against our Google Apps domain has worked pretty well up until now, we really needed a way to auth against out Github organization. Not everyone who is accessing some of our protected development content has an email account in our Google Apps domain. They do, however, have access to our github org.

Sadly it seems that apache and nginx modules for doing oauth are lacking.

I was hoping to avoid the whole lua approach (and mod_authnz_external was a no go from the start). However I realized that Brian Akins (@bakins) had done some fancy omnibus work that got me 90% of the way there.

From there it was a matter of patching up the omnibus repo to bring it to current versions as well as adding in a few additional components.

lusis

print("hello from a python script")