hashbrowncipher

I had a bad brush with the single version rule at work, and I’d like to warn you so that you can avoid the pain it caused me.

The best thing about monorepos

The best thing about monorepos is that they require the author of any given change to make the entire repository’s tests pass before committing that change. This puts the onus of managing incompatible changes on the original author, which:

1. Centralizes the responsibility of solving incompatibilities onto the person or team with the most knowledge of the change.
2. (In the best case) exposes any incorrect assumptions they may have about their change early-on in the process.

Largely as a result of the above property, monorepos obviate most of the overhead around dependency versioning. When dependencies are consumed at HEAD, it is no longer necessary to maintain separate artifact versioning, and the build and release pipelines for intermediate artifacts (e.g. base images, lib

hashbrowncipher

tl;dr: If you run apt-get install reproducibly, there's no reason to use "golden images".

Background (about me)

I've now run into the "how do we manage and update a base operating system" problem at three different roles over the course of many years. At each role, my colleagues and I landed on a reproducible installations using apt . Fair warning: the rest of this post will be apt flavored, although I hope that the general lessons will be useful for any operating system.

The most advanced incarnation of this system used Bazel as part of a monorepo to provide automated upgrades for hundreds of individual services. In that incarnation, it was possible to build new container images that remediated a given CVE org-wide in a single commit, which was very useful for me as a security engineer.

What are golden images?

hashbrowncipher

One of my favorite talks from fwd:CloudSec this year was Jarom Brown's AWS Presigned URLs: The Good, The Bad, and the Ugly (abstract). In his talk, Jarom demonstrates a way to use javascript within an unsuspecting user's browser to make requests to AWS. Because most AWS services provide their APIs over HTTP, he was able to use standard browser-side facilities (e.g. XMLHTTPRequest, fetch()) to synthesize requests to AWS, and then exfiltrate their responses to a server he controlled. This is most relevant in a post-compromise context, where an attacker has access to a victim's AWS keys, and gives the attacker lots of leeway for disguising or changing the source IP address presented to AWS.

The first question on my mind was "I thought CORS was supposed to stop this. Why didn't it?". It looks like the answer is that most AWS services are very happy to send an Access-Control-Allow-Origin: * header. For

hashbrowncipher

But actually, how do flamegraphs work?

Q: So when I make a flamegraph, how does that work?

The specific type of flamegraph you're seeing is an on-cpu flamegraph. It is synthesized by periodically stopping the CPU, asking "what are you doing right now", and storing the answer. The flamegraph itself is a convenient representation of the data produced by doing that very frequently (see the -F argument to perf record) over some period of time.

hashbrowncipher

One time at work, my team was upgrading an open source search-engine-cum-database that had an unfortunate predilection for breaking its external API. We had already deployed the new version of the database with its breaking changes, and now it was time to herd our customers off of the old version and onto the new version. Our customers were naturally reticent: for most of them it was just a bunch of work for very little reward. The migration would require careful testing, and just generally it didn't sound like a fun time. To top the situation off, some of these customers' services hadn't been touched in years, and the original authors had long since left.

I'm proud to say that my team was significantly more interested in accommodating our customers' needs than some other DBA teams I've worked with or around. During the migration we spent a fair bit of time chewing on ways to lessen the burden we placed on our customers. At one point the possibility of simply "handing off" the outdated search engines was dis

hashbrowncipher

tl;dr:

• don't define security tiers; use security cells instead
• each service should have its own security cell

One time at JOB_1, we had a problem: we were about to start serving a dataset that was significantly more sensitive than our typical dataset. Leaking it would likely produce scary consequences in the real world, with ripple effects beyond just the company and its shareholders. We hadn't dealt with issues of that kind before; and as engineers on the project, my colleagues and I felt duty-bound to find the "right" solution. We started looking for other places where high security data was stored, and we found one owned by another team in the org. Perfect! We would store our dataset alongside the existing dataset, and they'd both be safe together in the protective cocoon of the high security environment.

This didn't work. The folks who owned the existing dataset were rightly distrustful of sharing their meticulously constructed environment with our team. And for good reason: adding our application t

hashbrowncipher

The %util column in iostat has always been something of a weird bird. It was created back in the day when hard disks could only effectively process a single request at once. Back then, if %util was 100, then the disk was completely saturated and could go no faster. Modern block devices have the ability to perform multiple operations concurrently, and so %util's usefulness has ebbed. The manpage gives a clear warning about the meaning (or lack thereof) of %util, describing it as:

Percentage of elapsed time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100% for devices serving requests serially. But for devices serving requests in parallel, such as RAID arrays and modern SSDs, this number does not reflect their performance limits.

hashbrowncipher

//Invoke like `./bin <frequency in Hz (e.g. 100)> <filename to read>`

#define _GNU_SOURCE
#include <fcntl.h>
#include <stdbool.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <time.h>
#include <unistd.h>

hashbrowncipher

#!/usr/bin/env python3
"""Cross platform, rootless, and fast debootstrap.

Designed to work anywhere you can run Python and Docker/Podman (e.g. a Mac laptop).

* Tested only with Ubuntu Focal and Jammy
* Right now LZMA decoding takes up most of the time. Parallelize it? Python's LZMA
  library does release the GIL.

"""

hashbrowncipher

#!/bin/bash
# Copyright 2021 Josh Snyder
# See license and documentation in lambda_function.py
set -x -o errexit -o nounset

ebs_name() { 
  nvme id-ctrl -o binary $1  | cut -c3073-3104 | tr -d ' '
}

export AWS_DEFAULT_REGION=$AWS_REGION