ashfurrow

#!/usr/bin/ruby

# Instructions:
# Go to https://app.textexpander.com/settings/export and download all the CSVs into the same directory as this script.
# TextExpander's Export doesn't produce valid CSV files (lol) so make sure to open them with Numbers and re-export them.
require 'csv'

plist = <<~PLIST
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">

ashfurrow

May 12 2018 Intermittent Outage Retrospective

On May 12, for about five hours in the afternoon EDT, mastodon.technology experienced intermittent outages, lasting from twenty seconds to five minutes. The outages were caused in the course of an investigation into a failure to backup the database. The database is fine, and turned out that the backup method itself wasn't working properly.

Timeline

March 10, 2018: I upgrade the instance to Mastodon 2.3.0, from 2.1.x. After the upgrade, I notice Sidekiq keeps segfaulting and dropping jobs. I investigate and open an issue, which is later resolved. However, in the process of investigation, I upgrade my host's Docker version.

May 12, 2018, Around Noon EDT: I notice that local backups are not nearly as large as they should be. The offsite backups on S3 indicate the problem started on the March 11. I reproduce the failure to run pg_dump, which hangs every time. It gets stuck around the same table

ashfurrow

# Sometimes a user will realize they've been accidentally tooting with the wrong privay setting in a specific browswer or app, and they meant their posts to be private. This script does that.

# Hey, uh, admins? Don't do this, unless you *really* trust the user. Changing status privacy after the fact can be abused. See: https://github.com/tootsuite/mastodon/issues/3933

# Open a rails console.
ids = %w( PASTE_IDS_LIST )
statuses = ids.map { |i| Status.find(i) }
statuses.reject { |s| s.uri.include? "USERNAME" } # Check that all the statuses belong to the user
statuses.first.uri # Open in an incognito browser
statuses.first.update_attributes!(visibility: :private) # Test one status change

ashfurrow

--- old	2018-09-08 10:53:00.000000000 -0400
+++ new	2018-09-08 10:53:09.000000000 -0400
@@ -12,8 +12,11 @@
   <li>No xenophobia or violent nationalism.</li>
   <li>No sexual depictions of children.</li>
   <li>No holocaust denialism, no Nazi symbolism, no promotion of National Socialism.</li>
-  <li>No stalking or harassment.</li>
+  <li>No stalking or harassment. In general, if someone asks you to stop interacting with them, then stop.</li>
+  <li>Posting (or threatening to post) other people's personally identifying information ("doxing").</li>
+  <li>Unwelcome sexual attention.</li>

ashfurrow

This is an attempt to document the downtime that occurred on mastodon.technology on August 18, 2018 from roughly 10am to 3pm, Eastern Daylight Time. It's not hyper-accurate. Please contact @ashfurrow for any clarifications.

Ash sincerely apologizes for the downtime.

Timeline

• August 17, 02:44 UTC (10:44PM EDT): the @announcements account posts notice of maintenance window for the following day. The maintenance window was described as two hours long, beginning at 14:00 UTC the next day.
• August 18, beginning at roughly 14:10UTC, mastodon.technology experienced several periods of extended downtime. Site monitoring results are attached in a text file, as well as a screenshot of the site response time graph.
• August 18, roughly 15:45 UTC, mastodon.technology came back online but with severe latency issues. See attached graph screenshot of site latency times. User experience was severely degraded.
• August 18, roughly 19:88 UTC, mastodon.technology came back onlin

ashfurrow

CIImage* ciImage = [[CIImage alloc] initWithData:[drawnImage TIFFRepresentation]];
if ([drawnImage isFlipped])
{
    CGRect cgRect    = [ciImage extent];
    CGAffineTransform transform;
    transform = CGAffineTransformMakeTranslation(0.0,cgRect.size.height);
    transform = CGAffineTransformScale(transform, 1.0, -1.0);
    ciImage   = [ciImage imageByApplyingTransform:transform];
}
CIFilter* filter = [CIFilter filterWithName:@"CIColorInvert"];

ashfurrow

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>DVTConsoleDebuggerInputTextColor</key>
	<string>1 0.721569 0.819608 1</string>
	<key>DVTConsoleDebuggerInputTextFont</key>
	<string>Monaco - 14.0</string>
	<key>DVTConsoleDebuggerOutputTextColor</key>
	<string>0.760784 1 0.87451 1</string>

ashfurrow

#!/bin/sh
df -H | grep '/mnt/volume-nyc1-01' | awk '{ print $5 " " $1 }' | while read output;
do
  echo $output
  usep=$(echo $output | awk '{ print $1}' | cut -d'%' -f1  )
  partition=$(echo $output | awk '{ print $2 }' )
  if [ $usep -ge 80 ]; then
    echo "Running out of space \"$partition ($usep%)\" on $(hostname) as on $(date)"
    curl -s --user "api:$MAILGUN_API_SECRET" \
      https://api.mailgun.net/v3/mastodon.technology/messages \

ashfurrow

The Eidolon Build Process

Since we originally built Eidolon – an auction bidding kiosk app – the project has largely remained in maintenance mode. Eidolon was one of the first projects that we used automated deploys for, and the deploy process has remained largely unchanged. I believe this stability of the deploy process is a testament to how well the automated deploys have gone.

This post is going to detail the mechanics of automated deploys for an enterprise-distributed iOS application, discuss lessons we learned and applied to other projects' deploy processes, and describe some of the changes we'd like to make. Our project is entirely open source, so you can check out any part of the code on your own or open an issue with questions.

Deploying Eidolon

It's one command on the terminal to deploy Eidolon:

ashfurrow

error: <EXPR>:1:8: error: consecutive statements on a line must be separated by ';'
(void*)dlopen("/Applications/Reveal.app/Contents/SharedSupport/iOS-Libraries/libReveal.dylib", 0x2);
       ^
       ;
error: <EXPR>:1:53: error: expected ',' separator
(void)[(NSNotificationCenter*)[NSNotificationCenter defaultCenter] postNotificationName:@"IBARevealRequestStart" object:nil];
                                                    ^
                                                   ,
<EXPR>:1:68: error: expected ',' separator
(void)[(NSNotificationCenter*)[NSNotificationCenter defaultCenter] postNotificationName:@"IBARevealRequestStart" object:nil];