Skip to content

Instantly share code, notes, and snippets.

@fasterthanlime

fasterthanlime/butler-tasks-as-json.md

Last active October 1, 2017 17:30
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save fasterthanlime/ba900a01f6b6799621f3bab88905cd46 to your computer and use it in GitHub Desktop.
Save fasterthanlime/ba900a01f6b6799621f3bab88905cd46 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
butler-tasks-as-json.md

JSON output

Ever since butler has become responsible for downloading, patching, unzipping in the itch app, it has had a JSON-lines output mode.

If you pass --json (or -j) — which the app always does — it'll output lines like this:

{"level":"info","message":"Resuming at 46 MiB / 286 MiB","type":"log"}
{"bps":0,"eta":630,"percentage":16.112211547817594,"progress":0.16112211547817595,"type":"progress"}
{"bps":0,"eta":303,"percentage":16.374732306234176,"progress":0.16374732306234174,"type":"progress"}

The app parses them like this for example (here, token is a single line of output from butler):

  let status: any;
  try {
    status = JSON.parse(token);
  } catch (err) {
    logger.warn(`Couldn't parse line of butler output: ${token}`);
    return;
  }

  switch (status.type) {
    case "log": {
      if (!showDebug && status.level === "debug") {
        return;
      }
      return logger.info(`butler: ${status.message}`);
    }
    case "progress": {
      return ctx.emitProgress(status as IProgressInfo);
    }
    case "error": {
      logger.error(`butler error: ${status.message}`);
      return onError(new Error(status.message));
    }
    case "result": {
      onResult(status);
      return;
    }
    default:
    // muffin
  }

Passing parameters

Up until now, every parameter to a butler command was passed as command-line arguments.

Here's how the app calls apply for example:

/* Apply a wharf patch at ${patchPath} in-place into ${outPath}, while checking with ${signaturePath} */
async function apply(opts: IApplyOpts) {
  const { patchPath, outPath, signaturePath } = opts;
  let args = [patchPath, "--inplace", outPath, "--signature", signaturePath];

  if (opts.archivePath) {
    args = [...args, "--heal", `archive,${opts.archivePath}`];
  }

  return await butler(opts, "apply", args);
}

This is cool as long as each parameter is a string (or a number), but I'm now thinking of moving other, high-level operations to butler, such as:

  • Installing a game, which involves
    • downloading, extracting (possibly in parallel)
    • maybe running an installer with certain command-line arguments
    • maybe copying a bunch of files, writing a list of them somewhere
    • etc.
  • Upgrading a game, which involves
    • downloading+applying a bunch of patches in sequence
    • downloading the latest version & extracting it, updating the list of installed files
  • Launching a game, which involves
    • parsing the manifest, if any, asking the user which manifest action they'd like to pick
    • setting up everything for the itch.io sandbox (the Linux, macOS & Windows one all work differently)

And so on. The problem with these tasks is that:

  • They need a lot more parameters
    • Some strings: API key, download key, etc.
    • Some definitely not strings: game API object, upload API object, build API object
  • They also need to prompt the user to make a choice
    • For example, a permissions dialog if we have an installer that requires admin right
    • or, which manifest action to launch for a game
    • or, if there's multiple uploads, which upload to install for a game

So command-line parameters and JSON output don't cut it anymore.

Is that reasonable?

So the idea is to group these higher-level tasks (install, check-update, upgrade, uninstall, launch etc.) into a single command and have it run "task specifications" in JSON from stdin - so the JSON-lines communication would go both ways.

To install a game for example:

itch: Hi! Please install this {game}, with this {API key} and this {download key} to {this install location}, you can use {this staging folder} for temporary files
butler: Ok! There's 3 uploads that are platform-compatible, which one does the user want?
itch: The second one
butler: Ok! I've started a download task, I'll send progress updates
butler: 30% done
butler: 80% done
butler: I'm done with the download task
butler: It's an installer that requires admin privileges, is the user ok with that?
itch: Yes they are!
butler: Ok! I'm starting an install task
butler: 40% done
butler: 90% done
butler: It's all good! Here's a {list of installed files} and everything else you need to know about this game
*butler exits stage left*

Each 'request' and 'response' in that conversation corresponds to a JSON object with a given structure and type field.

In my current version, the command is named cave (that's how "installed games" are called in the app), and the first thing it does is try and read a JSON line of type cave-command that contains all the info we need.

It then validates that it has everything it needs, then does the actual work.

butler is in charge of the control flow here (it's "driving" the task), the app should listen for JSON lines (as it already does actually) and reflect the current state in the UI, and reply with another JSON object when the user responds to a prompt, for example.

One last problem

The itch.io API returns objects with snake_case keys, for example https://itch.io/api/1/me/game/3 returns (shortened):

{
  "game": {
    "id": 3,
    "title": "X-Moon",
    "created_at": "2013-03-03 23:02:14",
    "short_text": "Humans have been colonizing planets. It's time to stop them!"
  }
}

But the itch app, being TypeScript, normalizes those to camelCase - here's part of the schema for the 'games' table for example:

And when doing an operation for a game, like launching it, the game info object might be coming from a fresh API call (normalized to camelCase) or cached from the local database (already camelCase) - either of those could be passed to butler for launching it.

butler uses go-itchio, which unmarshals the JSON API responses into structs:

type Upload struct {
	ID          int64
	Filename    string
	Size        int64
	ChannelName string `json:"channel_name"`
	Build       *BuildInfo

	OSX     bool `json:"p_osx"`
	Linux   bool `json:"p_linux"`
	Windows bool `json:"p_windows"`
	Android bool `json:"p_android"`
}

so it "groks" snake_case. There's no camelCase here. Assuming butler can also send objects to the app (for example: to pick an upload, or a manifest action), it should be able to marshal these objects to JSON with camelCase fields rather than snake_case (ie. meet the app's expectations).

So my idea is to do the following changes:

  • Unmarshal API responses into a map[string]interface{}
    • that's how you handle arbitrary JSON structures in golang
  • camelify keys recursively just like the app does
  • change the tags on API structs from json:"p_osx" to json:"pOsx" mapstructure:"pOsx"
  • use mapstructure#Decode to fill the actual API structs from our 'generic' map[string]interface{}
  • when we need to send data back to the app, we can simply json.Marshal them, and it'll produce camelCase-keyed objects

Handling API responses will be very slightly less efficient (but that's really not a performance concern) because we'll have to camelify keys first, but this brings us to a place where butler & the app can exchange objects without any issues because they agree on the naming conventions.

Is that crazy or..

Does the whole thing sound reasonable-ish ?

FAQ

Q: Why not just make the app use snake_case?
A: That's a lot more changes to make, and the code would be half camelCase, half snake_case. I don't want that.

Q: Why have butler handle these 'higher-level' tasks? What's wrong with the app handling it?
A: Most of the logic is already into butler, as separate commands - the app executes butler to download, then to extract, then to find executables+fix permissions, etc. It kinda feels like butler should be able to connect the dots.

Q: Other reasons why it'd be neat if butler handled those tasks?
A: It's easier to silently push updates to butler than to the app - upgrading the app always involves restarting it, whereas there's no reason why the app couldn't update butler while it's running - even if there's already butler instances running handling some other things! (it'd keep both versions around, clean out the old one when it's done).

Q: Why is it important to be able to ship updates to that part of the code?
A: since v23 a lot of the logic for installing games has been rewritten - via https://github.com/itchio/itch-compatibility-watchlist we discover edge cases (games that aren't currently handled) - it'd be really really cool to push a fix to everyone without them needing to restart the app.

Q: Isn't butler like 9-12MB ? Isn't that going to waste bandwidth if you push updates often?
A: Compressed, butler for windows-amd64 is 4MB - but that's not all. The idea is to use butler to patch butler to the latest version. For example, an optimized patch from butler 7.3.0 to butler 8.0.0 is 400KB. Pretty nice!

Q: Are there other advantages in using butler for tasks like these?
A: Yes! golang makes it to write concurrent code, so for example, a long time ago, the app used to download a patch file somewhere, then call butler to apply it. Now, it calls butler with a specially-crafted URL (that includes your credentials to access this file from our CDN), and butler applies the patch while it's downloading. I'm excited about doing the same for download+extracting zip files (which I'm 90% done making resumable - having butler be in charge of the whole process will make it easier to silently roll out the functionality hidden behind an 'experimental' checkbox in itch's UI).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment