Using Turborepo for build and Overmind for process management

readme.md

This is a suggested way in which we can combine the power of Turborepo with the consistent, reliable process management of Overmind for managing local services during development.

If we like this approach, I believe it would be a good solution to the "services not ending correctly" issue we've encountered with turborepo.

The only install step for devs introduced by this PR would be: brew install overmind tmux.

Overview:

1. We want a reliable process manager, rather than having ports left hanging or processes in an uncertain state.
2. Turbo knows what packages have a dev task per package.json, and what needs to happen before running processes, but isn't great at process management.
3. It would be nice if we could pass that information off to overmind which does a better job of the process management.
4. We can! The --dry=json dry run data from turbo contains everything we need to construct an on-the-fly Procfile, which is what overmind needs to manage processes (The Procfile format comes from Heroku, and is just a one-line-per-service list of services, see https://devcenter.heroku.com/articles/procfile#procfile-format)
5. Now overmind is in charge, and we can see logs together, or separately per service, as well as benefit from its time-tested reliability.
6. See the scripts/run-dev.mjs script for an explanation of how this approach lets turborepo and overmind "talk", i.e. how overmind knows what services to run, given turborepo's direction.

To pass --filter args for turborepo, you can just pass them via the pnpm command. Examples:

pnpm dev --filter=@gosynthschool/av-client
pnpm dev --filter="*/av-*" --filter="*/play-*"

package.json

{
  //... snip ...
  
  "scripts": {
    "dev": "sh -c '(export TURBO_ARGS=\"${*}\"; turbo dev:deps $TURBO_ARGS && overmind start --root . --no-port --procfile $(node run-dev.mjs))' zero",
    "dev:dry": "turbo dev --dry=json $TURBO_ARGS"
  }
}

run-dev.mjs

/**
 *
 * This script creates a Procfile for [overmind](https://github.com/DarthSim/overmind) from
 * whatever `dev` tasks exist in the monorepo sub-packages, as reported by turborepo.
 *
 * So for example, if `av/client' has a `dev` task in its `package.json` file, turbo will
 * report that it exists, and in what directory. We then create a single line entry in a
 * Procfile that we place in a temp directory that looks like this:
 *
 *   av-client: pnpm -C av/client dev
 *
 * All `dev` services are similarly added to this list, so that overmind can consistently,
 * reliably, and quickly manage our services.
 *
 * This requires `overmind` to be installed: `brew install overmind tmux`.
 *
 *
 * Tips:
 *
 * - Run this script via `pnpm dev`, and it will also run `pnpm dev:deps` as a prior step,
 *   preparing things like databases and building any required transpiled code.
 *
 * - You can use Ctrl-C to shut down all processes.
 *
 * - You can check the status of all services via:
 *
 *   `overmind status`
 *
 * - You can connect (via tmux) to any service to see just its logs via:
 *
 *   `overmind connect [service]`
 *
 * - You can stop or restart just one process (see `overmind --help`).
  *
 */

import { exec } from "child_process"
import fs from "fs"
import os from "os"
import path from "path"

const dryRun = process.env.TURBO_ARGS
  ? `pnpm run --reporter=silent dev:dry ${process.env.TURBO_ARGS}`
  : `pnpm run --reporter=silent dev:dry`

// Run the "dev:dry" script using pnpm
exec(dryRun, (error, stdout, _stderr) => {
  if (error) {
    console.error(`exec error: ${error}`)
    return
  }

  // Parse JSON output of turborepo's dry run
  const data = JSON.parse(stdout)

  // Transform the data into Procfile entries
  const transformDataToProcfileEntries = (data) => {
    return data.tasks
      .filter(
        (task) => task.task === "dev" && !task.command.includes("NONEXISTENT"),
      )
      .map(
        (task) =>
          `${task.directory.replace("/", "-")}: pnpm -C ${task.directory} dev`,
      )
      .join("\n")
  }

  // Transform the data and write it to a tempfile
  const procfileEntries = transformDataToProcfileEntries(data)
  const tempDir = os.tmpdir()
  const tempFile = path.join(tempDir, "overmind.procfile")

  fs.writeFile(tempFile, procfileEntries, (err) => {
    if (err) {
      console.error(`writeFile error: ${err}`)
      return
    }
    console.log(tempFile)
  })
})