`deis run bash` proposal

deis-pty-proposal.md

deis run bash via PTY component

This document proposes a solution for an interactive deis run session, as described in:

• deis/workflow-cli#98
• deis/deis#117

Generally, the intended functionality should match closely with heroku run.

Architectural Diagram

                                 ┌──────────────┐
                                 │   deis CLI   │
                                 └──────────────┘
                                    ▲        ┃
             K8s Cluster━━━━━━━━━━━━╋━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━┓
             ┃                      ▼        ▼                        ┃
             ┃                ┌────────────────────┐                  ┃
             ┃                │       Router       │                  ┃
             ┃                │ deis-pty.my.domain │                  ┃
             ┃                └────────────────────┘                  ┃
             ┃                      ▲        │                        ┃
             ┃                ┌─────┘        └──────┐                 ┃
             ┃                ▼                     ▼                 ┃
             ┃       ┌────────────────┐    ┌────────────────┐         ┃
             ┃       │    new PTY     │    │                │         ┃
             ┃       │   component    │◀───│   Controller   │         ┃
             ┃       │                │    │                │         ┃
             ┃       └────────────────┘    └────────────────┘         ┃
             ┃                ▲                                       ┃
             ┃                └──────────┐                            ┃
             ┃    App Namespace ─ ─ ─ ─ ─│─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─     ┃
             ┃                           ▼                       │    ┃
             ┃    │   ┌────────────────────────────────────┐          ┃
             ┃        │run pod                             │     │    ┃
             ┃    │   │                                    │          ┃
             ┃        └────────────────────────────────────┘     │    ┃
             ┃    │                                                   ┃
             ┃        ┌────────────────────────────────────┐     │    ┃
             ┃    │   │app pod                             │          ┃
             ┃        │                                    │     │    ┃
             ┃    │   └────────────────────────────────────┘          ┃
             ┃        ┌────────────────────────────────────┐     │    ┃
             ┃    │   │app pod                             │          ┃
             ┃        │                                    │     │    ┃
             ┃    │   └────────────────────────────────────┘          ┃
             ┃     ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘    ┃
             ┃                                                        ┃
             ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

Flow Explanation

1. The deis CLI initiates things with a deis run bash-type command. The CLI connects to a new server "/pty" endpoint and passes it the app name and command to be run.

2. The controller creates a UUID to identify the session, then creates the k8s "run" pod. It notifies the deis-pty component to associate that pod with the UUID and wait for it to start. Then the controller returns the UUID to the CLI.

3. The CLI connects to a new router "deis-pty.<domain>" endpoint using websockets. It passes the UUID and begins sending heartbeats to keep the connection alive.

4. The deis-ptybroker component receives an HTTP POST from the controller with a pod name and a UUID. It stores those in a persistent, shared data structure / database, and begins polling for the pod to actually start.

5. The deis-ptybroker component accepts the CLI websocket connection assuming the UUID is known. When the pod starts, the websocket is connected to the equivalent of kubectl exec -it for the remainder of the session.

6. When the "run pod" process exits or the deis CLI closes its connection, the PTY component closes the connections opened for the session, then terminates and cleans up after the pod.

Tasks

Deis CLI

• Refactor deis run to POST to a new /pty (or /app/pty?) controller endpoint
• Add websocket client support to the CLI
• Have deis run connect to deis-pty.<domain> with websockets and provide the UUID

Controller

• Add /pty endpoint
  • create UUID
  • start deis run pod for app asynchronously (fire-and-forget)
  • notify deis-pty to wait for that pod, identified by UUID
  • return from the overall POST by the CLI with a UUID

PTY Component

• Receive websocket clients and keep their connections alive
• Receive HTTP POST from controller to initiate a session
• Start a goroutine to wait for the pod, then connect it to the websocket
• Clean up the pod when either side terminates
• Persist sessions state to PostgreSQL for scalability and durability
• Create a reaper process to clean up orphaned run pods

Charts

• Add new deis-ptybroker component
• Expose its websocket endpoint through the router as "deis-pty.<domain>"

Issues

The PTY component should be scalable--there can be several replicas behind a service. Is it possible (or desirable) for the router to re-establish the connection if a PTY pod dies, in a way that's transparent to the deis CLI?

Can the PTY component just use kubectl binary inside the container? Don't think the client-go library has kubectl attach code, and importing k8s/kubernetes sucks.

If a PTY pod dies suddenly, how can we ensure that any "run pods" it had spawned get terminated? (Persist sessions to PostgreSQL?)

Does PTY component need to be configured similarly to deis-builder, with a shared secret enabling controller to trust it? Or is forwarding the CLI auth token sufficient?

Can this architecture also accomodate deis logs -f (log streaming) without major modifications? See also heroku run:detached which is essentially the same.

Can this be implemented so it's useful without Workflow? That is, can the app- and auth-specific bits and the controller request be abstracted, and could such a component work without the deis CLI?