control-plane-schema-v1.md

Strawman Schema

Activation

• id
• build_id (Build)
• created_by_id (User)
• state (json)
  • finished_at
  • failed_at
• is_delete?
• created_at
• updated_at

Alert

• id
• task_id (Task)
• type?
• title
• details? (json)
• resolved_at
• created_at
• updated_at

Build

• id x org_id (Organization)
• created_by_id (User)
• contents (json)
• state (json)
  • finished_at
  • failed_at
• created_at
• updated_at

Collection

• id
• entity_id (Entity)
• license
• description
• provenance
• created_at
• updated_at

CollectionTask

• id
• collection_id (Collection)
• task_id (Task)
• created_at
• updated_at

Connector

• id
• type (capture/materialization)
• name
• owner
• description
• created_at
• updated_at

ConnectorImage

• id
• connector_id
• image
• tag
• created_at
• updated_at

Credential

• id
• user_id (User)
• issuer (google/okta/etc, local)
• subject (oidc "sub", basic username)
• hashed_secret (oidc "id_token"/basic password)
• expires_at
• last_authorization_at
• created_at
• updated_at

Entity (FlowEntity?)

• id
• build_id (Build)
• org_id (Organization)
• name (eg. acmeCo/foo/bar)
• norm_name
• type
• next_activation_id (Activation)
• next_entity_type
• created_at
• updated_at

Organization

• id
• name
• prefix
• created_at
• updated_at

Role

TODO

ServiceAccount

TODO?

Task

• id
• entity_id
• connector_image_id (ConnectorImage)
• disabled
• created_at
• updated_at

User

• id
• name
• email
• username
• created_at
• updated_at

---

Questions

• Which Build defines Collection XYZ?

  SELECT build_id
  FROM entities
  WHERE entities.name = ?
  LIMIT 1

• Which version of Capture ABC is active?

  SELECT build_id
  FROM entities
  WHERE entities.name = ?
  LIMIT 1

• Which Materializations read from Collection JKL?

  SELECT name
  FROM entities e1
  INNER JOIN tasks ON tasks.entity_id = e1.id
  INNER JOIN collection_tasks ON collection_tasks.task_id = tasks.id
  INNER JOIN collections ON collections.id = collection_tasks.collection_id
  INNER JOIN entities e2 ON e2.id = collections.entity_id
  WHERE e2.name = ?

• What is the history of Builds that involved Entity X?

  SELECT build_id
  FROM entity_audit_log
  WHERE entity_audit_log.name = ?
  ORDER BY build_history.created_at DESC

• Which Tasks have used Connector:0xabcd?

  SELECT name
  FROM entities
  INNER JOIN tasks ON tasks.entity_id = entities.id
  INNER JOIN connector_images ON connector_images.id = tasks.connector_image_id
  WHERE connector_images.image = ?
  AND connector_images.tag = ?

• Which Builds were built by Flow version=x?

  • Eh, perform a bucket listing on the BUILDS_ROOT based on deployment times.

• What are the definitions for all these Entities as a single Catalog?

  SELECT entities.name, builds.id, builds.contents
  FROM builds
  INNER JOIN entities ON entities.build_id = builds.id
  WHERE entities.name IN ?
  LIMIT 1

• Activate a Build

  def activate(build)
    transaction do
      act = build.create_activation
      act.entities.each do |e|
        e.next_activation_id = act.id
        e.save
      end
    end
  end

  BEGIN
  INSERT INTO activations (build_id, created_at, updated_at)
                   VALUES (?, NOW(), NOW())
  
  INSERT INTO entities (build_id, next_activation_id, org_id, name, created_at, updated_at)
      VALUES (?, ?, ?, ?, NOW(), NOW())
      ON CONFLICT (name) SET next_activation_id = ?, updated_at = NOW()
  
  COMMIT