Docker compose files for temporal.io with external MySQL databases for temporal and temporal_visibility tables (using PlanetScale as example)

README.md

Build prerequisites

• Please note: I was testing successfully on my local Mac with Intel chipset and amd64 Linux using GitHub CodeSpaces. If you are running into issues as you build/run docker-compose up, it may be worth checking whether you can get it to run in a GitHub Codespace or GitPod.

• There have been reported issues with Apple M1 chips: please click here for more info

Go Lang (minimum version required is 1.18):

• Install on macOS with brew install go
• Install on Ubuntu with sudo apt install golang.

Protocol buffers compiler (only if you are going to change proto files):

• Install on macOS with brew install protobuf.
• Install on Ubuntu with sudo apt install protobuf-compiler.

Temporal CLI tctl

• Homebrew brew install tctl
• Go install make update-tctl or download it from here https://github.com/temporalio/tctl

---

Getting Started with Temporal.io in PlanetScale

Step 1: Clone and compile Temporal

• In the directory of your choice, clone the temporal git repository

mkdir ~/Documents/GitHub/temporalio
cd ~/Documents/GitHub/temporalio
git clone https://github.com/temporalio/temporal.git

• In bash shell, compile the temporal-sql-tool

cd ~/Documents/GitHub/temporalio
make temporal-sql-tool

• Verify that temporal-sql-tool sucessfully compiled.

./temporal-sql-tool --version 
./temporal-sql-tool --help

Step 2: Create two databases in PlanetScale use temporal-sql-tool to connect

• Create an account at https://auth.planetscale.com/sign-up and two databases named temporal and temporal_visibility (assuming you have not used PlanetScale before, you may need to install the PSCALE CLI. More steps can be found here: https://github.com/planetscale/cli#installation

• After compiling your temporal-sql-tool and creating your 2 temporal temporal_visibility PlanetScale databases, you will need to generate passwords & use the connection strings provided in the PlanetScale CLI

Docker compose files for temporal.io with external MySQL databases for temporal and temporal_visibility tables (using PlanetScale as example)

As the docker-compose files 👇 are using PlanetScale's MySQL-compatible Vitess database as an example, each database (temporal and temporal_internal) use different keyspaces and connection strings. Unfortunately, temporalio/auto-setup does not seem to support multiple connection strings for database creation and schema updates (using temporal-sql-tool), so the following commands would need to be run manually before starting up docker-compose:

./temporal-sql-tool --ep $TEMPORAL_PSCALE_HOSTSTRING --user $TEMPORAL_PSCALE_USER --tls  --password $TEMPORAL_PASSWORD-p 3306 --plugin mysql --db temporal setup-schema -v 0.0
./temporal-sql-tool --ep $TEMPORAL_PSCALE_HOSTSTRING --user $TEMPORAL_PSCALE_USER --tls  --password $TEMPORAL_PASSWORD-p 3306 --plugin mysql --db temporal update-schema -d ./schema/mysql/v57/temporal/versioned
./temporal-sql-tool --ep $TEMPORAL_VISBILITY_PSCALE_HOSTSTRING --user $TEMPORAL_VISBILITY_PSCALE_USER --tls  --password $TEMPORAL_VISBILITY_PASSWORD-p 3306 --plugin mysql --db temporal_visibility setup-schema -v 0.0
./temporal-sql-tool --ep $TEMPORAL_VISBILITY_PSCALE_HOSTSTRING --user $TEMPORAL_VISBILITY_PSCALE_USER --tls  --password $TEMPORAL_VISBILITY_PASSWORD-p 3306 --plugin mysql --db temporal_visibility update-schema -d ./schema/mysql/v57/visibility/versioned

• Please note that we populated the necessary environment variables from the PlanetScale connection dialog (TEMPORAL_VISIBILITY_USER, TEMPORAL_VISIBILITY_PASSWORD, TEMPORAL_VISIBILITY_PSCALE_HOSTSTRING, TEMPORAL_USER, TEMPORAL_PASSWORD, TEMPORAL_PSCALE_HOSTSTRING

• You should see something similar to the picture below.

• Verify the Temporal schema and updates in the PlanetScale UI.

Step 3: Start the Temporal Server using Docker with PlanetScale

• In the previous step we created the persistence part of Temporal (the DB) and now the next step is to start the Temporal server.

The default configuration file (docker-compose.yml) uses a PostgreSQL database, an Elasticsearch instance, and exposes the Temporal gRPC Frontend on port 7233. The other configuration files in the repo spin up instances of the Temporal Server using different databases and dependencies.

After successful initialization, you can run temporal like this:

cd docker-compose/
docker-compose -f docker-compose-mysql.yml up
# or, if you like advanced visibility
# docker-compose -f docker-compose-mysql-es.yml up

docker-compose-mysql-es.yml

version: "3.5"
services:
  elasticsearch:
    container_name: temporal-elasticsearch
    environment:
      - cluster.routing.allocation.disk.threshold_enabled=true
      - cluster.routing.allocation.disk.watermark.low=512mb
      - cluster.routing.allocation.disk.watermark.high=256mb
      - cluster.routing.allocation.disk.watermark.flood_stage=128mb
      - discovery.type=single-node
      - ES_JAVA_OPTS=-Xms256m -Xmx256m
      - xpack.security.enabled=false
    image: elasticsearch:${ELASTICSEARCH_VERSION}
    networks:
      - temporal-network
    ports:
      - 9200:9200
  temporal:
    container_name: temporal
    depends_on:
      - elasticsearch
    environment:
      - DB=mysql
      - MYSQL_USER=${TEMPORAL_USER}
      - MYSQL_PWD=${TEMPORAL_PASSWORD}
      - MYSQL_SEEDS=${TEMPORAL_PSCALE_HOSTSTRING}
      - VISIBILITY_MYSQL_USER=${TEMPORAL_VISIBILITY_USER}
      - VISIBILITY_MYSQL_PWD=${TEMPORAL_VISIBILITY_PASSWORD}
      - VISIBILITY_MYSQL_SEEDS=${TEMPORAL_VISIBILITY_PSCALE_HOSTSTRING}
      - SQL_TLS=true
      - SKIP_DB_CREATE=true
      - SKIP_SCHEMA_SETUP=true
      - SQL_TLS_ENABLED=true
      - DYNAMIC_CONFIG_FILE_PATH=config/dynamicconfig/development-sql.yaml
      - ENABLE_ES=true
      - ES_SEEDS=elasticsearch
      - ES_VERSION=v7
    image: temporalio/auto-setup:${TEMPORAL_VERSION}
    networks:
      - temporal-network
    ports:
      - 7233:7233
    volumes:
      - ./dynamicconfig:/etc/temporal/config/dynamicconfig
  temporal-admin-tools:
    container_name: temporal-admin-tools
    depends_on:
      - temporal
    environment:
      - TEMPORAL_CLI_ADDRESS=temporal:7233
    image: temporalio/admin-tools:${TEMPORAL_VERSION}
    networks:
      - temporal-network
    stdin_open: true
    tty: true
  temporal-web:
    container_name: temporal-web
    depends_on:
      - temporal
    environment:
      - TEMPORAL_GRPC_ENDPOINT=temporal:7233
    image: temporalio/web:${TEMPORAL_WEB_VERSION}
    networks:
      - temporal-network
    ports:
      - 8088:8088
networks:
  temporal-network:
    driver: bridge
    name: temporal-network

docker-compose-mysql.yml

version: "3.5"
services:
  temporal:
    container_name: temporal
    environment:
      - DB=mysql
      - DB_PORT=3306
      - VISIBILITY_MYSQL_USER=${TEMPORAL_VISIBILITY_USER}
      - VISIBILITY_MYSQL_PWD=${TEMPORAL_VISIBILITY_PASSWORD}
      - VISIBILITY_MYSQL_SEEDS=${TEMPORAL_VISIBILITY_PSCALE_HOSTSTRING}
      - MYSQL_USER=${TEMPORAL_USER}
      - MYSQL_PWD=${TEMPORAL_PASSWORD}
      - MYSQL_SEEDS=${TEMPORAL_PSCALE_HOSTSTRING}
      - SQL_TLS=true
      - SKIP_DB_CREATE=true
      - SKIP_SCHEMA_SETUP=true
      - SQL_TLS_ENABLED=true
      - DYNAMIC_CONFIG_FILE_PATH=config/dynamicconfig/development-sql.yaml
    image: temporalio/auto-setup:${TEMPORAL_VERSION}
    networks:
      - temporal-network
    ports:
      - 7233:7233
    volumes:
      - ./dynamicconfig:/etc/temporal/config/dynamicconfig
  temporal-admin-tools:
    container_name: temporal-admin-tools
    depends_on:
      - temporal
    environment:
      - TEMPORAL_CLI_ADDRESS=temporal:7233
    image: temporalio/admin-tools:${TEMPORAL_VERSION}
    networks:
      - temporal-network
    stdin_open: true
    tty: true
  temporal-web:
    container_name: temporal-web
    depends_on:
      - temporal
    environment:
      - TEMPORAL_GRPC_ENDPOINT=temporal:7233
      - TEMPORAL_PERMIT_WRITE_API=true
    image: temporalio/web:${TEMPORAL_WEB_VERSION}
    networks:
      - temporal-network
    ports:
      - 8088:8088
networks:
  temporal-network:
    driver: bridge
    name: temporal-network