digdag tutorial?

digdag.md

1. How to setup local development environment?

1.1 setup method

• Downloading the latest version

1.1.1 install directly

• Running sample workflow

1.1.2 using docker

• Running tasks on Docker

1.2 including folder structure and naming conventions

2. How to test and debug locally?

2.1 This is how I did

2.1.1 command line

digdag run myproj
If it fails
Fix issue
digdag run myproj --retry
Got error saying attempts already exist
digdag attempts
Find attempt id
digdag retry 3 —last-revision —all
diadag log 7

digdag init hello
cd hello
digdag run hello
digdag run --rerun hello

2.1.2 ui

2.3 how to stop and resume

• kill

2.4 how to change schedule

• Repush your new shcedule projects.

3. How to deploy?

3.1 I’m considering to use CI tool(i.e. Jenkins) to push changes together at a regular interval.

3.2 Do you have any recommendations?

FAQ

Q. How to stop workflow

• digdag kill

Caution: digdag never stop current running task. digdag kill does just not execute next tasks.

Q. How to rerun from the failed task

Q. How to delete project

Q. How to check currently running workflows and status

Q. How to schedule a workflow

Q. Where can I find the description of repository tables

Q. How to deploy changes to production

Q. Where can I find workflow examples

The examples in the digdag github repository is a good starting point Treasure data which provides hosted service also created good examples

• https://github.com/treasure-data/digdag/tree/master/examples
• https://github.com/treasure-data/workflow-examples

Q. How to keep secret(password) securely

• Secrets
• Secret Encryption Key

Q. How to pass variables between tasks

• Programmable workflow in Ruby
• Programmable workflow in Python
• Param operators

Q. Types of variable in digdag

Q. Where can I find liquid template for embulk configuration

• How to use Liquid template in Embulk Japanese.