cdcon-local-pipelines.org

Local execution of Pipelines

Agenda

An exploration of how and why running our pipelines locally.

• Why ?
• How ?
• Expectation ?

Goal: compare what exists today (with or without tekton), and discuss what’s next ?

Why ?

• Use the same “recipe” for building locally and on the CI
• Change the recipe at one place and that’s it
• Debug CI pipelines (?)

Use the same thing for inner-loop and outer-loop.

Expected user experience

Local execution should be as close as possible to the CI execution (minus some environment and different secrets). Ideally the exact same command is ran.

What about Tekton ?

• What is the role of CI/CD (and Tekton) in a developer’s local workflow? #145
• CI/CD and the development workflow from Christie
• It is up to us to define it ✍️.
• Tekton Mission

  Be the industry-standard, cloud-native CI/CD platform components and ecosystem. […]

  • Engineers who need CI/CD: (aka all software engineers!) […]
    • Be able to use multiple Tekton conformant systems instead of being locked into one or being forced to build glue between multiple completely different systems
    • Use an ecosystem of tools that know how to interact with Tekton components, e.g. IDE integrations, linting, CLIs, security and policy systems

How ?

How does it work today

Jenkins 🪧

Lost time is never found again 😼 … … or, you are on your own 😉, not possible

The easy way (with some limitations) : jenkinsfile-runner The hard way : setup a local jenkins properly

If no Jenkinsfile, you are on your own 🙃

Gitlab 🪧

$ gitlab-runner exec docker \
                # --env CI_COMMIT_REF_SLUG="main" \
                # […] \
                {job}

• Same runner used on Gitlab CI (same binary)
• Relatively smart (guess most environment variables necessary)

GitHub 🪧

nektos/act — run your GitHub Actions locally 🚀 It does have some limitations.

$ act -j {job}

… or, *officially* you are on your own 😉, not possible

Tekton based (kubernetes) 🪧

• kind
• Docker Desktop
• Minikube

$ kubectl …

Does not build local working directory (yet)

Tekton based (experimental) 🪧

buildkit-tekton

• Very young and experimental
• Not fully supports all Tekton features
• Does not build local working directory (yet)

$ tkn local run -f …

Dagger 🪧

Dagger is a portable devkit for CI/CD. […] Benefits of using Dagger include:

Unify dev and CI environments. Write your pipeline once, Dagger will run it the same everywhere. Reduce CI lock-in. No more re-writing everything from scratch every 6 months. Rapid debugging of your pipelines. Dagger runs equally well on your local machine, and in CI.

dagger do …

Conclusion

• dagger seems the closest to that ideal.
• Does it make sense for Tekton to “care” about inner-loop ?
• Does it make sense for the tekton community to “invest” ?

Note: Opinions matter for local execution but Tekton is not opiniated 😅.

• GitHub and Gitlab are 1 images for all the “steps”
• Dagger and Tekton are DAGs of containers (steps in Tekton)
• This is where we need to discuss, brainstorm, experiment, play 🏓