Skip to content

Instantly share code, notes, and snippets.

@larrycai

larrycai/argo-workflow-for-ci.md

Last active May 28, 2022 15:37
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save larrycai/b5dc214f4d12024fec36b8c1ad14f003 to your computer and use it in GitHub Desktop.
Save larrycai/b5dc214f4d12024fec36b8c1ad14f003 to your computer and use it in GitHub Desktop.
Download ZIP
argo workflow for normal CI flow
Raw
argo-workflow-for-ci.md

It is for discussiion in argo-workflows forum

Introduction

It is just my understanding now, can be totally wrong ;-)

Here gives a short explanation for user wants to use argo workflow as alternative for your CI framework (gitlab-ci, github actions, travis-ci, jenkins)

argo workflow is not a complete CI solution, it needs other compoments to fullfil your tasks. it is cloud native workflow execution engine

If you want to have complete simple solution quickly, recommended to use "jenkins + argoworkflow"

(TODO) Getting started python project using argoworkflow

How CI works

Let's take a look at how CI general works, and how argoworkflow and other ci works in this area

  • trigger: when the pipeline can be triggered, it needs to monitor git repo, branch, file changes
  • token: how to access git repo (pull/push)
  • workspace: how to share the data across steps
  • archive (optional): how to visible the result (like unit test result)

trigger

argo-workflow doesn't cover this, you can use argo-events or webhook to trigger the flow, for different type of git repo, solution could be different.

If you are familiar with jenkins, it could be good choice as well to have jenkins in front, since it has lots of mature plugin to cover it.

anyway, it is not nature enough like github action and gitlab-ci

token to access git repo

for public repo, just git clone, see sample input-artifact-git.yaml

otherwise it needs token to access, which leads to secrets and sealedsecrets

workspace

for github action, it is VM based, shared workspace via volume sharing, for argo workflow, u need to arrange pvc to be shared

see example ci.yaml, this is a good start!

archive the result

all ci framework has different solutions for it, be prepared.

using extra service like nginx/artifactory could be good simple solution.

Example for github python project

TODO

Jenkins + argo workflow (without events)

Use jenkins plugin mechanism to get triggered and inject flow to argo workflow

Suggestion to argoproj

I do hope argoproject can create argo-ci project again to focus on providing complete CI solution

  • integrate needed components (argo-events / argo-workflow)
  • create extra service or enhance argo workflow to make key data are default without extra parameters (token, pvc)
  • extra UI system (ui for archives/logs/reports ..)
  • better "plugin" system like github action marketplace, argo-workflows-catalog still has long way to go

Reference

@larrycai
Copy link
Author

larrycai commented May 28, 2022 •
edited

Getting started with argo ci workflow for golang project

argo workflows is a nice cloud native workflow execution engine, surely it can work like normal CI engine though with different setup.

Below is a simple guideline using official documentation and examples

Installation

Surely you need one kubernetes cluster installed with argo server and argo-cli locally, see argo workflows quick start

For client at mac, I use brew command to install directly, it default uses your kube config without extra token to access argo server

$ brew install argo

Getting started with golang project for CI

The example yaml file is ci.yaml@99359a0950, so the lines I indicated below is always correct.

In short, do below

$ kubectl create ns argo-ci
$ argo submit -n argo-ci --watch https://raw.githubusercontent.com/argoproj/argo-workflows/master/examples/ci.yaml
Name:                ci-example-6ttdw
Namespace:           argo-ci
ServiceAccount:      unset (will run with the default ServiceAccount)
Status:              Running
Conditions:          
 PodRunning          False
Created:             Sat May 28 15:53:20 +0200 (16 seconds ago)
Started:             Sat May 28 15:53:20 +0200 (16 seconds ago)
Duration:            16 seconds
Progress:            1/4
ResourcesDuration:   3s*(1 cpu),3s*(100Mi memory)
Parameters:          
  revision:          cfe12d6

STEP                                                                                       TEMPLATE              PODNAME                      DURATION  MESSAGE
 ● ci-example-6ttdw                                                                        ci-example                                                     
 ├───✔ build                                                                               build-golang-example  ci-example-6ttdw-2944230572  9s          
 └─┬─◷ test(0:image:debian,tag:9.1)    run-hello             ci-example-6ttdw-984197245   6s          
   ├─◷ test(1:image:alpine,tag:3.6)    run-hello             ci-example-6ttdw-1191131657  6s          
   └─◷ test(2:image:ubuntu,tag:17.10)  run-hello             ci-example-6ttdw-3275768438  6s          

This workflow does not have security context set. You can run your workflow pods more securely by setting it.
Learn more at https://argoproj.github.io/argo-workflows/workflow-pod-security-context/
$ argo logs -n argo-ci @latest
ci-example-6ttdw-2944230572: HEAD detached at cfe12d6
ci-example-6ttdw-2944230572: nothing to commit, working directory clean
ci-example-6ttdw-2944230572: github.com/golang/example/stringutil
ci-example-6ttdw-2944230572: github.com/golang/example/hello
ci-example-6ttdw-984197245: Linux ci-example-6ttdw-984197245 5.4.0-72-generic #80~18.04.1-Ubuntu SMP Mon Apr 12 23:26:25 UTC 2021 x86_64 GNU/Linux
ci-example-6ttdw-984197245: PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
ci-example-6ttdw-984197245: NAME="Debian GNU/Linux"
ci-example-6ttdw-984197245: VERSION_ID="9"
ci-example-6ttdw-984197245: VERSION="9 (stretch)"
ci-example-6ttdw-984197245: ID=debian
ci-example-6ttdw-984197245: HOME_URL="https://www.debian.org/"
ci-example-6ttdw-984197245: SUPPORT_URL="https://www.debian.org/support"
ci-example-6ttdw-984197245: BUG_REPORT_URL="https://bugs.debian.org/"
ci-example-6ttdw-984197245: Hello, Go examples!
...

Use argo-ci namespace

since argo namespace is used to install argo server, I recommend to use another namespace for pipeline execution

git checkout

It is stated in Hardwired Artifacts, git is supported is built-in support already, which is missed from beginning.

it is in ci.yaml#L55

      artifacts:
      - name: code
        path: /go/src/github.com/golang/example
        git:
          repo: https://github.com/golang/example.git
          revision: "{{inputs.parameters.revision}}"

the repo and revision can be passed with parameters as template concept, which is skipped here.

workspace

workspace is managed by volumes, which is kubernetes pvc concept, which needs to be specified in each step. depends on your storage in your cluster, it could be nfs or others as default

  volumeClaimTemplates:
  - metadata:
      name: workdir
....
      volumeMounts:
      - name: workdir
        mountPath: /go

Further check

You can further check with argo workflow templates, cron workflows and all in quick-start

Reference

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment