Skip to content

Instantly share code, notes, and snippets.

@michaelsauter

michaelsauter/ods-pipeline-introduction.md

Last active March 9, 2021 14:46
Show Gist options
  • Save michaelsauter/db826ba8ee078dcd468291af931cbba6 to your computer and use it in GitHub Desktop.
Save michaelsauter/db826ba8ee078dcd468291af931cbba6 to your computer and use it in GitHub Desktop.
Download ZIP
ods-pipeline-introduction.md
Raw
ods-pipeline-introduction.md
@michaelsauter
Copy link
Author

michaelsauter commented Mar 4, 2021 •
edited
Loading

@henrjk Thanks for your comments! I have been thinking about your use case as well and also came to the point of the SO answer. However, I don't have a complete picture right now what a good balance would be. My thoughts so far:

  • One point of my proposal is to have better support for monorepo, as I see the current Jenkins / orchestration pipeline / component-type concept has quite poor support for this
  • The proposal I made above would not allow parallel builds easily. If you used different build tasks for different subfolders, they would try to mount the same workspace, and therefore would be sequential. However if you build the parallelism into the build task itself, you basically require teams to build their own "agent images" as one cannot predict which toolchains to put into the image (and the other option of putting every conceivable toolchain into one image is not good either)
  • I've added a bullet point to the "concerns / limitations" section reflecting one potential solution I see, which is mounting several volumes, one per subfolder ...

In any case, the proposal is still quite flexibel (right now it is mainly "readme-ware") so I'm sure we can adapt it until we have a nice concept. Would love to workshop this with you / the team so that we learn from your experience with the monorepo!

@michaelsauter
Copy link
Author

michaelsauter commented Mar 5, 2021 •
edited
Loading

I re-read the docs (always a good thing to do 😄) and realised that I was wrong on the "no task parallelism when tasks mount the same workspace" assumption. See https://tekton.dev/docs/pipelines/workspaces/#specifying-workspace-order-in-a-pipeline-and-affinity-assistants. I verified this to work as expected in our cluster, meaning tasks still run in parallel by default even if they share the same workspace (because they are assigned to the same node by default). This removes a big limitation outlined above - I've updated the proposal accordingly. Especially build tasks in monorepos will benefit hugely from this.

@felipecruz91
Copy link

With regards to running tasks in parallel sharing files in the same workspace, there's a nice thread about it:

tektoncd/pipeline#2586

@michaelsauter
Copy link
Author

@felipecruz91 thanks for the link. I discovered this as well and the current proposal already takes this into account :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment