Crane 2 pipeline notes

crane2pipeline.md

Crane 2 Pipeline Notes

https://github.com/openshift/pipelines-tutorial

Tekton pipeline concpets: The custom resources needed to define a pipeline are listed below:

• Task: a reusable, loosely coupled number of steps that perform a specific task (e.g. building a container image)
• Pipeline: the definition of the pipeline and the Tasks that it should perform
• TaskRun: the execution and result of running an instance of task
• PipelineRun: the execution and result of running an instance of pipeline, which includes a number of TaskRuns

Crane 2 experience can be a guided wizard that builds this pipeline, and we can execute pipelines using runs.

Export

./bin/crane export --namespace=nginx-example --export-dir=export`

UI input needs:

• Namespace to export
• Maybe an export directory if there's a reason to override the location on the scratch space PV?

Transform

./bin/crane transform \
  --export-dir=$_dir/export/resources \
  --plugin-dir=$_dir/bin/plugins \
  --transform-dir=$_dir/transform \
  --optional-flags="dest-namespace=$DEST_NAMESPACE"

UI input needs:

• export-dir will need to match the location of the export command
• plugin-dir is another scratch location, although we'll definintely want some UX around allowing users to discover and select plugins for usage
• transform-dir more scratch space that will have to feed into the next step
• Optional Flags, this one we may actually want to provide at runtime. I'm seeing a divide between pipeline construction, and runtime. Possibly want two wizards here.

• Runtime would be able to provide things like kustomize values and plugin flags

Apply

./bin/crane apply \
  --export-dir=$_dir/export/resources \
  --transform-dir=$_dir/transform \
  --output-dir=$_dir/output

UI input needs:

• Unsure about this step, again the output-dir is maybe overridable, and the user may want to actually get manual access to these files somehow other than having the pipeline actually automatically commit these to a git repo.

That's the likely next step here is the integration with Argo. This is a lot more fuzzy. Does a pipeline task do that?

State

Haven't had a chance to get hands on with state transfer with crane yet since it's still in development but first iteration should be available this week (8/9).

Will want to stage data as part of a migration, quiesce applicaiton, and capture the final amount. State management might be its own pipeline, independent of the gitops deployment. Gitops deployment can happen even without a PV, and it will sit and wait for a volume coming from the pipeline to present itself.

Plugins

Enhancement for plugin discovery and usage: konveyor/enhancements#41

Considering the distribution method. We'll want an option for users to be able to discover and select that leverages the underlying crane discovery mechanism.

Also definitely want a way for a user to point at either their own binary location for download, or a git repo that we can use to build plugin from source?

Input/Output

Need to wireup inputs and outputs:

• Git credentials + coordinates available to all tasks
• Need to pipe outputs from one task to another

Paramaterization?

May want the ability to parameterize the resulting manifests in some way and possible use kustomize here? Examples: punching holes for specifying namespaces at runtime, resource quotas, node selectors/taints etc.

Custom Validation?

Between each step in the pipeline, should the user have the ability to hook in or tweak to add validations? Are these tasks in the pipeline that will fail?