Plugin Framework in Rancher Pipeline

plugin.md

Plugin Framework in Rancher Pipeline

Objective

Pipeline currently provides extensibility by using different context images in pipeline steps. An image with certain utilities can be considered as a plugin. But we lack a centralized place to maintain available plugins, either provided by Rancher or community users. Also, the user experience for using different plugins can be improved so that users with poor experience in container/kubernetes can use existing util images more easily.

Proposed Design

We use questions files to describe the usage of a pipeline plugin. An example for using drone s3 plugin:

name: s3
image: plugins/s3
command: drone-s3
categories:
- publish
- storage
questions:
- variable: S3_ENDPOINT
  default: ""
  label: "s3 endpoint"
  description: "endpoint for the s3 connection"
  type: string
  required: true
- variable: S3_BUCKET
  default: ""
  label: "bucket"
  description: "s3 bucket"
  type: string
  required: true
- variable: PLUGIN_SOURCE
  default: ""
  label: "upload files from source folder"
  type: string
  required: true
- variable: PLUGIN_TARGET
  default: ""
  label: "upload files to target folder"
  type: string
  required: true
- variable: AWS_ACCESS_KEY_ID
  default: ""
  label: "aws access key"
  type: secret
  required: true
- variable: AWS_SECRET_ACCESS_KEY
  default: ""
  label: "aws secret key"
  type: secret
  required: true 

In pipeline project settings, add a configuration for plugin list:

It should be a list of question files for the plugins:

#pipeline-plugins.yaml
name: s3
image: xx
command: xx
questions: xx
---
name: xx
image: xx
command: xx
question: xx
---
...

When users add a pipeline step, they can choose the plugin step in the list:

As the result, the step configuration will be converted to the pipeline yaml format as:

...
  steps:
  - pluginConfig:
      plugin: s3
      # args in map type
      args: 
        S3_ENDPOINT: https://play.minio.io:9000
        S3_BUCKET: my-bucket-name
        PLUGIN_SOURCE: public/**/*
        PLUGIN_TARGET: /target/location
    envFrom:
    - sourceName: secret1
      sourceKey: secretkey1
      targetKey: AWS_ACCESS_KEY_ID
    - sourceName: secret1
      sourceKey: secretkey2
      targetKey: AWS_SECRET_ACCESS_KEY
        

In this way, users can have customized/additional pipeline steps by simply changing the plugin questions yaml file.

Implementation

1. enhancements for questions.yml we are using for catalog
   • Add name/image/command for Plugin support
   • Add secret type question for referencing secrets in pipeline steps
   • It would be nice to have reference field type like reference[workload] #or other resources for a question, so that users can pick a project workload for upgrade in dropdown in pipeline step configuration. But this is not a critical one.
2. UI supports rendering plugin steps according to the questions file.
3. Backend supports setting plugin questions file and plugin steps.