Pipeline API spec-updated

pipeline.md

Pipeline API

API Design

Cluster level pipeline configs

ClusterPipeline(CRUD),

action=authapp{Input:GithubConfig+code,Output:clusterPipeline}, action=authuser{Input:code,Output:sourcecodecredential}, action=deploy,destroy,revokeapp

HTTP/1.1 POST /v3/clusters/cluster-id/clusterpipelines
Content-Type: application/json
{
  "clusterId": "local",
  "deploy": false,
  "githubConfig": {
    "tls": true,
    "host": "github.com",
    "clientId": "xxxxxx",
    "clientSecret": "xxxxxxxxx",
  }
  //TODO gitlabConfig,bitbucketConfig, etc.
}

Project level pipeline APIs

Pipeline(CRUD),

action=run{Output: pipelineExecution}, activate, deactivate, link=view,export(view and export yaml config)

HTTP/1.1 POST /v3/projects/cluster-id:project-id/pipelines
{
  "projectId": "project-abc",
  "name": "pipeline-test",
  "stages": [
    {
      "name": "build/test",
      "steps": [
        {
          //one of the following config.
          "sourceCodeConfig": {
            "sourceCodeCredentialId": "sc-id",
            "url": "https://github.com/rancher/rancher.git",
            "branch": "master" // onlyBranch/exceptBranch/all options maps to this field in regex way.
          },
          "runScriptConfig": {
            "image": "golang:1.8",
            "isShell": true,
            "shellScript": "echo example script",
            //omit following if isShell==true
            "entrypoint": "custom_entrypoint.sh",
            "command": "--optional args",
            "env": [
              "FOO=BAR"
            ]
          },
          "publishImageConfig": {
            "dockerfilePath": "./Dockerfile",
            "buildContext": ".",
            "tag": "reg.example.com/rancher/server:dev"
          }
        }
      ]
    }
  ],
  "triggerCronTimezone": "Asia/Hong_Kong".
  "triggerCronExpression": "* 4 * * *", //will add validation to limit minimum run sequence
  "triggerWebhook": true,  //pipeline can run multiple times at the same time
  "nextRun": 2,
  "lastExecutionId": "abc",
  "lastRunState": "success",
  "lastStarted": "2012-09-27T18:39:00Z",
  "nextStart": "2012-09-27T18:40:00Z",
  "webhookId": "123", //webhook id created on github repo, use to delete webhook when delete the pipeline. drop and do not expose to users.
  "token": "***", //use for webhook validation, drop and do not expose to users.
}

PipelineExecution(RD), action=stop,rerun, link=pipelineExecutionLog

HTTP/1.1 GET /v3/projects/cluster-id:project-id/pipelineexecutions/id
{
  "projectId": "project-abc",
  "pipelineId": "pipeline-test",
  "id": "pipeline-test-1",
  "run": 1,
  "triggeredBy": "user", //webhook,cron
  "triggerUserId": "user-ld78",
  "pipeline": {}, //pipeline def of this execution 
  "commit": "e5ed7510",
  "state": "running",
  "started": "2012-09-27T18:39:00Z",
  "ended": "2012-09-27T18:39:00Z",
  //want to keep status and pipeline def seperated here
  "stages": [
    {
      "state": "running",
      "started": "2012-09-27T18:39:00Z",
      "ended": "2012-09-27T18:39:00Z",
      "steps": [
        {
          "state": "running",
          "started": "2012-09-27T18:39:00Z",
          "ended": "2012-09-27T18:39:00Z",
        }
      ]
    }
  ]

}

PipelineExecutionLog(R)

HTTP/1.1 GET /v3/projects/cluster-id:project-id/pipelineexecutionlogs/id
// the log will be trimmed to latest section if it exceed store limit
{
  "projectId": "project-abc",
  "pipelineExecutionId": "pipeline-test-1",
  "stage": 1,
  "step": 1,
  "message": "here's the logs"
}

User Level APIs

SourceCodeCredential(CRUD), action=refreshrepos, link=repos

HTTP/1.1 GET /v3/sourcecodecredentials/id?creatorId=user-kb8l5
link=avatar,profile
{
  "clusterId": "local"
  "kind": "github",
  "userId": "rancherUserId",
  "name": "Lawrence Li",
  "loginName": "gitlawr",
  "accessToken": "***"
}

SourceCodeRepository(CRUD)

HTTP/1.1 GET /v3/sourcecoderepositories/id?creatorId=user-kb8l5
{
  "clusterId": "local"
  "kind": "github",
  "userId": "user-ldk8",
  "sourceCodeCredentialId": "somecredentialId",
  "url": "https://github.com/rancher/rancher.git",
  "permissions": {
    "pull": true,
    "push": true,
    "admin": true
  },
  "language": "go"
}

Pipeline

• triggerCron.timezone -> triggerTimezone (don't nest deeply unless there's a compelling reason too)
• triggerCron.expression -> triggerCronExpression
• Is there some validation to restrict this to a minimum of once a minute or something?
  • I can't imagine * * * * * is going to go well.
  • Can the same pipeline be running multiple times at the same time? What if there's a manual run or webhook call while a cron run is happening (or any other combination)?
• status.* -> fields directly on pipeline;
• lastRunId -> lastExecutionId (the type is execution, not run)
• Is lastRunTime the time it started or ended? Change to lastStarted or lastEnded.
• nextRunTime -> nextStart
• Dates should always be ISO-8601 strings ("2012-09-27T18:39:53Z"). We also add a separate "{fieldName}TS: [milliseconds since 1970-01-01" but I think the framework does that automatically if the field is a date.
• What's webhookId and token? Should anyone with access to the pipeline be supposed to be able to read them? Token sounds like something that should have type: masked (or type: password if the framework doesn't support masked, not sure).

Execution

• All of status up a level. The spec and status pattern in k8s is not exposed in the user API
• Is commitInfo just the commit hash? -> commit
• {start, end}Time -> {start, end}ed, as ISO-8601
• stageStatus -> stages
• stepStatus -> steps
• Seems like it would make more sense to combine the stage/step status and the data in pipeline so you have all the steps with what they do and their start/end/state in one map instead of having to cross-reference two different maps to show all the steps and their status
• I know etcd doesn't really like large objects (>1MB?), the entire pipeline/execution being one object might become a problem.

Log

• Seems like storing all the logs as one object is definitely going to be a problem here.. you can easily log many megabytes in one step.

SourceCodeCredential

• sourceCodeType -> kind
• When I said links.avatar last time I meant a field called "avatar" in the "links" map.. {type: '...', links: {"avatar": "http://..."}}.
• Don't say "rancher" in user-facing things unless absolutely necessary. Customers sometimes want to white-label the product. rancherUserId -> userId
• accessToken should be masked/password and not readable by user
• Why are credentials tied to a clusterId when SourceCodeRepositories aren't?

SourceCodeRepository

• URL still says gitrepocaches