s3resource.md

What we did

Ran a simple pipeline in a vagrant concourse-lite machine, consisting of:

• a S3 resource pointing at a bucket
• a job with a get step referencing that resource

What we learned

When we run the pipeline for the first time, the S3 bucket did not exist:

• clicking on the resource in the concourse UI showed an error message that says "The specified bucket does not exist"
• the job is left in the pending state This seems OK, because it is fairly easy to find out what is happening.

When we run the pipeline and the bucket exists, but is empty:

• there's no obvious error when clicking on the S3 resource in the UI
• the job is shown as pending This is not OK, because it's not obvious what the problem is.

Investigation

We captured the JSON that atc is sending to the S3 resource and it showed us that the check call includes a null version, this is as expected according to http://concourse.ci/implementing-resources.html

We saw that the existing behaviour of the S3 resource was to return an empty array of versions

We modified the S3 resource to return instead return a single element array of versions containing the version "null". When we did this, the check succeeeded, but the job then failed when trying to retrieve the null version from the S3 resource.

Concourse doesn't seem to have the concept of a resource that doesn't have a version, judging by http://concourse.ci/implementing-resources.html, which describes no version being passed into the check, but versions being returned OR a version being passed in and an emtpy array being passed out (to mean no new versions since the one you passed in).

Options

1. Investigate making atc/concourse report something sensible when a resource returns an empty version array to assist debugging.
2. Raise an issue about the current behaviour to find out what the concourse team think the behaviour - both of the resource and of concourse - should be when a resource has no versions.
3. Try using concourse v0.69.1, which has this change: "Put and Get steps will now appear to start running as soon as they start running rather than when their first byte of output appears." However, there's no concourse-lite vagrant VM yet, so it's more work to test.
4. Extend/modify S3 resource to support the concept of a default version and content, that are returned if the bucket is empty. In the paas-cf create-deployer pipeline, we would specify a default version of 0 and default content of an empty terraform state and everything should just work.