In cloud build, we rely on the source code that we want to build against having been
checkout out to the /workspace directory in the build environment, and the $PWD
of the build environment being that same directory. In short, when the build starts,
your source code repository is checked out in ./, the current directory. From there,
it's easy to refer to different pieces of source code by relative path, and makes using
docker-related tools like docker and docker-compose feel natural, despite the fact
that the whole build, itself, is, in fact, running inside a container. This water-tight
abstraction is what breaks down in the defect in cloud-build-local.
The build defined by cloudbuild.yaml uses the docker-related tooling at two different levels.
First, it illustrates the build implicitly invoking docker run to run some
bash commands on the ubuntu image, to show what's visible with 1 layer of
indirection. Here, we see the output of our ls -R /workspace command showing what we
would expect; the contents of ./ have been mounted to /workspace:
Starting Step #0
Step #0: Already have image (with digest): ubuntu
Step #0: /workspace:
Step #0: README.md
Step #0: cloudbuild.yaml
Finished Step #0
2019/02/14 13:47:42 Step Step #0 finished
Second, we'll try to build on this abstraction and run an explicit
invocation of docker run nested inside the implicit invocation.
Inception? No. Essentially, the step gcr.io/cloud-builders/docker
tells docker to run a docker container, and that docker container
is also capable of running docker. Inside, we invoke docker run
and try to mount /workspace (which we showed has our local files)
onto a different container in the nested call, this time at /project;
ignore for the time being that the outer mount (/workspace) is defined
in a default substitution, as that will be explained in the section on testing. This will totally work in the real cloud build, producing:
Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/docker
Step #1: File: /project/README.md
Step #1: Size: 9110 Blocks: 24 IO Block: 4096 regular file
Step #1: Device: 801h/2049d Inode: 10109094 Links: 1
Step #1: Access: (0664/-rw-rw-r--) Uid: ( 0/ root) Gid: ( 0/ root)
Step #1: Access: 2019-02-14 21:59:21.979924047 +0000
Step #1: Modify: 2019-02-14 21:59:04.000000000 +0000
Step #1: Change: 2019-02-14 21:59:21.979924047 +0000
Step #1: Birth: -
Finished Step #1
By contrast, the same specification run in cloud-build-local will fail with:
Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/docker
Step #1: stat: cannot stat '/project/README.md': No such file or directory
Finished Step #1
2019/02/14 13:58:17 Step Step #1 finished
2019/02/14 13:58:18 status changed to "ERROR"
Looking into this problem, there were couple of clues that were essential to
understanding the behavior. First, was the fact that the arguments to
docker run -v to mount volumes inside cloud build are not
local directories, but are, instead, names of local volumes, as the solution to mounting an excrypted volume illustrates.
What this means, in practice, is that there is an implicit volume:
specification for the /workspace mount in cloud build that is not
replicated, locally. Interestingly, if ./ is supplied, it can also
be mounted by name, successfully (try it yourself by changing the
default substitution).
The other clue came from inspecting the actual source code of cloud-build-local. More specifically, armed with the knowledge that
they name the local mount that shows up as /workspace something, and it needs to be referred to by name, then it was
left to just figure out what they named it. The relevant source code to collect arguments for docker, and the local setup code to initialize them
makes it clear that the name of the volume is not as simple as ./
or /workspace (that would be too easy). Instead, it looks like the
volume name includes a UUID if the source code is copied to /workspace,
but is just the full path ($PWD) if it's being mounted with -bind-mount-source.
To test the concept, the file cloudbuild.yaml was refactored, and now
parameterizes the name of the mount that's being re-mounted to the inner
docker container. In the default substitution, it's still set to the value that we know won't work (/workspace). However, on command-line substitutions, it can be set to $PWD, and
the source mounted with -bind-mount-source so that what's being
used as the name in the reference is the full path, to match what it's
set to in the referenced code. The full command looks like:
cloud-build-local -bind-mount-source -substitutions=_MOUNT_NAME=`pwd` -dryrun=false -push=false .
Sure enough, that name allows us see the local directory, now re-mounted
by name, and the inner stat /project/README.md command succeeds:
Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/docker
Step #1: File: /project/README.md
Step #1: Size: 5817 Blocks: 16 IO Block: 4096 regular file
Step #1: Device: 4dh/77d Inode: 7592429 Links: 1
Step #1: Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Step #1: Access: 2019-02-14 22:56:59.000000000 +0000
Step #1: Modify: 2019-02-14 22:56:37.000000000 +0000
Step #1: Change: 2019-02-14 22:56:37.000000000 +0000
Step #1: Birth: -
Finished Step #1
2019/02/14 14:57:18 Step Step #1 finished
Unfortunately, this is a clunky work-around, which involves creating user-defined substitutions which then have the only logical value as the default in the definition.