Hello, Ben.
It was good to meet you in the Ask The Experts Booth at this year's DockerCon. Thank you for answering my questions about LDAP integration with the DataCenter's role-based access control, and for patiently attempting to explain to me how we at Medidata, the "docker-compose uninitiated", might overcome our need to bundle Ruby gems from non-public Git repositories, using only docker-compose. I'm still unclear on the specifics of the technique you laid out, so thanks even more for offering to provide an example!
Fortunately, our problem is very simply expressed as a single line added to the excellent Rails example that you guys already posted for Docker compose. Only the build phase is relevant here, so our problem can be quickly reproduced by following those instructions faithfully, until the part where it says to add the rubyracer gem to the Gemfile.
At that point, if we also add a gem whose source is a non-public git repo, of the form:
gem 'mygem', :git => 'git@github.com:user/mygem'
then the "docker-compose build" command fails, naturally - there's no secret credential to do the checkout. Our challenge is to make that command succeed, with the following 3 requirements:
- the secret used to perform the git checkout, whether password or SSH key, must not be embedded in the resulting docker image in any way. The build environment is free to pass this info to docker-compose by ENV, shared volume, or any other means that does not violate this prime constraint.
- docker only in the build environment (no Ruby, bundler, etc. from outside a container)! However, using the docker-compose YAML equivalent of "docker run -v .:/my_app --entrypoint=/some/binary some/image" to invoke these tools from the build environment is legit, because then we're still just using Docker's software for the build.
- DRY - the Gemfile is the single-source of truth for expressing the Rubygems dependencies, so no listing them explicitly in some other file -- unless that file is derived at build time from the Gemfile, which is pretty tricky given constraint #2. Also, recall that private gems frequently depend on other private gems that are not even in the Gemfile!
I think those are the only conditions, but I'll write again if I've forgotten something. 😳
Here is some more info on this issue as we've confronted it, including links to the hack I built to get around it. In fact, there is a long and still open GitHub Issue in the docker repo itself, where you may find much wailing and gnashing of teeth about this problem. As you can see from the 121 comments, a lot of people are interested in a docker-native solution. On behalf of us all: don't be shy! 😁
Thanks again,
benton
Benton Roberts Cloud Engineer, Medidata Solutions Inc. broberts@mdsol.com