(names removed to protect the guilty)
On one day, the chef-client on our opennebula hosts failed. After looking into it, someone had updated the base recipe and removed a repository which was no longer reachable. So, I'll update the environment of the application cookbook.
$ berks update
Now test it.
$ kitchen converge
...
The rack gem suddenly depends on ruby > 2.2.2, let's pin it to a well known working version.
$ cd ../flavour-cookbook; vim recipes/server.rb
There, update from berkshelf, run kitchen, commit the changes and push it to my repo. Lets submit a merge request which puts it through jenkins. Jenkins fails. Jenkins fails because some gems depend on ruby > 2.2.2. A colleague hands me a magic Gemfile which will fix this problem apparently.
Put in the Gemfile, test with kitchen, commit, push, error. Rubocop was complaining about a missing issues_url and source_url. Because the rubocop version on the jenkins is newer than on my machine. chef exec bundle install, rinse, repeat.
Edit, test with kitchen, commit, push, error. This time it took a bit longer. After all, kitchen converge finished successfully on my machine. Not that it would on Jenkins.
The recipe couldn't install the opennebula package because someone thought it was a smart idea to mirror the opennebula repository and overwrite the yum repo I specifically defined in my cookbook. And to make matters worse, the mirrored version was an even newer version with breaking changes.
Instead of mirroring the repository URL I stated in the cookbook intentionally, they just mirrored "stable".
Today I found out, that repositories in the repo cookbook are disabled by default. If I enable it through the environment, the wrong repository will be used and production will go to hell.
So I decided to remove to repository from the shared repos cookbook.
Remove 4 lines from the attributes.rb, do a kitchen converge, every thing looks fine. Let's push it and create a merge request.
Jenkins Build Failed - what else did I expect? Sprinkle the magic Gemfile onto the recipe, commit and push.
Jenkins failed again.
11:52:09 You have requested:
11:52:09 chef ~> 12.12.15
11:52:09
11:52:09 The bundle currently has chef locked at 12.14.89.
11:52:09 Try running `bundle update chef`
Seriously, what the fsck? So the bundle is locked at version 12.14.89 and I am requesting a version around 12.12.
Go read www.slideshare.net/isotopp/be-simple-be-boring-be-obvious.
Removed the version contraints for chef, chef-solo, buff-extensions - let's see what breaks next.
Currently I'm in a edit, commit, push loop.
Edit, commit, push:
12:19:23 You have requested:
12:19:23 buff-ruby_engine ~> 0.1.0
12:19:23
12:19:23 The bundle currently has buff-ruby_engine locked at 1.0.0.
Locking buff-ruby_engine to 1.0.0 leads to:
13:52:44 Installing buff-ruby_engine 1.0.0
13:52:44
13:52:44 Gem::InstallError: buff-ruby_engine requires Ruby version >= 2.2.0.
I'm stuck in a maze of dependencies, all different.
Copying the gem file with the fixed versions back to the recipe, do a chef exec bundle install and committing the Gemfile and Gemfile.lock was what I should have done earlier. At least Jenkins starts to test now...
Jenkins failed again, complaining about the now non-existant opennebula repository. Right. Remove the test, commit, push.
The repo recipe now works without the opennebula repository. Back to fixing the original problem with rack being too new for our rubby. Edit, commit, push, fail after 15ish minutes.
Why? Someone decided that it was a good idea to have the Gemfile.lock in the repository. As the recipe is about 9 months old, this wasn't in the old skeleton cookbook and still has Gemfile.lock in its .gitignore. Remove Gemfile.lock from .gitignore, commit both and push.
Jenkins failed because I forgot to do the chef exec bundle update dance. Dance, commit, push, swear at display.
After 8 hours: 00:11:39.739 Finished: SUCCESS
And all I wanted to do was do update the cookbooks in a single environment.
sounds so familiar.. 👎