Matt's awesome blog
The last post in this series covers a range of techniques for using TeamCity to build a continuous delivery deployment pipeline in a scalable way. This post builds on those techniques by providing some more options to consider given the new features in TeamCity 8 as well as what we've learnt since the original post.
This post is part of a blog series jointly written by myself and Matt Davies called Maintainable, large-scale continuous delivery with TeamCity:
todo list of contents
Following is a bunch of (mostly new) TeamCity features we didn't cover in the last post that we think are useful to know if you are looking after a large-scale TeamCity deployment.
One of the big improvements in Teamcity 8 is the addition of Projects Hierarchy, which allows you to nest projects. In a large-scale TeamCity deployment this is probably very useful:
- You can group related projects e.g. per-client, per-team, per-department etc.
- If you follow our advice in the last post about splitting up production deployments to a separate project (if you need to have different permissions) then you can group the prod and non-prod projects together
As well as a logical grouping, project hierarchy gives you the ability to share the following:
- Build configuration parameters and templates
- Clean-up rules
- VCS Roots
- Users and group roles
- Shared resources
- Meta-runners
Rob has written before about using the importance on using pull requests for commercial software development. One of the really cool things in TeamCity is the ability to automatically build pull requests and then to report back their status via one of a number of plugins (if using GitHub, BitBucket, Stash). You can see more about how to do this in the excellent post by Mehdi Khalili.
Semantic Versioning is really useful for software libraries because it helps communicate the scope of a change to the users of that library. Jake Ginnivan has talked to us before about using semver for commercial projects to communicate the status of a build to the client e.g. major revision can be used for a large change / feature / rewrite, a minor revision can be used for a new feature and a patch can be used for bug fixes or small changes to an existing feature. If you want to use custom versioning with TeamCity then it actually provides you with the ability to dynamically change the build number to help accommodate this.
If you are using Git for your source control, then you are in luck because Jake Ginnivan has created a continuous delivery compatible way of using semver with TeamCity via the GitHubFlowVersion project (note: it's currently in the stages of being merged into the GitVersion project). It's important to note that you can no longer use the Assembly Info Patcher Build Feature when using GitHubFlowVersion, but it has the ability to patch the assemblies for you.
In the last post we discussed the disadvantage of not having the build number propagate to each of the consequential build configurations and provided a number of links to potential solutions. We have since discovered a way of sharing versions between build configurations that share a dependency. You can simply put in %dep.<DEPENDENT_BUILD_CONFIG_ID>.system.build.number% where <DEPENDENT_BUILD_CONFIG_ID> is the id of the build configuration you want to use the build number from (note: it must have a snapshot dependency for you to be able to reference it). TeamCity 8 makes this much easier since you can specify the build configuration ID and the auto-completion of dependent variable names works a lot better.
If you find you need multiple build templates and these templates all repeat two or more build configuration steps or a single complex build configuration step then you can extract the common step(s) into a meta-runner. For an example of a Meta Runner check out the TeamCity Meta-runner power pack and the recent post by Rob.
When you are administrating a large scale TeamCity deployment in a limited resource environment you may find you need to quickly monitor where your hard disk space is being allocated. One of the new features in TeamCity 8 that can help in this situation is the disk usage report. You should also keep in mind the clean-up configuration options available to you.
If you have a large TeamCity deployment then it's likely some of your builds may depend on limited external resources such as test databases, metered resources etc. If this applies to you then you may want to look into TeamCity 8's shared resources feature which allows you to limit the number of builds using these shared resources at any point in time.
If you find you have a lot of contention for your build agents you can take advantage of a new feature in TeamCity 8 which allows you to monitor all current builds queued and running on the server.
TeamCity simplifies using custom NuGet packages for your own libraries and products by providing a built-in NuGet server (introduced in TeamCity 7) which integrates with package artifacts generated by a build configuration. This feature is incredibly useful for large scale deployments where you have shared code between projects and you are using NuGet to manage your inter-project dependencies.
An important philosophy in Continuous Delivery is to make sure that your application is compiled once only and those same binaries are reused for every subsequent part of the deployment pipeline. This gives confidence that you are deploying the same code that has been tested and deployed earlier in the pipeline and form a part of the confidence that a proper deployment pipeline gives you.
One of the weaknesses in our last post was that the code could be rebuilt across build configurations if there is more than one agent (or MSBuild decided it wanted to rebuild). This was correctly pointed out in the comments by Marcin.
We have experimented with tweaking the pipeline to get build once and we have three main options we have explored:
- Ensure you have one agent only and trust MSBuild to not rebuild if the files haven't changed (aka the last post)
- Rather than using Web Deploy to perform the XDT transformation for each environment, create the MSDeploy package while building (using
/p:DeployOnBuild=true), put that package as an artifact, populate the package to deployment steps as an artifact dependency and then use the parameters functionality in MSDeploy to perform environment transformations at deploy time: - Use OctopusDeploy, which supports deploy-time configuration changes (see below)
If you can have a single tool to create your deployment pipeline and include continuous integration as well as deployments then there are obvious advantages in terms of simplicity of configuration and management (since set of project definitions, user accounts and permissions, single UI to learn, etc.). This is one of the reasons we created this blog series; we loved how powerful TeamCity is out of the box and wanted to expose that awesomeness so other people could experience what we were.
Lately, we have also been experiementing with combining TeamCity with another tool to take care of the deployments; OctopusDeploy. There are a number of reasons we have been looking at it:
- Curiosity; OctopusDeploy is getting a lot of attention in the .NET community so it's hard not to notice it - it's worth looking at it to see what it does
- If you are coordinating complex deployments then OctopusDeploy takes care of managing that complexity and the specification of the deployment process in a manageable way (that would start to get complex with TeamCity)
- For instance, if you need to perform database migrations then deploy multiple websites and a background worker together then OctopusDeploy makes this easy
- Visualising the deployment pipeline is much easier in OctopusDeploy, which is great when you are trying to get non-technical product owners involved in deployments
- It gives you a lot more flexibility around performing deploy-time actions out-of-the-box and makes it easier to do build once packages
- It is possible to do the same things with MsDeploy, but it is more complex to do
- It has great documentation
- It comes with plugins that make using it with TeamCity a breeze
We wouldn't always use OctopusDeploy exclusively, but it's definitely a tool worth having a good understanding of to use judiciously because it can make your life a lot easier.
OctopusDeploy uses the NuGet package format to wrap up deployments; there are a number of ways you can generate the NuGet package in TeamCity:
- OctoPack (if you install OctoPack into your project then it's dead easy to get the NuGet package - you can install the plugin to invoke it or pass through
/p:RunOctoPack=trueto MSBuild when you build your solution/project if using .NET) - TeamCity NuGet Pack step (if you are using a custom
.nuspecfile then it's easy to package that up and automatically get it as an artifact by using TeamCity's NuGet Pack step) - PowerShell or another scripting language (if you need more flexibility you can create a custom script to run that will package up the files)
In order for OctopusDeploy to access the deployment packages they need to be published to a NuGet feed that the OctopusDeploy server can access. You have a number of options of how to do this from TeamCity:
- Publish the package to TeamCity's NuGet feed (this is easy - simply include the
.nupkgfile as an artifact and it will automatically publish to it's feed) - Publish the package to OctopusDeploy's internal NuGet server
- Publish the package to some other NuGet server you set up that OctopusDeploy can access
In order to create releases and trigger releases to those deployments from TeamCity there are a number of options:
- Use the TeamCity plugin from OctopusDeploy
- Use Octo.exe
- Write a custom script or program to talk to the OctopusDeploy API
One thing that wasn't mentioned in the previous post about TeamCity pipelines was running UI Tests. Since that post we have done a lot of work with integrating automated UI tests as part of our TeamCity pipelines. Providing guidance about running automated UI testing in TeamCity is a post in itself, but here are a few pointers to get you started:
- Consider what point in the pipeline you want to run your UI tests e.g. alongside your unit tests, after automatically deploying to your first environment after unit tests, after deploying to a test environment to provide more confidence before hitting prod, after deploying to a dedicated (static or on the fly) UI testing environment
- Remember that UI tests are generally slow so preferably you want to run it out of band of your first build configuration with unit tests
- If possible include a separate TeamCity agent that is dedicated for running UI tests so your continuous integration builds don't get delayed and you increase your immediate feedback loop
- Consider whether you want to run UI tests against all pull requests or not (on one hand you get a lot of confidence, on the other it takes a lot more time and requires isolated environments, which may be more complex
- Configure your agent machines so they are configured to be able to run the automated UI tests if you need interactive mode; note: if you are using PhantomJS then you might not need the agent to have interactive mode
- Make sure that you take screenshots whenever your UI tests fail and that the artifacts are dropped in a directory that is configured in your build configuration to pick up all files as artifacts
- Make sure you clear that directory before every build otherwise the screenshots will start stacking up from previous builds - TeamCity has an option for this