In MSBuild 15, we've been hard at work adding features that make it easier to manage your build. The development world has evolved greatly in the last 10 years with NuGet packages, more platforms, and the internet in general. We wanted to modernize some of MSBuild to propel it forward to a new era.
MSBuild 15 is open source for the first time. This has allowed us to work in the open in a tighter loop with our customers. It also has allowed community contribution so that features of MSBuild can be provided that the core MSBuild development team does not have time to implement.
A lot of other components in the Microsoft developer stack are open source as well which has greatly improved the developer experience. If you have more suggestions, please visit our repository.
Perhaps the most important "feature" in any product is its performance. In MSBuild 15 we spent time measuring performance and making targeted changes. Some of these changes make builds faster while others reduce memory consumption. When we make changes to MSBuild core components we measure the difference in performance before committing the change. We also experiment with changes to see what improvements can be made. Although MSBuild 15 has improveed, the performance of MSBuild continues to be a focus.
Source code repositories vary greatly in their complexity and scale. A majority of repositories are quite small with only a few projects but there are countless examples of very large repositories with hundreds or thousands of projects. Large teams have unique needs and we've been adding features to make MSBuild scale to both large and small repositories. Scalibility does not only include how the product performs but also how easy it is to own and maintain something.
We took a look at some common patterns that repositories had and added addressed them. These changes are discussed in detail below.
The language of MSBuild is XML. In MSBuild 15.0, we added a few features to make it easier to author your projects.
In previous versions of MSBuild, metadata on items is expressed as a child element:
<ItemGroup>
<MyItem Include="file1.txt">
<IsPlainTextFile>true</IsPlainTextFile>
</MyItem>
</ItemGroup>In MSBuild 15, you can express the metadata as an attribute instead which is easier to read and reduces the amount of XML:
<ItemGroup>
<MyItem Include="file1.txt" IsPlainTextFile="true" />
</ItemGroup>In previous versions of MSBuild, you could not update or remote items unless it was running inside of a target. With MSBuild 15.0, we've made it so you
can update and remove items in top-level ItemGroup elements. This makes it easier to define items so that they show up correctly in Visual Studio and
opens up new extensibility.
For example, you add this to a common import:
<ItemGroup>
<MyItems Include="*.items*" />
</ItemGroup>Now an individual project can filter out items by using Remove:
<ItemGroup>
<MyItems Remove="file.items*" />
</ItemGroup>Additionally, an individual project can add metadata using Update:
<ItemGroup>
<MyItems Update="*.items*" Visible="false" />
</ItemGroup>Traditionally, MSBuild projects contain a lot of information. Much like source code, projects should share as much logic as possible to make it easier to maintain. Unfortunately, older project templates place a lot of properties in each project when they are created. While the newer SDK-style projects have addressed this by leaving out the properties from the template, there are still hundreds of thousands of projects that already exist.
A traditional way of adding a common import to all projects was to add a line like this:
<Import Project="$([MSBuild]::GetDirectoryNameOfFileAbove($(MSBuildThisFileDirectory), build.props))\build.props" />The above line configures the project to search directories above until it finds one that contains a file named build.props so it can be imported. By
placing this import in every project in a repository, a user can configure functionality for their projects in one place. However, it can be difficult
to enforce this import in every project. All developers working in a repository must remember to add the import manually to newly created projects. If
your code base is open source, pull request reviewers must remember to watch for new projects to ensure that the import exists.
To address this, we added a common import directly into the MSBuild's in box logic. Any project that imports Microsoft.Common.props and
Microsoft.Common.targets automatically imports a file in your repository named Directory.Build.props and Directory.Build.targets. This means
that any project that is newly created already from a Visual Studio template will automatically import a file you have created. You will no longer need
to manually add an import. This works for the legacy projects and the newer SDK-style projects.
You want all projects in your tree to have a property set if the build is taking place in your hosted build environment. The hosted build environment sets
and environment variable named JENKINS but you might move to a different solution in the future. You can create a property that indicates if the build
is hosted and update the logic if you ever moved your build.
You create a file at the root of your repository named Directory.Build.props:
<Project>
<PropertyGroup>
<IsHostedBuild Condition=" '$(JENKINS)' != '' ">true</IsHostedBuild>
</PropertyGroup>
</Project>Since Directory.Build.props is imported very early in a project, the IsHostedBuild property is available to all projects. You can now build extra
projects depending on the build environment or enable/disable features of your build.
Depending on your needs, you can use Directory.Build.targets. Since Directory.Build.targets is at the end of a project, it can be used to override
targets or settings that can't be set by Directory.Build.props. Suppose you want to inject a target to run after every project is done:
<Project>
<Target Name="PostBuild" AfterTargets="AfterBuild">
<Message Text="Finished building project $(MSBuildProjectFile)" />
</Target>
</Project>The common import functionality addresses the need to maintain a repository at scale. By allowing you to control all projects in a central location, it is much easier to manage a repository with thousands of projects. You can read more about this functionality here.
NuGet packages having become a vital component to everyone's build. They allow you to define dependencies to make them available without requiring users to install
things prior to building. Visual Studio will automatically restore your packages before you build but command-line based builds will not work unless packages have
been restored first. Visual Studio also contains all of the logic needed to restore packages. However, to restore packages from the command-line, you traditionally
have to download and run NuGet.exe. Hosted build environments also must have some prerequisite task that restore package prior to building.
To addess this, we added a command-line argument to MSBuild.exe named /restore. This tells MSBuild to run a target named Restore and then run a build.
Before executing the build, the projects are re-evaluated to ensure that if any packages affected the import graph that the changes are taken into account. The
Restore target is defined by in-box logic that comes with MSBuild or you can define your own target to do whatever you want. So now MSBuild's experience is
much more like Visual Studio because packages are restored before building and you don't get a bunch of build errors if you forget to manually restore first.
When MBuild is running the Restore target, projects are loaded in a way that imports can be missing, invalid, or empty. This helps ensure that if the restore
itself pulls down or generates imports that the restore process will still work.
If you need different global properties to be set during the command-line based restore, you can also use the /restoreProperty argument. This sets properties
just like the /property argument but only when the Restore target is running. If the /restoreProperty argument is not set, any properties defined by
/property arguments will be set for both Restore and the other build targets.
You want to restore packages first and then run a target named Publish:
> MSBuild.exe /Restore /Target:Publish
You want to restore packages first but you want a property named IsRestoring to be set during Restore:
> MSBuild.exe /Restore /RestoreProperty:IsRestoring=true
You may find it difficult to remember to always specify /restore when doing a command-line based build. Please read the next section to learn how to control
command-line arguments for your repository.
To control the command-line build experience of MSBuild, users have traditionally turned to batch files. This allows users to ensure that the correct arguments
are used when building a repository. There are common patterns like having a build.cmd, init.cmd, or compile.cmd just to name a few. By forcing users
to run batch files, repository owners also must maintain documentation on how a repository must be built.
To address this, we added functionality to MSBuild.exe to look for a file named Directory.Build.rsp in the project directory or any directory above. This
allows you to configure the command-line arguments used by command-line based builds. This allows repository owners to configure global properties, customer
loggers, and any other argument that is necessary for your build.
When users build repository or your repository is building in a hosted environment, you want to the output to have minimal verbosity with a summary, a text log should be written in case the build fails, and you want packages to be restored first.
You create a file at the root of the repository named Directory.Build.rsp:
/ConsoleLoggerParameters:Verbosity=Minimal;Summary
/FileLoggerParameters:Verbosity=Diagnostic
/Restore
Now users can simply run msbuild and those command-line arguments will apply. Hosted builds also pick up the arguments so your output is clean but you have a
text log file with full diagnostic output in case you need to investigate a failure. You can read more about this functionality
here.
Warnings and errors that are logged during a build can come from various sources. They come from task libraries, compilers, or other tools. Traditionally, each
task would have to implement the ability to elevate or suppress warnings. This is done in the managed compilers (CSC.exe and VBC.exe) which can be controlled
with MSBuild properties. However, this leaves other warnings that are not mutated which can make it hard to ensure that warnings are not introduced into your
repository.
We addressed this in MSBuild by adding command-line arguments and MSBuild properties. They allow you to elevate all warnings or a list of warning codes to an error and to suppress warnings.
The command-line arguments are /warnaserror and /warnasmessage and the MSBuild properties are:
| Property | Usage |
|---|---|
MSBuildTreatWarningsAsErrors |
true or false to elevate all warnings to errors |
MSBuildWarningsAsErrors |
A semi-colon delimited list of warning codes to elevate to errors |
MSBuildWarningsAsMessages |
A semi-colon delimited list of warning codes to demote to message |
Warnings are suppressed first and elevated to an error second which allows a way of ignoring some warnings but treating everything else as an error.
You own a repository and don't want people to introduce warnings into your build. However, there is one warning code that is okay so you want it to be ignored. You
can use the Directory.Build.props functionality detailed above or set properties in individual projects.
Sample Directory.Build.props:
<Project>
<PropertyGroup>
<!-- Treat all warnings as errors -->
<MSBuildTreatWarningsAsErrors>true</MSBuildTreatWarningsAsErrors>
<!-- Suppress these warning codes so they do not fail the build -->
<MSBuildWarningsAsMessages>NU1234;BY183</MSBuildWarningsAsMessages>
</PropertyGroup>
</Project>If warnings are introduced over time, it can be very difficult to go back later and fix them. Preventing them from ever being introduced into a code base is very valuable so this functionality can help small and large repositories.
When MSBuild was first developed, there was no online store to download applications, NuGet packages didn't exist, and Windows dominated the operating system landscape. MSBuild's core design requires that all projects and build logic be on disk prior to building. This also meant that users typically would have to install all software development kits (SDKs) before they could build a particular repository. Installing Visual Studio would usually bring along SDKs that were commonly used but there was always something else that needed to be installed that was not in box.
The primary solution to this problem was to commit all of the assembly references and build logic into a repository so that you could build the source code without having everything installed first. As NuGet has become more popular, it has allowed developers to declare dependencies and the NuGet client can download and make them available for the build. This works well today for assembly references and most build logic. However, there is some build logic that might need to change how NuGet works or implement something fundamental that just can't be a NuGet package.
Some repositories still rely on a batch file to download or configure the build environment so that the build can proceed. This forces developers to run the batch file before opening projects in Visual Studio and can lead to a higher documentation cost.
We addressed this in MSBuild by adding the ability to pull special NuGet packages as part of MSBuild loading a project. We call these special packages MSBuild Project SDKs. Project SDKs can modify how NuGet pull packages or provide functionality that is needed before standard packages can be used. We shipped project SDKs that let you manage your NuGet package versions in a central location, configure traversal projects, or have a project that doesn't compile an assembly. More project SDKs are becoming available which reduces the need to install anything before building a repository.
You can read more about project SDKs here.
If you own a widely used build extension, it has traditionally been difficult to make your component extinsible. A common way of extending some MSBuild logic is to provide a way for users to specify a file to import before or after your build logic is imported. This provides users a way of configuring your build extension.
Since imports are evaluted very early by MSBuild, the only way to make them programatic is to import a property:
<Project>
<!-- A user can set the property CustomBeforeMyExtension which will be imported before this extension in case they want to override something -->
<Import Project="$(CustomBeforeMyExtension)" Condition=" '$(CustomBeforeMyExtension)' != '' " />
<!--
All of the logic of my extension
-->
<!-- A user can set the property CustomAfterMyExtension which will be imported after this extension in case they want to override something -->
<Import Project="$(CustomAfterMyExtension)" Condition=" '$(CustomAfterMyExtension)' != '' " />
</Project>The above approach works well but can only be set in one place. This is because older versions of MSBuild can only import a single file. The properties used
above like CustomBeforeMyExtension could not be set to a list of projects.
To address this, we made the <Import /> element work with a semi-colon delimited list. This allows build extensions to be extensible in a more powerful way.
They can document a property like CustomBeforeMyExtension which can be set multiple times:
<Project>
<PropertyGroup>
<CustomBeforeMyExtension>$(CustomBeforeMyExtension);..\MyConfiguration.props</CustomBeforeMyExtension>
</PropertyGroup>
</Project>The CustomBeforeMyExtension property can be set in my repository or by a NuGet package that extends another NuGet package. This allows build logic to be more
easily extended.
MSBuild's plain text logging can be very difficult to understand. As projects become more complex, the logging does not make it easier to understand issues or build logic problems. This issue is exacerbated when your build is using multiple processors because the log is full of information from multiple threads all mixed together.
To address this, we built into MSBuild a new log format we call the Binary Logger. Rather than emitting plain text, each logging event is written in its full form as a binary stream. This has some major advantages over plain text logs:
- Since every event is stored in the log, it can be transformed into any other format later.
- GUI viewers can read the binary format to present a more human-readable representation. The most popular is the Structured Log Viewer
- The log can be highly compressed since something else needs to read it first before presenting the contents to a user.
Another major improvement is that by default the Binary Logger captures the import graph of your project. This allows viewers of the logs to see your project XML in its entirety to make investigating issues much easier.
To capture a binary log, specify the /BinaryLogger command-line argument:
> MSBuild.exe /BinaryLogger
You can then download the Structured Log Viewer to open msbuild.binlog:
Binary logs are highly recommended for hosted builds since they are compressed and contain all diagnostic information available to a build. You can specify the
/BinaryLogger command-line arguments in your Directory.Build.rsp file to automatically emit a binary log for all command-line based builds.
This is nice but would've liked it if this is added to the docs along with the newer versions, v16, v17 too!