proposal.md

Building Applications with Native Dependencies

Date: April, 2019

The first set of .NET Core releases, up to and including .NET Core 2.2, focused on the needs of web applications built with ASP.NET Core. Web applications don't directly depend on native libraries or OS-specific APIs as a general rule. As a result, we didn't focus on the usability of native libraries as much as we need to looking forward.

Client applications differ in this respect, often relying on native libraries and APIs and typically need to integrate with OS-provided experiences (like notifications). With the addition of client scenarios in .NET Core 3.0, we need to reassess the usability and capability of building applications and libraries that include native assets.

With .NET Core 3.0 Preview 3, the default model for building .NET Core applications with native dependencies is to produce artifacts that can be run on any supported operating system or chip architecture. This "architecture-neutral" model is beneficial because it offers the most flexibility across deployment targets. However, this model isn't the best, and in fact is harmful, if you intend to deploy your application to a single operating system. It also isn't clear that it is the best model for a developer inner-loop default.

Many .NET Framework developers unfamiliar with .NET Core will start using the newer environment with .NET Core 3.0 and will expect a similar experience as they have been used to, particularly for those that only ever intend to target Windows with Windows Forms and WPF applications. The architecture-neutral model may result in significant developer confusion and extra work to move to a better configuration.

The .NET Core build system offers multiple options for building applications, each of which offers different characteristics for managing native assets. This document explores the various build options, their current usability and proposes improvements where appropriate.

Dystopian Customer Experience

The premise of this document can be summarized into a single (ficticious) conversation between two developers in a typical .NET shop in a Fortune 500 company:

• [Anna] "Hey Frank, why does your Windows Forms application include Linux and Mac binaries in this 'runtimes' directory? They make the app bigger, and it looks strange. In fact, why do you haved that directory at all?"
• [Frank] "What?!? Let me check ... Oh, it's because I'm using this Cpu Math library from Microsoft and you get all builds of it by default in your app. This is how its supposed to work with .NET Core. It wasn't like that before, when we were using .NET Framework for this app."
• [Anna] "Are you going to change the default? Seems totally unneccessary and odd."
• [Frank] "I wasn't going to."
• [Anna] "Did you get approval from legal to ship Linux binaries?"
• [Frank] "OK, OK ... let me check project properties .... not finding anything obvious here ... off to StackOverflow"

Proposal

The following section proposes how the experience should be improved.

Later in the document, several scenarios are demonstrated and explored. At the risk of ruining the suspence, this exercise proves that building applications that depend on native assets is too hard, and some scenarios don't work as expected, for both .NET Core 2.2 and 3.0.

Goals

The following broad goals should be satisfied for building applications with native dependencies:

• .NET CLI defaults (for native dependencies) are intuitive and align with key user scenarios.
• .NET CLI opt-in arguments (for native dependencies) have good usability for key user scenarios, with a bias to short and memorable arguments and good defaults for those arguments.
• It is straightforward to transform CLI opt-in arugments (for native dependencies) that enable your project use-case into msbuild properties in your project file, resulting in a custom default for that project.
• It is straightforward to produce deterministic builds, independent of OS or architecture of the build machine.
• It is easy and intutive to share applications with others or deploy them to other machines (like a Rasperry Pi, or a Windows ARM64 laptop) as part of a fluid workflow.

The following are more specific goals to improve CLI usability:

• build and publish should have the same capability, except where there is a good scenario reason why a task should only be made available via publish.
• Defaults should be the same between build and publish.
• All common scenarios should work without specifying a RID if you are happy targeting the current machine.
• We should strongly avoid any breaking changes.

These goals may present tension on each other.

Assumptions and Positioning

Looking at the current experience in .NET Core 3.0, we have the following default models for builds:

• Framework-dependent: Architectural-neutral
• Self-contained: Architecture-specific (and that's the only option)

Instead, we should model the product on targetting architecture-specific applications as the default build type and enable targetting multiple architectures as an opt-in experience. The lack of symmetry is part of what has led to the current experience being awkward. In addition, we have very limited evidence that developers want to publish applications with native dependencies as a single unit to multiple native environments, while at the same time we can easily prove that it will be harmful for (at least) Windows client developers.

The .NET Framework produces framework-dependent architecture-specific applications by default for applications with native dependencies. It is the better default for development and for many deployment scenarios. We should adopt this model instead of the framework-neutral architecture-specific default adopted in .NET Core 3.0.

Application Types

For the purposes of this document, there are three types of applications:

• Architecture-specific, framework-dependent -- Applications of this kind target and can only run on a single architecture, like Windows x64 or Linux ARM32. They require an installed runtime to run. They carry one set of dependent native libraries for a single target architecture. All application binaries are contained in a single directory.
• Architecture-neutral, framework-dependent -- Applications of this kind target and can only run on multiple architectures. They require an installed runtime to run. They carry multiple sets of dependent native libraries for a variety of target architectures. All application and other architecture-agnostic binaries are contained in a single directory and architecture-specific binaries are carried in RID-specific directories within a runtimes directory.
• Architecture-specific, self-contained -- Applications of this kind target and can only run on a single architecture, like Windows x64 or Linux ARM32. They carry a runtime and one set of dependent native libraries for a single target architecture. All application, runtime and native binaries are contained in a single directory.

Proposed Changes

The following changes are proposed for .NET Core 3.0:

• Adopt framework-dependent architecture-specific apps as the default for build and publish. This aligns with the behavior in .NET Core 2.2 and .NET Framework and also aligns thematically with the architecture-specific default for self-contained builds.
• Adopt a new set of arguments for controlling how to build architecture-specific apps. It isn't possible to produce both a non-breaking and intuitive model with the way -r and --self-contained interact today. In short, we chose the wrong default for --self-contained.
• A new --arch (-a alias) argument replaces --runtime / -r and --runtime / -r is deprecated. It is an error to use the two arguments together. -a by itself means current architecture.
• Make framework-dependent architecture-neutral application an opt-in case, using a new --arch-neutral argument.
• --arch-project provides a way of specifying the RuntimeIdentiers listed in the project file on the command-line.

Experience:

• build and publish produce architecture-specific framework-dependent builds by default.
• Specifying an architecture via -a makes the app architecture-specific for a specific RID, but not self-contained. --self contained or self-contained true must be specified to create a self-contained app when -a is present.
• Specifying --self-contained with one or more architectures, by some combination of -a (possibly multiple) and --arch-projects (only once) arguments, results in one of more self-contained applications.
• Specifying --arch-neutral results in an architecture-netural application. Multiple architectures must be set, either via the project file, or some combination of -a (possibly multiple) and --arch-project arguments.
• For build, specifying --self-contained by itself remains unsupported (same as 2.2 behavior).
• For publish, specifying a rid with -r results in a self-contained app. self-contained false must be specified to create an architecture-specific app (same as 2.2 behavior).

For completeness, two other options were considered, as follows.

Symmetric but breaking option

• build and publish produce architecture-specific framework-dependent builds by default.
• Specifying -r species the specific RID to be used, but does not result in a self-contained application.
• --self-contained or --self-contained true must be specified to create a self-contained application
• --arch-neutral must be specified to create an architecture-neutral application.

non-breaking but assymetric option

• build and publish produce architecture-specific framework-dependent builds by default.
• For build, specifying a rid with -r makes the app architecture-specific, but not self-contained. --self contained or self-contained true must be specified to create a self-contained app (this is not breaking, per 2.2 behavior).
• For publish, specifying a rid with -r results in a self-contained app. self-contained false must be specified to create an architecture-specific app.
• --arch-neutral must be specified to create an architecture-neutral application.

Proposal for apphost launchers

Note: This section falls into the "would be nice" category. We shouldn't do this unless motivated by customer feedback. This section will be deleted when it is posted to a public forum.

Today, the build will produce an architecture-specific launcher -- like app.exe -- even though an app might be architecture-neutral. It's very useful to have a launcher for user experience reasons, but it is also confusing if you want to use an architecture-neutral app on architectures where a given launcher is not supported.

We could enable two new modes:

• Opt-in to multiple launchers for some set of architectures, with appropriately differentiated names.
• Opt-out of including an executable launcher in the build output.

With .NET Framework, Windows developers could copy Windows Forms apps to a share and expect other developers in their organizations to run them, even directly from the share. In the fullness of time, one can expect that .NET Core will support Windows client applications on x86, x64 and ARM64. Developers on ARM64 machines will not be able to run the applications that they are given to run from developers on x64 machines, even though the app might be 99% architectural neutral. Multiple launchers could resolve this situation. There is no other obvious solution, modulo relying entirely on app stores for delivery.

Exploration of current behavior for building .NET Core applications with native dependencies

The following sections explore the current bejavior for building applications with native dependencies.

Architecture-neutral, framework-dependent applications with NO native dependencies

This type of application is the easiest to consider because it doesn't depend on native libraries. It benefits from the architecture-neutral nature of .NET binaries and doesn't need to consider the complications of native dependencies. As a result, it is out of scope of this document, but is important to compare to as a baseline.

The following example demonstrates the set of assets produced by an application of this type and the associated user experience.

C:\git\testapps\threezeroapp>type threezeroapp.csproj
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.0</TargetFramework>
  </PropertyGroup>

</Project>

C:\git\testapps\threezeroapp>dir /b /s bin\*.dll bin\*.exe bin\*.so bin\*.dylib
C:\git\testapps\threezeroapp\bin\Debug\netcoreapp3.0\threezeroapp.dll
C:\git\testapps\threezeroapp\bin\Debug\netcoreapp3.0\threezeroapp.exe

C:\git\testapps\threezeroapp>bin\Debug\netcoreapp3.0\threezeroapp.exe
Hello World!

C:\git\testapps\threezeroapp>dotnet bin\Debug\netcoreapp3.0\threezeroapp.dll
Hello World!

The resulting application is very simple, containing only a .NET dll with the application logic and a native launcher (aka "app host").

The application is invoked in two different ways, once with a native launcher (that can only be used on X64 Windows, due to being built with the Windows X64 .NET Core SDK) and the other with the standard dotnet launcher enabling the application to be used anywhere. The dichotomy of these choices is discussed later in the document. Only one of the launchers will be used in subsequent examples.

Architecture-neutral, framework-dependent applications with native dependencies

Architectural neutral applications with native dependencies must contain native assets for all supported operating systems and architectures. This means that this type of applications needs to include multiple builds of the same native library, in order to provide a compatible asset on each native environment on which the application is supported to run.

The architecture-neutral model depends on three characteritics: an installed .NET Core runtime (for that environment) on the target machine, multiple variants of dependent native binaries carried with the application (the .NET assembly binder know how to pick the correct one), application binaries are composed of architectural-neutral byte code (IL) as opposed to pre-compiled code (using crossgen). This is a great model for cases where a developer wants to deploy the same build of an application to machines that use multiple operating systems and chips. It isn't a good option for developers that use conditional compilation that targets operating system or chip.

The following example demonstrates the set of assets produced by an application of this type and the associated user experience. This application depends on the Microsoft.ML.CpuMath NuGet library, which includes native binaries that support a variety of operating systems and chip combinations.

C:\git\testapps\cpumath>type cpumath.csproj
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.ML.CpuMath" Version="0.11.0" />
  </ItemGroup>

</Project>

C:\git\testapps\cpumath>dotnet build
Microsoft (R) Build Engine version 16.1.21-preview+gc36f772a85 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Persisting no-op dg to C:\git\testapps\cpumath\obj\cpumath.csproj.nuget.dgspec.json
  Restore completed in 105.76 ms for C:\git\testapps\cpumath\cpumath.csproj.
C:\Program Files\dotnet\sdk\3.0.100-preview4-010761\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(151,5): message NETSDK1057: You are using a preview version of .NET Core. See: https://aka.ms/dotnet-core-preview [C:\git\testapps\cpumath\cpumath.csproj]
  cpumath -> C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\cpumath.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:02.11

C:\git\testapps\cpumath>dir /s /b bin\*.dll bin\*.exe bin\*.so bin\*.dylib
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\cpumath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\Microsoft.ML.CpuMath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\cpumath.exe
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\runtimes\linux-x64\native\libCpuMathNative.so
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\runtimes\osx-x64\native\libCpuMathNative.dylib
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\runtimes\win-x64\native\CpuMathNative.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\runtimes\win-x86\native\CpuMathNative.dll

C:\git\testapps\cpumath>bin\Debug\netcoreapp3.0\cpumath.exe
Hello World!

The resulting application contains all architecture-agnostic code within the root directory and then multiple copies of native libraries in RID-specific runtime directories that support the various environments. The .NET Core assembly binder discovers and loads the correct native library for the given environment. The launcher is specific to the RID of the SDK that was used to build the application.

The intention of this model is that the build output will be deployed as a uniform quantity on all operating systems, even though only one quarter of the native assets (in this particular example) will be used in any given environment. The benefit is that you only to need to produce a single build of an application. The downside is that the application will be larger than it needs to be because it contains the aggregate of native assets that will be needed across all supported deployments. If that's not desirable, then the next option might be a better option.

Architecture-specific, framework-dependent applications with native dependencies

Framework-dependent architecture-specific applications only contain native assets for the specific environment that they support. That specialization can help reduce the size and complexity (in terms of disk layout) of framework-dependent applications that depend of native assets.

The following example demonstrates the set of assets produced by an application of this type and the associated user experience, using the same Microsoft.ML.CpuMath package used earlier. In order to produce the desired application, the target architecture as a runtime ID (RID) must be specified as well as opting out of the default self-contained behavior (when a RID is specified).

C:\git\testapps\cpumath>dotnet build -r win-x64 --self-contained false
Microsoft (R) Build Engine version 16.1.21-preview+gc36f772a85 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

MSBUILD : error MSB1001: Unknown switch.
Switch: --self-contained

For switch syntax, type "MSBuild -help"

That approach doesn't currently work. Instead, we can publish the application with publish, which produces the expected results. Due to the way publish works, we have two separate copies of the application. The publish directory is the one to pay attention to.

C:\git\testapps\cpumath>dotnet publish -r win-x64 --self-contained false
Microsoft (R) Build Engine version 16.1.21-preview+gc36f772a85 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Persisting no-op dg to C:\git\testapps\cpumath\obj\cpumath.csproj.nuget.dgspec.json
  Restore completed in 109.27 ms for C:\git\testapps\cpumath\cpumath.csproj.
C:\Program Files\dotnet\sdk\3.0.100-preview4-010761\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(151,5): message NETSDK1057: You are using a preview version of .NET Core. See: https://aka.ms/dotnet-core-preview [C:\git\testapps\cpumath\cpumath.csproj]
  cpumath -> C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\cpumath.dll
  cpumath -> C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\publish\

C:\git\testapps\cpumath>dir /s /b bin\*.dll bin\*.exe bin\*.so bin\*.dylib
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\cpumath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\CpuMathNative.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\Microsoft.ML.CpuMath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\cpumath.exe
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\publish\cpumath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\publish\CpuMathNative.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\publish\Microsoft.ML.CpuMath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\publish\cpumath.exe

The resulting application contains all binaries, architecture-agnostic and architecture-specific, in the same directory. In this case, look at the C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\publish\ directory.

The command-line required to enable this style of build is long and may be hard for people to remember. Those options can be similarly added to a project file for a similar result (what we would have been expected if dotnet build had worked as desired).

C:\git\testapps\cpumath>type cpumath.csproj
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.0</TargetFramework>
    <RuntimeIdentifier>win-x64</RuntimeIdentifier>
    <SelfContained>false</SelfContained>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.ML.CpuMath" Version="0.11.0" />
  </ItemGroup>

</Project>

C:\git\testapps\cpumath>dotnet build
Microsoft (R) Build Engine version 16.1.21-preview+gc36f772a85 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Persisting no-op dg to C:\git\testapps\cpumath\obj\cpumath.csproj.nuget.dgspec.json
  Restore completed in 106.23 ms for C:\git\testapps\cpumath\cpumath.csproj.
C:\Program Files\dotnet\sdk\3.0.100-preview4-010761\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(151,5): message NETSDK1057: You are using a preview version of .NET Core. See: https://aka.ms/dotnet-core-preview [C:\git\testapps\cpumath\cpumath.csproj]
  cpumath -> C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\cpumath.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:01.11

C:\git\testapps\cpumath>dir /s /b bin\*.dll bin\*.exe bin\*.so bin\*.dylib
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\cpumath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\CpuMathNative.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\Microsoft.ML.CpuMath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\cpumath.exe

The resulting application is the same as the one demonstrated above, with publish, but in this case with build, which produces simpler output.

A further optimization on this experience is changing the project file to always build for the RID of the SDK you are using, which will always result in an application that targets the current OS and chip combination. The following project file can be used for that need. It will produce a similar result, depending on the RID of the SDK used to build an application.

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.0</TargetFramework>
    <RuntimeIdentifier>$(NETCoreSdkRuntimeIdentifier)</RuntimeIdentifier>
    <SelfContained>false</SelfContained>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.ML.CpuMath" Version="0.11.0" />
  </ItemGroup>

</Project>

Architecture-neutral but explicitly limited, framework-dependent applications with native dependencies

The next intuitive step is to extend the single RID listed in the project file to multiple, with the goal of supporting a smaller set of runtimes than what an underlying library supports. This is a hybrid case of the two preceding models. Let's suppose we want to support Windows X64 and Linux X64. This model could be used to build a Windows version (that supports x86, x64 and ARM64) and a separate Linux version. The split can be on whatever pivot you want, not just the one chosen in this example. It should be possible to enable our chosen example with the addition of the RuntimeIdentifiers property in the project file.

C:\git\testapps\cpumath>type cpumath.csproj
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.0</TargetFramework>
    <RuntimeIdentifiers>win-x64;linux-x64</RuntimeIdentifiers>
    <SelfContained>false</SelfContained>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.ML.CpuMath" Version="0.11.0" />
  </ItemGroup>

</Project>

C:\git\testapps\cpumath>dotnet build
Microsoft (R) Build Engine version 16.1.21-preview+gc36f772a85 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Persisting no-op dg to C:\git\testapps\cpumath\obj\cpumath.csproj.nuget.dgspec.json
  Restore completed in 192.52 ms for C:\git\testapps\cpumath\cpumath.csproj.
C:\Program Files\dotnet\sdk\3.0.100-preview4-010761\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(151,5): message NETSDK1057: You are using a preview version of .NET Core. See: https://aka.ms/dotnet-core-preview [C:\git\testapps\cpumath\cpumath.csproj]
  cpumath -> C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\cpumath.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:01.23

C:\git\testapps\cpumath>dir /s /b bin\*.dll bin\*.exe bin\*.so bin\*.dylib
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\cpumath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\Microsoft.ML.CpuMath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\cpumath.exe
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\runtimes\linux-x64\native\libCpuMathNative.so
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\runtimes\osx-x64\native\libCpuMathNative.dylib
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\runtimes\win-x64\native\CpuMathNative.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\runtimes\win-x86\native\CpuMathNative.dll

C:\git\testapps\cpumath>

Strangely, the runtimes are not limited in this case. That seems like a bug. The RuntimeIdentifier property had no observable effect.

I tried the CLI equivalent of the same thing, but got a strange error.

C:\git\testapps\cpumath>dotnet publish -r linux-x64 -r win-x64
Microsoft (R) Build Engine version 16.1.46-preview+ge12aa7ba78 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

MSBUILD : error MSB1009: Project file does not exist.
Switch: win-x64

Architecture-specific, self-contained applications

Self-contained applications have one model, architecture-specific, with all dependencies contained in a single directory. As a result, it doesn't matter (for the purpose of this document) if a self-contained application contains native dependencies.

The following example demonstrates the set of assets produced by an application of this type and the associated user experience, using the same cpumath example used earlier. A target runtime is specified as part of building the application.

C:\git\testapps\gpioapp>dotnet build -r win-x64
Microsoft (R) Build Engine version 16.1.46-preview+ge12aa7ba78 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restore completed in 238.61 ms for C:\git\testapps\gpioapp\gpioapp.csproj.
C:\Program Files\dotnet\sdk\3.0.100-preview4-011022\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(151,5): message NETSDK1057: You are using a preview version of .NET Core. See: https://aka.ms/dotnet-core-preview [C:\git\testapps\gpioapp\gpioapp.csproj]
  gpioapp -> C:\git\testapps\gpioapp\bin\Debug\netcoreapp3.0\win-x64\gpioapp.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:01.15

C:\git\testapps\cpumath>dir /s /b bin\*.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\api-ms-win-core-console-l1-1-0.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\api-ms-win-core-datetime-l1-1-0.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\api-ms-win-core-debug-l1-1-0.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\api-ms-win-core-errorhandling-l1-1-0.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\api-ms-win-core-file-l1-1-0.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\api-ms-win-core-file-l1-2-0.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\api-ms-win-core-file-l2-1-0.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\api-ms-win-core-handle-l1-1-0.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\api-ms-win-core-heap-l1-1-0.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\api-ms-win-core-interlocked-l1-1-0.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\api-ms-win-core-libraryloader-l1-1-0.dll
<snip/> -- many more lines follow -- including the application files

C:\git\testapps\cpumath>dir /s /b bin\cpu*
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\cpumath.deps.json
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\cpumath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\cpumath.exe
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\cpumath.pdb
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\cpumath.runtimeconfig.dev.json
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\cpumath.runtimeconfig.json
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\win-x64\CpuMathNative.dll

C:\git\testapps\cpumath>dir /s /b bin\*.so bin\*.dylib
File Not Found

The resulting application contains the application, architecture-specific dependent binaries and an architecture-specific .NET Core runtime.

Architecture-specific, self-contained applications

Like we did with framework-dependent applications in the prior example, we might want build an application as self-contained for multiple architectures. Let's try that.

C:\git\testapps\cpumath>type cpumath.csproj
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.0</TargetFramework>
    <RuntimeIdentifiers>linux-x64;win-x64</RuntimeIdentifiers>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.ML.CpuMath" Version="0.11.0" />
  </ItemGroup>

</Project>

C:\git\testapps\cpumath>dotnet build
Microsoft (R) Build Engine version 16.1.21-preview+gc36f772a85 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Persisting no-op dg to C:\git\testapps\cpumath\obj\cpumath.csproj.nuget.dgspec.json
  Restore completed in 194.37 ms for C:\git\testapps\cpumath\cpumath.csproj.
C:\Program Files\dotnet\sdk\3.0.100-preview4-010761\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(151,5): message NETSDK1057: You are using a preview version of .NET Core. See: https://aka.ms/dotnet-core-preview [C:\git\testapps\cpumath\cpumath.csproj]
  cpumath -> C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\cpumath.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:01.32

C:\git\testapps\cpumath>dir /s /b bin\*.dll bin\*.exe bin\*.so bin\*.dylib
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\cpumath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\Microsoft.ML.CpuMath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\cpumath.exe
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\runtimes\linux-x64\native\libCpuMathNative.so
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\runtimes\osx-x64\native\libCpuMathNative.dylib
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\runtimes\win-x64\native\CpuMathNative.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\runtimes\win-x86\native\CpuMathNative.dll

This doesn't produce the result we wanted. Let's try again.

C:\git\testapps\cpumath>type cpumath.csproj
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.0</TargetFramework>
    <RuntimeIdentifiers>linux-x64;win-x64</RuntimeIdentifiers>
    <SelfContained>true</SelfContained>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.ML.CpuMath" Version="0.11.0" />
  </ItemGroup>

</Project>

C:\git\testapps\cpumath>dotnet build
Microsoft (R) Build Engine version 16.1.21-preview+gc36f772a85 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Persisting no-op dg to C:\git\testapps\cpumath\obj\cpumath.csproj.nuget.dgspec.json
  Restore completed in 192.85 ms for C:\git\testapps\cpumath\cpumath.csproj.
C:\Program Files\dotnet\sdk\3.0.100-preview4-010761\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(127,5): error NETSDK1031: It is not supported to build or publish a self-contained application without specifying a RuntimeIdentifier.  Please either specify a RuntimeIdentifier or set SelfContained to false. [C:\git\testapps\cpumath\cpumath.csproj]

Build FAILED.

C:\Program Files\dotnet\sdk\3.0.100-preview4-010761\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(127,5): error NETSDK1031: It is not supported to build or publish a self-contained application without specifying a RuntimeIdentifier.  Please either specify a RuntimeIdentifier or set SelfContained to false. [C:\git\testapps\cpumath\cpumath.csproj]
    0 Warning(s)
    1 Error(s)

Time Elapsed 00:00:00.72

Intuitively, I expected RuntimeIdentifiers to have similar behavior as TargetFrameworks, as is demonstrated in the example below (which builds the application for multiple TargetFrameworks with a single build command).

C:\git\testapps\cpumath>type cpumath.csproj
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFrameworks>netcoreapp3.0;netcoreapp2.1</TargetFrameworks>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.ML.CpuMath" Version="0.11.0" />
  </ItemGroup>

</Project>

C:\git\testapps\cpumath>dotnet build
Microsoft (R) Build Engine version 16.1.21-preview+gc36f772a85 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Persisting no-op dg to C:\git\testapps\cpumath\obj\cpumath.csproj.nuget.dgspec.json
  Restore completed in 114.14 ms for C:\git\testapps\cpumath\cpumath.csproj.
C:\Program Files\dotnet\sdk\3.0.100-preview4-010761\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(151,5): message NETSDK1057: You are using a preview version of .NET Core. See: https://aka.ms/dotnet-core-preview [C:\git\testapps\cpumath\cpumath.csproj]
C:\Program Files\dotnet\sdk\3.0.100-preview4-010761\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(151,5): message NETSDK1057: You are using a preview version of .NET Core. See: https://aka.ms/dotnet-core-preview [C:\git\testapps\cpumath\cpumath.csproj]
  cpumath -> C:\git\testapps\cpumath\bin\Debug\netcoreapp2.1\cpumath.dll
  cpumath -> C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\cpumath.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:01.17

C:\git\testapps\cpumath>dir /s /b bin\*.dll bin\*.exe bin\*.so bin\*.dylib
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.1\cpumath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\cpumath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\Microsoft.ML.CpuMath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\cpumath.exe
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\runtimes\linux-x64\native\libCpuMathNative.so
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\runtimes\osx-x64\native\libCpuMathNative.dylib
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\runtimes\win-x64\native\CpuMathNative.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\runtimes\win-x86\native\CpuMathNative.dll

This output looks very suspicions, having native assets only for one of the two TFMs.

.NET Framework application with native dependencies

We should take a look at what the behavior is like for using native dependencies with .NET Framework.

C:\git\testapps\cpumath>type cpumath.csproj
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFrameworks>netcoreapp3.0;net472</TargetFrameworks>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.ML.CpuMath" Version="0.11.0" />
  </ItemGroup>

</Project>

C:\git\testapps\cpumath>dotnet build
Microsoft (R) Build Engine version 16.1.46-preview+ge12aa7ba78 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restore completed in 219.86 ms for C:\git\testapps\cpumath\cpumath.csproj.
C:\Program Files\dotnet\sdk\3.0.100-preview4-011022\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(151,5): message NETSDK1057: You are using a preview version of .NET Core. See: https://aka.ms/dotnet-core-preview [C:\git\testapps\cpumath\cpumath.csproj]
C:\Program Files\dotnet\sdk\3.0.100-preview4-011022\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(151,5): message NETSDK1057: You are using a preview version of .NET Core. See: https://aka.ms/dotnet-core-preview [C:\git\testapps\cpumath\cpumath.csproj]
  cpumath -> C:\git\testapps\cpumath\bin\Debug\net472\cpumath.exe
  cpumath -> C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\cpumath.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:01.19

C:\git\testapps\cpumath>dir /s /b bin\*.dll bin\*.exe bin\*.so bin\*.dylib
C:\git\testapps\cpumath\bin\Debug\net472\Microsoft.ML.CpuMath.dll
C:\git\testapps\cpumath\bin\Debug\net472\System.Buffers.dll
C:\git\testapps\cpumath\bin\Debug\net472\System.Memory.dll
C:\git\testapps\cpumath\bin\Debug\net472\System.Numerics.Vectors.dll
C:\git\testapps\cpumath\bin\Debug\net472\System.Runtime.CompilerServices.Unsafe.dll
C:\git\testapps\cpumath\bin\Debug\net472\cpumath.exe
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\cpumath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\Microsoft.ML.CpuMath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\cpumath.exe
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\runtimes\linux-x64\native\libCpuMathNative.so
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\runtimes\osx-x64\native\libCpuMathNative.dylib
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\runtimes\win-x64\native\CpuMathNative.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp3.0\runtimes\win-x86\native\CpuMathNative.dll

This build results in two applications, one targeting .NET Framework 4.7.2 and the other targeting .NET Core 3.0. It is plain to see that the .NET Framework layout is simpler, aligning closest with the architecture-specific framework-dependent model described earlier.

I should note that the problem space of managing native dependencies (at least in theory) has been easier to manage with .NET Framework since .NET Framework applications only target Windows and by default run as 32-bit applications, while at the same time being supported on both 32- and 64-bit Windows (including ARM64).

.NET Core 2.2 Behavior

.NET Core 2.2 has different behavior than .NET Core 3.0 in multiple ways. It is useful to define this behavior as the baseline experience for considering compatibility burden. Let's take a look.

Let's produce an architecture-neutral application with build.

C:\git\testapps\cpumath>dotnet --version
2.2.105

C:\git\testapps\cpumath>dotnet build
Microsoft (R) Build Engine version 15.9.20+g88f5fadfbe for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restoring packages for C:\git\testapps\cpumath\cpumath.csproj...
  Generating MSBuild file C:\git\testapps\cpumath\obj\cpumath.csproj.nuget.g.props.
  Generating MSBuild file C:\git\testapps\cpumath\obj\cpumath.csproj.nuget.g.targets.
  Restore completed in 241.38 ms for C:\git\testapps\cpumath\cpumath.csproj.
  cpumath -> C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\cpumath.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:01.23

C:\git\testapps\cpumath>dir /s /b bin\*.dll bin\*.exe bin\*.so bin\*.dylib
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\cpumath.dll

The resulting application is framework-dependent. It is very small because it relies on loading any dependent binaries from the NuGet cache (which will have already been restored). I stated above that we were building an architecture-neutral application, in the sense that no architecture was specified, but the resulting application can be thought of as architecture-specific because it is only intended to run on the developer machine (by virtue of not being complete, even for a framework-dependent application) and will only load native binaries from the NuGet cache for the current operating system and architecture.

Let's produce an architecture-specific application with build. In short, it doesn't do anything useful.

C:\git\testapps\cpumath>dotnet --version
2.2.105

C:\git\testapps\cpumath>dotnet build -r win-x64
Microsoft (R) Build Engine version 15.9.20+g88f5fadfbe for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restoring packages for C:\git\testapps\cpumath\cpumath.csproj...
  Generating MSBuild file C:\git\testapps\cpumath\obj\cpumath.csproj.nuget.g.props.
  Generating MSBuild file C:\git\testapps\cpumath\obj\cpumath.csproj.nuget.g.targets.
  Restore completed in 246.19 ms for C:\git\testapps\cpumath\cpumath.csproj.
  cpumath -> C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\win-x64\cpumath.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:01.59

C:\git\testapps\cpumath>dir /s /b bin\*.dll bin\*.exe bin\*.so bin\*.dylib
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\win-x64\cpumath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\win-x64\hostfxr.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\win-x64\hostpolicy.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\win-x64\cpumath.exe

The resulting application isn't correct or useful. This result is a random set of files.

Let's produce an architecture-neutral application with publish.

C:\git\testapps\cpumath>dotnet --version
2.2.105

C:\git\testapps\cpumath>dotnet publish
Microsoft (R) Build Engine version 15.9.20+g88f5fadfbe for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restoring packages for C:\git\testapps\cpumath\cpumath.csproj...
  Generating MSBuild file C:\git\testapps\cpumath\obj\cpumath.csproj.nuget.g.props.
  Generating MSBuild file C:\git\testapps\cpumath\obj\cpumath.csproj.nuget.g.targets.
  Restore completed in 233.52 ms for C:\git\testapps\cpumath\cpumath.csproj.
  cpumath -> C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\cpumath.dll
  cpumath -> C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\publish\

C:\git\testapps\cpumath>dir /s /b bin\*.dll bin\*.exe bin\*.so bin\*.dylib
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\cpumath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\publish\cpumath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\publish\Microsoft.ML.CpuMath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\publish\runtimes\linux-x64\native\libCpuMathNative.so
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\publish\runtimes\osx-x64\native\libCpuMathNative.dylib
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\publish\runtimes\win-x64\native\CpuMathNative.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\publish\runtimes\win-x86\native\CpuMathNative.dll

The resulting application is architecture-neutral framework-dependent, which aligns closely with the same behavior as dotnet build and dotnet publish with .NET Core 3.0.

Let's produce an architecture-specific application with publish.

C:\git\testapps\cpumath>dotnet --version
2.2.105

C:\git\testapps\cpumath>dotnet publish -r win-x64
Microsoft (R) Build Engine version 15.9.20+g88f5fadfbe for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restoring packages for C:\git\testapps\cpumath\cpumath.csproj...
  Generating MSBuild file C:\git\testapps\cpumath\obj\cpumath.csproj.nuget.g.props.
  Generating MSBuild file C:\git\testapps\cpumath\obj\cpumath.csproj.nuget.g.targets.
  Restore completed in 272.61 ms for C:\git\testapps\cpumath\cpumath.csproj.
  cpumath -> C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\win-x64\cpumath.dll
  cpumath -> C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\win-x64\publish\

C:\git\testapps\cpumath>dir /s /b bin\*.dll bin\*.exe bin\*.so bin\*.dylib
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\win-x64\cpumath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\win-x64\hostfxr.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\win-x64\hostpolicy.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\win-x64\cpumath.exe
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\win-x64\publish\api-ms-win-core-console-l1-1-0.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\win-x64\publish\api-ms-win-core-datetime-l1-1-0.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\win-x64\publish\api-ms-win-core-debug-l1-1-0.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\win-x64\publish\api-ms-win-core-errorhandling-l1-1-0.dll
<snip/> -- many more lines follow -- including the application files

The resulting application is architecture-specific and self-contained. This is the same as it would be with .NET Core 3.0.

Let's produce an architecture-specific framework-dependent application with publish.

C:\git\testapps\cpumath>dotnet --version
2.2.105

C:\git\testapps\cpumath>dotnet publish -r win-x64 --self-contained false
Microsoft (R) Build Engine version 15.9.20+g88f5fadfbe for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restoring packages for C:\git\testapps\cpumath\cpumath.csproj...
  Generating MSBuild file C:\git\testapps\cpumath\obj\cpumath.csproj.nuget.g.props.
  Generating MSBuild file C:\git\testapps\cpumath\obj\cpumath.csproj.nuget.g.targets.
  Restore completed in 256.12 ms for C:\git\testapps\cpumath\cpumath.csproj.
  cpumath -> C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\win-x64\cpumath.dll
  cpumath -> C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\win-x64\publish\

C:\git\testapps\cpumath>dir /s /b bin\*.dll bin\*.exe bin\*.so bin\*.dylib
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\win-x64\cpumath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\win-x64\cpumath.exe
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\win-x64\publish\cpumath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\win-x64\publish\CpuMathNative.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\win-x64\publish\Microsoft.ML.CpuMath.dll
C:\git\testapps\cpumath\bin\Debug\netcoreapp2.2\win-x64\publish\cpumath.exe

The resulting application is architecture-specific framework-dependent, targeting Windows x64 in this example.