Skip to content

Instantly share code, notes, and snippets.

@richlander
Created July 23, 2019 16:32
  • Star 5 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save richlander/63645f12d1bbe697fe5dc58df45862b1 to your computer and use it in GitHub Desktop.
.NET Core 3.0 SDK Size Improvements

.NET Core 3.0 SDK Size Improvements

The .NET Core SDK is significantly smaller with .NET Core 3.0. The primary reason is that we changed the way we construct the SDK, by moving to purpose-built “packs” of various kinds (reference assemblies, frameworks, templates). In previous versions (including .NET Core 2.2), we constructed the SDK from NuGet packages, which included many artifacts that were not required and wasted a lot of space.

The following sections demonstrate the size improvements for Windows, Linux and macOS, including container delivery. They detail the process and commands that were used to determine the product sizes, enabling you to reproduce the same results in your own environment. To keep thing simple, zips and tar balls were downloaded from dotnet/core-sdk as opposed to the official installers.

Some readers will be shocked on how large the .NET Core 2.2 installer directory grows when the NuGetFallback archive is expanded to the NuGetFallBackFolder. We maximally compressed this archive to make the .NET Core SDK installer/zip smaller. IL compresses really well, which helps to explain the degree of expansion.

Summary of Improvements

.NET Core 3.0 SDK Size (size change in brackets)

Operating System Installer Size (change) On-disk Size (change)
Windows 164MB (-440KB; 0%) 441MB (-968MB; -68.7%)
Linux 115MB (-55MB; -32%) 332MB (-1068MB; -76.2%)
macOS 118MB (-51MB; -30%) 337MB (-1063MB; -75.9%)

The size improvements for Linux and macOS are dramatic. The improvement for Windows is smaller because we have added WPF and Windows Forms as part of .NET Core 3.0. It’s amazing that we added WPF and Windows Forms in 3.0 and the installer is still (a little bit) smaller.

You can see the same benefit with .NET Core SDK Docker images (here, limited to x64 Debian and Alpine).

Distro 2.2 Size (Compressed; Uncompressed) 3.0 Size (Compressed; Uncompressed)
Debian 598MB; 1.74GB 264MB; 706MB
Alpine 493MB; 1.48GB 148MB; 422MB

Note: .NET Core Runtime Docker images are much smaller than the SDK (both in 2.2 and 3.0).

.NET Core SDK for Windows

Note: Recent Windows 10 releases include curl and tar, which are used below. Alternative tools/methods will be needed for earlier Windows versions.

Download

C:\temp>curl -o dotnet22.zip https://dotnetcli.blob.core.windows.net/dotnet/Sdk/release/2.2.3xx/dotnet-sdk-latest-win-x64.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  157M  100  157M    0     0  26.2M      0  0:00:06  0:00:06 --:--:-- 25.9M
C:\temp>curl -o dotnet30.zip https://dotnetcli.blob.core.windows.net/dotnet/Sdk/release/3.0.1xx/dotnet-sdk-latest-win-x64.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  156M  100  156M    0     0  17.4M      0  0:00:09  0:00:09 --:--:-- 13.6M

List files sizes

C:\temp>dir dot*
 Volume in drive C is Windows
 Volume Serial Number is 384B-0B6E

 Directory of C:\temp

07/22/2019  09:27 AM       165,048,802 dotnet22.zip
07/22/2019  09:28 AM       164,558,259 dotnet30.zip

Unpack

C:\temp>mkdir dotnet22
C:\temp>mkdir dotnet30
C:\temp>tar -zxf dotnet22.zip -C dotnet22
C:\temp>tar -zxf dotnet30.zip -C dotnet30

List directory sizes

C:\temp>dir /s dotnet22
     Total Files Listed:
             945 File(s)    331,044,056 bytes
             548 Dir(s)  102,613,372,928 bytes free
C:\temp>dir /s dotnet30
     Total Files Listed:
            2718 File(s)    441,104,628 bytes
            1160 Dir(s)  102,616,453,120 bytes free

“First run” for .NET Core 2.2

C:\temp>set DOTNET_MULTILEVEL_LOOKUP=0
C:\temp>dotnet22\dotnet.exe new
Configuring...
--------------
A command is running to populate your local package cache to improve restore speed and enable offline access. This command takes up to one minute to complete and only runs once.
Decompressing 100% 5857 ms
Expanding 100% 23611 ms

Note: dotnet new produces a lot of extra console output that was elided for the sake of easy reading.

Note: The "first run" concept doesn't exist with .NET Core 3.0 (that's a big part of this overall improvement).

Note: See dotnet for more information on the DOTNET_MULTILEVEL_LOOKUP environment variable (only applies to Windows).

Re-list directory sizes

dir /s dotnet22
     Total Files Listed:
           11381 File(s)  1,409,344,902 bytes
           25679 Dir(s)  101,508,927,488 bytes free
C:\temp>dir /s dotnet30
     Total Files Listed:
            2718 File(s)    441,104,628 bytes
            1160 Dir(s)  101,507,854,336 bytes free

Note: dir /s produces a lot of extra console output that was elided for the sake of easy reading.

.NET Core SDK for Linux

Download

rich@trenzalore:~$ curl -o dotnet22.tar.gz https://dotnetcli.blob.core.windows.net/dotnet/Sdk/release/2.2.3xx/dotnet-sd
k-latest-linux-x64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  162M  100  162M    0     0  13.9M      0  0:00:11  0:00:11 --:--:-- 15.9M
rich@trenzalore:~$ curl -o dotnet30.tar.gz https://dotnetcli.blob.core.windows.net/dotnet/Sdk/release/3.0.1xx/dotnet-sdk-latest-linux-x64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  109M  100  109M    0     0  20.2M      0  0:00:05  0:00:05 --:--:-- 22.0M

List file sizes

rich@trenzalore:~$ ls -l dot*
-rw-rw-rw- 1 rich rich 170192549 Jul 22 10:43 dotnet22.tar.gz
-rw-rw-rw- 1 rich rich 115335892 Jul 22 10:43 dotnet30.tar.gz

Unpack

rich@trenzalore:~$ mkdir dotnet22
rich@trenzalore:~$ tar -zxf dotnet22.tar.gz -C dotnet22
rich@trenzalore:~$ mkdir dotnet30
rich@trenzalore:~$ tar -zxf dotnet30.tar.gz -C dotnet30

List directory sizes

rich@trenzalore:~$ du -hs dotnet22 dotnet30
380M    dotnet22
332M    dotnet30

“First run” for .NET Core 2.2

rich@trenzalore:~$ dotnet22/dotnet new
Configuring...
--------------
A command is running to populate your local package cache to improve restore speed and enable offline access. This command takes up to one minute to complete and only runs once.
Decompressing 100% 6861 ms
Expanding 100% 71135 ms

Note: dotnet new produces a lot of extra console output that was elided for the sake of easy reading.

Note: The "first run" concept doesn't exist with .NET Core 3.0 (that's a big part of this overall improvement).

Re-list directory sizes

rich@trenzalore:~$ du -hs dotnet22 dotnet30
1.4G    dotnet22
332M    dotnet30

.NET Core SDK for macOS

Download

thundera:~ rich$ curl -o dotnet22.tar.gz https://dotnetcli.blob.core.windows.net/dotnet/Sdk/release/2.2.3xx/dotnet-sdk-latest-osx-x64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  161M  100  161M    0     0  21.1M      0  0:00:07  0:00:07 --:--:-- 21.6M
thundera:~ rich$ curl -o dotnet30.tar.gz https://dotnetcli.blob.core.windows.net/dotnet/Sdk/release/3.0.1xx/dotnet-sdk-latest-osx-x64.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  113M  100  113M    0     0  20.9M      0  0:00:05  0:00:05 --:--:-- 23.2M

List file sizes

thundera:~ rich$ ls -l dot*
-rw-r--r--  1 rich  staff  169797728 Jul 22 08:57 dotnet22.tar.gz
-rw-r--r--  1 rich  staff  118791630 Jul 22 08:58 dotnet30.tar.gz

Unpack

thundera:~ rich$ mkdir dotnet22
thundera:~ rich$ tar -zxf dotnet22.tar.gz -C dotnet22
thundera:~ rich$ mkdir dotnet30
thundera:~ rich$ tar -zxf dotnet30.tar.gz -C dotnet30

List directory sizes

thundera:~ rich$ du -hs dotnet22 dotnet30
339M dotnet22
337M dotnet30

“First run” for .NET Core 2.2

thundera:~ rich$ dotnet22/dotnet new
Configuring...
--------------
A command is running to populate your local package cache to improve restore speed and enable offline access. This command takes up to one minute to complete and only runs once.
Decompressing 100% 5010 ms
Expanding 100% 6243 ms

Note: dotnet new produces a lot of extra console output that was elided for the sake of easy reading.

Note: The "first run" concept doesn't exist with .NET Core 3.0 (that's a big part of this overall improvement).

Re-list directory sizes

thundera:~ rich$ du -hs dotnet22 dotnet30
1.4G dotnet22
337M dotnet30
thundera:~ rich$ 

.NET Core SDK for Docker

Pull images

docker pull mcr.microsoft.com/dotnet/core/sdk:2.2
docker pull mcr.microsoft.com/dotnet/core/sdk:2.2-alpine
docker pull mcr.microsoft.com/dotnet/core/sdk:3.0
docker pull mcr.microsoft.com/dotnet/core/sdk:3.0-alpine

Look at image sizes (uncompressed)

docker image ls mcr.microsoft.com/dotnet/core/sdk
REPOSITORY                          TAG                 IMAGE ID            CREATED             SIZE
mcr.microsoft.com/dotnet/core/sdk   2.2-alpine          010b0ec97feb        4 days ago          1.48GB
mcr.microsoft.com/dotnet/core/sdk   2.2                 2344653cce7d        4 days ago          1.74GB
mcr.microsoft.com/dotnet/core/sdk   3.0-alpine          475b34386410        4 days ago          424MB
mcr.microsoft.com/dotnet/core/sdk   3.0                 d8cf7ef43247        4 days ago          710MB

You can use docker image inspect to streamline the docker image output.

Get just the size for a single image (uncompressed)

docker image inspect --format="{{.Size}}" mcr.microsoft.com/dotnet/core/sdk:3.0-alpine
424262383

Get the size for multiple images, with a nice table (uncompressed)

docker images --format "table {{.Repository}}:{{.Tag}}\t {{.Size}}" mcr.microsoft.com/dotnet/core/sdk
REPOSITORY:TAG                                  SIZE
mcr.microsoft.com/dotnet/core/sdk:2.2-alpine    1.48GB
mcr.microsoft.com/dotnet/core/sdk:2.2           1.74GB
mcr.microsoft.com/dotnet/core/sdk:3.0-alpine    424MB
mcr.microsoft.com/dotnet/core/sdk:3.0           710MB

Get the size for an image (compressed)

The Microsoft Container Registry doesn't have a public API for determining image sizes. As a workaround, I uploaded the images to Docker Hub using the following round-about approach. You'll need to use your own DockerHub account instead of the one I'm using below.

docker login
docker pull mcr.microsoft.com/dotnet/core/sdk:3.0-alpine
docker tag mcr.microsoft.com/dotnet/core/sdk:3.0-alpine richlander/sdk:3.0-alpine
docker push richlander/sdk:3.0-alpine

I then repeated for the other images (minus the docker login, which is only required once). After that, I requested tag information from Docker Hub and used those sizes.

C:\Users\rich>curl -s -H "Authorization: JWT " "https://hub.docker.com/v2/repositories/richlander/sdk/tags/?page_size=100"
{"count": 4, "next": null, "previous": null, "results": [{"name": "2.2", "full_size": 598313550, "images": [{"size": 598313550, "architecture": "amd64", "variant": null, "features": null, "os": "linux", "os_version": null, "os_features": null}], "id": 63199109, "repository": 7443285, "creator": 1124295, "last_updater": 1124295, "last_updated": "2019-07-23T04:29:54.654360Z", "image_id": null, "v2": true}, {"name": "2.2-alpine", "full_size": 493602817, "images": [{"size": 493602817, "architecture": "amd64", "variant": null, "features": null, "os": "linux", "os_version": null, "os_features": null}], "id": 63198797, "repository": 7443285, "creator": 1124295, "last_updater": 1124295, "last_updated": "2019-07-23T04:24:20.929408Z", "image_id": null, "v2": true}, {"name": "3.0-alpine", "full_size": 148263985, "images": [{"size": 148263985, "architecture": "amd64", "variant": null, "features": null, "os": "linux", "os_version": null, "os_features": null}], "id": 63198713, "repository": 7443285, "creator": 1124295, "last_updater": 1124295, "last_updated": "2019-07-23T04:22:46.625796Z", "image_id": null, "v2": true}, {"name": "3.0", "full_size": 263928266, "images": [{"size": 263928266, "architecture": "amd64", "variant": null, "features": null, "os": "linux", "os_version": null, "os_features": null}], "id": 63198469, "repository": 7443285, "creator": 1124295, "last_updater": 1124295, "last_updated": "2019-07-23T04:18:55.138668Z", "image_id": null, "v2": true}]}

It is probably easiest to load the resulting JSON in an online JSON viewer.

Context: The Local Package Cache

The local package cache — the NuGetFallbackFolder — is no longer in use with .NET Core 3.0. This change is a major improvement to the initial experience using .NET Core and is the biggest win for reducing the amount of disk space .NET Core requires.

You’ve probably seen the following experience when you installed and .NET Core 2.2 or earlier versions for the first time:

Configuring...
--------------
A command is running to populate your local package cache to improve restore speed and enable offline access. This command takes up to one minute to complete and only runs once.
Decompressing 100% 5857 ms
Expanding 100% 23611 ms

With .NET Core 2.2, we copied a large number of NuGet packages into a local cache — the NuGetFallback folder — on your machine. Having the cache available made certain operations faster, limited the packages you needed to restore from NuGet.org and also provided an offline scenario (useful for laptops).

The SDK that you downloaded contained all of these packages in a maximally compressed file (with LZMA compression). The maximal compression significantly reduces the size of the SDK installer/zip, but is computationally expensive to uncompress (this is a typical tradeoff with comrpession). On slow hardware, it could take tens of seconds for this operation to complete. It was so bad on ARM hardware that we removed the local cache from those installers.

You can skip using the local cache by setting the DOTNET_SKIP_FIRST_TIME_EXPERIENCE=true before installing the .NET Core SDK. This environment variable is no longer honored with .NET Core 3.0 since the basis for the setting has “disappeared”.

After you’ve fully adopted .NET Core 3.0 and uninstalled all older .NET Core versions, you can feel free to delete the NuGetFallbackFolder. The uninstaller does not remove this folder, so it will be left in place until explicitly deleted.

On Windows, you can find the NugetFallbackFolder at C:\Program Files\dotnet\sdk\NuGetFallbackFolder. It lives at similar locations (relative to the dotnet root) on macOS and Linux. This folder grows without bound. You can see that this folder is over 1GB on my desktop machine.

C:\>dir /s "C:\Program Files\dotnet\sdk\NuGetFallbackFolder"
     Total Files Listed:
           12517 File(s)  1,230,625,208 bytes

Closing

We learned a lot about building a whole new product — .NET Core — over the first four versions. We knew at the time that we didn’t have the best possible architecture in place for composing the product, but saw higher needs elsewhere, like establishing a compelling cross-platform and open source developer platform. With .NET Core 3.0, we’ve finally caught up on product composition and made the appropriate investments to resolve those challenges.

A smaller product helps many scenarios, with Docker containers and small devices being top of list. We’ve seen significant improvements in our own infrastructure, that uses Docker and/or ARM devices. We hope you see the same degree of improvements when you adopt .NET Core 3.0.

@mark-edf
Copy link

mark-edf commented Oct 9, 2019

this is really helpful, thanks! Unfortunately the removal of NuGetFallbackFolder has scuppered the cunning method we used to use to get msbuild to work on our build server. We used to add the NuGetFallbackFolder path to our nuget.config and that meant that msbuild could happily build things like .net standard projects.

We still use msbuild rather than dotnet build or dotnet msbuild as it works for both .net core and .net framework projects so we didn't have to do anything special to detect which we were working with.

I'm hoping to figure out a workaround, as we do hope to move to .net core 3 fully over time since it looks like it supports everything we need :)

@richlander
Copy link
Author

Want to reach out to me at rlander @ ms and I can connect you with the right people?

@mungojam
Copy link

thanks, I've sent you an email. We ended up installing .net core 2.1 SDK so that we would get the NuGetFallbackFolder from that and that should carry us up to the point when we migrate our projects to .net core 3

@livarcocc
Copy link

You don't need the fallback folder to use msbuild to build netstandard. What errors do you get if you don't do that? What do your net standard projects look like?

You should be able to simply run msbuild.exe /restore /t:build to restore your projects and build.

@mungojam
Copy link

hopefully my email with the full details got through. The error I get is Unable to find package NETStandard.Library with version (>= 2.0.3)

@livarcocc
Copy link

Do you have nuget.org as one of the feeds available to NuGet during restore?

@mungojam
Copy link

No, it's an offline build server so we clear the existing feeds and just add our own internal one (and previously the fallback folder)

I guess the email with that detail didn't come through, I wondered if it might get filtered silently at our end

@livarcocc
Copy link

In that case, the right thing to do is for you to generate your own feed content and provide it to your servers if you want to be offline for TFMs that are not those shipping with the SDK you use. That means that a 3.0 SDK will work offline for 3.0, within the constraints that always existed before, like same architecture as the one the SDK targets and for FDD apps.

@mungojam
Copy link

For other's benefit, I ended up adding both 2.2 and 3.0 SDK and then running dotnet new with 2.2 in order for it to expand the package cache. It's just a bit easier than adding loads of packages to our internal repo which is getting slow as it is.

@victor-yarema
Copy link

victor-yarema commented Nov 2, 2021

In Ubuntu 20.04.3 LTS the prepopulated folder is located at /usr/share/dotnet/sdk/NuGetFallbackFolder. I was able to get it when I installed the official dotnet-sdk-2.1 package. Currently apt show dotnet-sdk-2.1 reports it is Version: 2.1.818-1. It may change in the future.
Some stats about it. The apparent size of it is around 1.1GB (1083736821 bytes) in 10574 files. There is one empty file with name "2.1.818.dotnetSentinel" and 351 packages folders.
For some packages it restores more than one version. For example:

microsoft.csharp/4.0.1
microsoft.csharp/4.3.0
microsoft.csharp/4.5.0

The amount of packages versions restored is 440.
One more interesting thing is that those are not only packages from microsoft (microsoft.*, netstandard.library, nuget.frameworks, runtime.*, system.*, windowsazure.storage). For example:

libuv/1.10.0
messagepack/1.7.3.4
newtonsoft.json.bson/1.0.1
newtonsoft.json/10.0.1
newtonsoft.json/11.0.2
newtonsoft.json/9.0.1
remotion.linq/2.2.0
sqlitepclraw.bundle_green/1.1.11
sqlitepclraw.core/1.1.11
sqlitepclraw.lib.e_sqlite3.linux/1.1.11
sqlitepclraw.lib.e_sqlite3.osx/1.1.11
sqlitepclraw.lib.e_sqlite3.v110_xp/1.1.11
sqlitepclraw.provider.e_sqlite3.netstandard11/1.1.11
stackexchange.redis.strongname/1.2.4

BTW, the last one mentioned is deprecated according to https://www.nuget.org/packages/StackExchange.Redis.StrongName . I didn't check the others.

@victor-yarema
Copy link

I am wondering why would anyone need and SDK instead of just a runtime in a Docker container. Using SDK for any real deployment is simply wrong by definition.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment