dlang-packaging-issues.md

D Distribution Issue List

This is a list of issues found while making the D ecosystem available on Linux distributions such as Debian and Fedora. It is a smaller, but priorized list of the general list of issues I find while working with D that can be viewed at dlang-quirks.md.

This list is ordered by priority, more pressing issues being at the top.

LDC, DMD, DUB, DLang

1. D has no stable ABI. This kind of enforces recompiling all D shared libraries and binaries in a Linux distribution with every new compiler version. This is possible, but a major pain which requires quite some amount of manpower (Debian does this for OCaml and Haskell) - getting people to use D if it is that much extra work is harder. It also means that you can not use the libraries which were compiled with LDC in an application that is compiled with GDC and vice versa. This is a real PITA for distros and users who might have binaries compiled with the "wrong" compiler. It reduces the usefulness of development D packages in distributions a lot, if you need to recompile everything again anyway. Static linking is also frowned upon by distributions, because it triggers rebuild cascades on every bugfix that we introduce, making devlivering security updates much harder and error-prone (embedded code copies are very difficult to handle). A stable D ABI, or at least a guarantee that the ABI will not change unless there is a good reason (and the ABI change will be announced in the release notes) would help a lot. Having a D ABI that is shared between DMD, LDC and GDC would be even better.

2. Dub doesn't use packages installed to system locations (dlang/dub#838) - this renders dub pretty much unusable for using it as part of the Linux distribution package tooling. Dub also writes into $HOME as part of the build process which is considered bad practice and is disallowed on build daemons at Debian (you need to add a hack for tools which expect $HOME to be available)

3. Dub: Can not version dependencies on other libraries (dlang/dub#906) - this makes it hard to determine compatibility if we package software written in D.

4. dub test overrides the binary created by dub build (prevents automatically running tests as part of the build process) (dlang/dub#840)

5. No working Make/Ninja compatible depfile generation in LDC, DMD or GDC - this is especially important when not using dub (see the dub issues) and some other build system (Automake/CMake/Meson/..) is used instead. See ldc-developers/ldc#1802 for the LDC bug report.

6. There is no dub install command (dlang/dub#839)

7. LDC doesn't support a lot of architectures / architecture support breaks from time to time. It would be quite nice to have D tools available on as many architectures as possible. (see ldc-developers/ldc#1636 for a bug report, build logs are here: https://buildd.debian.org/status/package.php?p=ldc)

GDC Packaging Issues

The GDC issues have been separated out to not clutter the priorized overview.

1. GDC is not part of official GCC. This makes it difficult (or actually sometimes impossible) to ship it with some Linux distributions, e.g. RHEL and Fedora. On Debian, we inject GDC into the GCC build, which is also a bit weird. Having GDC in GCC proper would also give D much higher visibility.

2. GDC only supports an ancient version of the D standard library, which has many nice classes and also bugfixes missing. Because of that, it is almost impossible to compile modern D applications with GDC, and when developing a new application, support for GDC needs to be added explicitly and handled with a lot of care. This is an issue LDC doesn't have, and which is fragmenting the D ecosystem.

3. GDC does not support creating shared libraries at time, which is a big deal for distros which need it to reduce duplicate code and make security fixes easier.

Changes

• DMD is completely free software now, there is no obstacle to ship it in distributions anymore.

Is the point number 1 still true in the context of packaging, please?

Is the point number 1 still true in the context of packaging, please?

The DMD backend is now entirely Boost-licensed and can be shipped on all Linux distributions:

http://forum.dlang.org/thread/oc8acc$1ei9$1@digitalmars.com

Note that some distributions like Arch Linux always used to ship DMD and there also has been an "official" Debian repository.
However, afaik DMD hasn't been packaged for Debian yet, but I guess that it will happen soon in the foreseeable future :)

dub is still a big problem, but too many people continue to think as a windows user which will pull their needed librairies into the user home directory ...
I use MakefileForD which provide a complete makefile template.
You need only to fill some field and run:

$ make build
$ make install

The provided makefile follow usual linux guideline such as the ability to do a:

• staged build using DESTDIR
• override a prefix, data dir, include dir, lib dir, bin dir and so on
• generate pkg-config file automatically
• build documentation

The problem around packaging will be still true for many years, no hope ...

What does Debian do for Rust?

Cargo seems to have many of the same "problems" as Dub. And yet Debian somehow packages lots of Rust software, surely?

Please keep in mind that this list is probably now extremely out-of-date and would really need an update!

Rust is also a real pain, but the short of it is that all Rust code is built statically, and since cargo supports local repositories packages just install the source code into system locations and it's then used from there. There is also some quite insane tooling available in form of debcargo to automate as much of the busywork as possible (a perk when a language is popular like Rust and people invest time into creating solutions like this)
See https://wiki.debian.org/Teams/RustPackaging/Policy and https://salsa.debian.org/rust-team/debcargo-conf/blob/master/README.rst for details.

The primary difference is really cargo allowing for a first-class system-wide repository of packages, while dub does not (or did not - I think dub has this now but last time I tested it it tried to place files in a read-only location and crashed).

Also, a ton of Rust-using projects, like Mesa and many GNOME projects like librsvg, use Meson as buildsystem, which simplifies things a lot. Packaging Rust still is pretty painful though, compared to a language like Python which also has a native package manager but also plays very well with packaging.

Thanks!

This page is still linked to from https://wiki.debian.org/D, if possible it may make sense to point out there that the information is outdated.

The main thing that stands out to me from reading this gist today is that some things listed here are now considered completely normal today, such as writing to $HOME, which I think pretty much every modern language's package manager does. The other thing is that, at least from what I can see, your (much appreciated) efforts have been to try to package D and D software more like C, however I think D is much closer to languages like Rust and Go than C. For example, little attention is paid to ABI compatibility, both between the language and software written in it, and between user libraries written in the language; so, static linking is the norm and going against it would be against most upstreams' expectations.

I agree that manpower is the bottleneck, and even so, I would really like to see another attempt to look at this from today's perspective. For example, if there are perhaps features that Dub could borrow from Cargo, instead of from C/C++ build systems, in order to have a packaging story closer to Rust or other languages with similar qualities. Will check out the links!

This page is still linked to from https://wiki.debian.org/D, if possible it may make sense to point out there that the information is outdated.

The Debian D team could use a lot more manpower to update things like this and modernize the tools it uses... TBH, I think with just a few small tweaks to dub, packaging D code should be even nicer than packaging Rust code - D has everything there, even Meson integration, and Debian's tools have evolved. LDC is also truly excellent as a compiler now, it is less of a pain than it was before.
With GDC we even have a D compiler pretty much everywhere now (unfortunately GDC still can't compile a lot what LDC can).

The main thing that stands out to me from reading this gist today is that some things listed here are now considered completely normal today, such as writing to $HOME, which I think pretty much every modern language's package manager does.

Writing to HOME is not an issue, but being unable to disable that is what the problem is - writing stuff to HOME introduces state, which is not what we want when trying to reproducibly build packages.
So, many modern package managers either have the option to disable that, write to a temporary location, or at the very least don't fail if HOME is inaccessible (which it deliberately is on package autobuilders). Some do have the option to deflect HOME to a different place, like cargo with the CARGO_HOME environment variable that Rust builds in Debian just set to a temporary location.

For example, little attention is paid to ABI compatibility, both between the language and software written in it, and between user libraries written in the language; so, static linking is the norm and going against it would be against most upstreams' expectations.

Indeed, and we do have the tools now to handle static builds across many packages - it's highly automated to do cascade rebuilds.

For example, if there are perhaps features that Dub could borrow from Cargo, instead of from C/C++ build systems, in order to have a packaging story closer to Rust or other languages with similar qualities.

A way to make dub use a temporary location other than HOME, either by an env var like CARGO_HOME which sets a location, or just by a variable like DUB_USE_TMP_HOME=y where it uses a location in /tmp would be nice. Also, a way to load packages from a standardized system location (/usr/share/dub/<package>-<version>?) and a standardized command to install them there would be extremely nice - also, dub must not try to write into those locations, of course. Apart from that, I don't think there's much we would need (we could even set HOME already to fool dub, but that is a pretty messy approach).

In general I think the most helpful thing is standardization: If dub has standard system locations where it loads from, and there's established ways how D projects handle things, it will be super easy to write tooling to wrap them in packages (of course, someone would still need to do that - I do have a lot less time currently (and the "D time" is use on e.g. updating LDC), but I definitely could sponsor new packages into Debian if someone else worked on them).

The Debian D team could use a lot more manpower to update things like this and modernize the tools it uses...

How do I contact the Debian D team?

I tried to reach out to someone who was packaging other D software and got no response. There is also a RFP for a D program I maintain but it did not seem to get any packagers' interest either.

So, many modern package managers either have the option to disable that, write to a temporary location, or at the very least don't fail if HOME is inaccessible (which it deliberately is on package autobuilders). Some do have the option to deflect HOME to a different place, like cargo with the CARGO_HOME environment variable that Rust builds in Debian just set to a temporary location.

I see, thanks. My naive question would then be, why not just set HOME as part of the package manager invocation? This is what I do in my Arch Linux package.

But in any case, I see that Dub now does support DUB_HOME!

Also, a way to load packages from a standardized system location (/usr/share/dub/<package>-<version>?) and a standardized command to install them there would be extremely nice - also, dub must not try to write into those locations, of course.

I think DPATH=/usr/share dub fetch package-name-here && DPATH=/usr/share dub build package-name-here will do that (but only from a very quick test!)

How do I contact the Debian D team?

I fear the D team might just be me, as far as "active" members count... But since we don't have a mailinglist, you should be able to reach everyone via the E-Mail address on https://tracker.debian.org/teams/d-team/
All packaging is also maintained in Git at https://salsa.debian.org/d-team

There is also a RFP for a D program I maintain but it did not seem to get any packagers' interest either.

RFPs are really just that, requests for a new package by someone who would like it to be present in Debian. It still needs someone to invest the time and energy to package it.

I see, thanks. My naive question would then be, why not just set HOME as part of the package manager invocation?

It works of course, but it's also fairly messy because a lot of other tools and scripts may also use HOME and we do not accidentally want to set a home directory for those. A site-specific home is a lot cleaner, if one needs to exist.

But in any case, I see that Dub now does support DUB_HOME!

Indeed! - That is really great, we can set this globally in the dlang helper scripts, so packagers don't need to think about it anymore and can maybe just use dub as soon as it recognizes a system-wide registry of D packages.

I think DPATH=/usr/share dub fetch package-name-here && DPATH=/usr/share dub build package-name-here will do that (but only from a very quick test!)

It would be super nice if a command like dub --skip-registry=all --registry=local --cache=local did:

• Skip all remote registries and search for matching packages in a well-known local default location in /usr/lib or /usr/share
• (If possible) did not copy files out of the local registry (that may not be how dub works though...)
• Build the dependencies and actual application in a local directory in the project's directory, or in DUB_HOME if needed

This may actually already be implemented - I'd need a small dub project to test it :)
If a dub install would place binaries in %prefix%/bin etc. and if a dub install-devel would place the whole dub package in a well-known shared location on the system, it would be extra sweet, but that's something the distros can also do very easily.

With a framework like this, a lot of code could be shared between distros even (and much code may not be necessary at all).
TBH, I think the only thing stopping better D support in Debian and other distros is simply lack of manpower (and the occasional compiler bug, which is super stressful to handle, which is the reason why dub isn't up-to-date in Debian right now due to dlang/dub#2577, and the reason I dread LDC updates - something related to template instantiations almost always breaks. To be fair, it did get a lot better already though, the last LDC upgrade was completely painless).

RFPs are really just that, requests for a new package by someone who would like it to be present in Debian. It still needs someone to invest the time and energy to package it.

Right, sorry I should have mentioned that my attempt to reach out was to offer any help with packaging that I could provide from my side (package-wise or D-wise), but I guess it makes sense that these are not really monitored.

Skip all remote registries and search for matching packages in a well-known local default location in /usr/lib or /usr/share

I think --skip-registry=all --cache=system will do this.

I found two things that I should point out:

• I think registries in Dub are fundamentally online services which store packages, so there can't be a local registry (unless it's something like a local web server).
• I don't think Dub has a command that would put a local package into the local cache; it can only put things into the cache which it downloaded from a registry. Perhaps this is something that could be easy to implement.

However, it does have an add-local command. I'm not sure if it is suitable or not, though.

A pertinent question here would be how other languages do it. E.g., for Pypi packages, does Debian get the package from pypi.org, or from the upstream (git repository etc.)? If it is acceptable to use dub fetch to obtain packages' source code and populate the --cache=system cache then that may make things simpler.

Build the dependencies and actual application in a local directory in the project's directory, or in DUB_HOME if needed

I think that's how Dub already works, it creates a ./.dub directory for temporary files and puts the final binary in the current directory.

(and the occasional compiler bug, which is super stressful to handle, which is the reason why dub isn't up-to-date in Debian right now due to dlang/dub#2577

That sounds like something that may make sense to push to upstream CI systems. If we can catch changes that break Debian packaging before they are merged or at least released, I think that would make everyone happier.

---

I don't know if this is helpful at all, but I played a bit with a docker.io/debian:latest container to see how far I can get. I got my program to build with two methods, both of which I think satisfy some of the above constraints. First method is to clone the dependencies and add them with dub add-local:

Dockerfile

FROM docker.io/debian:latest

RUN apt-get update
RUN apt-get install -y curl xz-utils git mc gcc
RUN apt-get install -y libncursesw5-dev libz-dev
RUN curl -fsS https://dlang.org/install.sh | bash -s dmd
COPY build.sh .
RUN ./build.sh

build.sh

#!/bin/bash

# shellcheck disable=SC1090
source ~/dlang/dmd-2.103.1/activate

set -eEuo pipefail

mkdir /d
cd /d

# dub.selections.json is:
# {
#  	"fileVersion": 1,
#  	"versions": {
#  		"ae": "0.0.3284",
#  		"btrfs": "0.0.19",
#  		"ncurses": "1.0.0",
#  		"emsi_containers": "0.9.0"
#  	}
# }

git clone --depth=1 -b v0.0.3236 https://github.com/CyberShadow/ae ae
git clone --depth=1 -b v0.0.19 https://github.com/CyberShadow/d-btrfs btrfs
git clone --depth=1 -b v1.0.0 https://github.com/D-Programming-Deimos/ncurses ncurses
git clone --depth=1 -b v0.9.0 https://github.com/dlang-community/containers emsi_containers

git clone --depth=1 -b v0.5.1 https://github.com/CyberShadow/btdu
cd btdu
DUB_HOME=$PWD dub add-local ../ae
DUB_HOME=$PWD dub add-local ../btrfs
DUB_HOME=$PWD dub add-local ../ncurses
DUB_HOME=$PWD dub add-local ../emsi_containers
DUB_HOME=$PWD dub build --skip-registry=all --cache=system

Second method is to use dub fetch with --cache=system:

build.sh (second half only)

export DUB_HOME=$PWD

dub fetch --cache=system ae@0.0.3236
dub fetch --cache=system btrfs@0.0.19
dub fetch --cache=system ncurses@1.0.0
dub fetch --cache=system emsi_containers@0.9.0

git clone --depth=1 -b v0.5.1 https://github.com/CyberShadow/btdu
cd btdu
dub build --skip-registry=all --cache=system

BTW, are you in contact with the D foundation? I think they are interested in allocating resources for D packaging.

A pertinent question here would be how other languages do it. E.g., for Pypi packages, does Debian get the package from pypi.org, or from the upstream (git repository etc.)?

Either, it is the same code (ignoring wheel/binary distributions on PyPI which we can't use). The key difference is that these packages are natively installed into /usr/lib/python3/dist-packages/ by whatever tool the Python community has come up with this time to handle packaging (setuptools and things like poetry seem to be the hot new thing right now...), and the packages in the location are natively picked up by any Python package before they even try to reach out to the web.

If it is acceptable to use dub fetch to obtain packages' source code and populate the --cache=system cache then that may make things simpler.

It is not, no internet access is permitted at build-time at all. This ensures that we can reproducibly build all packages in the distribution and that the whole thing depends just on itself with no external service outages influencing anything, and no security issues in external tools having any impact.
Also, vendoring code is frowned upon and you usually need to justify code copies. So if your D project foo-d depends on dependency bar-d, you should not embed the code of bar-d in the foo-d codebase, but instead create a separate package for bar-d that foo-d can the (build)depend on. That allows much easier tracking of what code is available in the distribution. Having multiple versions of a dependency is usually avoided, but occasionally we do have multiple ones (a very notable example is C++ Boost libraries, or even LLVM where multiple versions are shipped).
There's an Upstream Guide, but that one is quite extensive and only a few sections may be relevant here: https://wiki.debian.org/UpstreamGuide

What you can do of course is set up a symlink farm where you link stuff from system locations into the dub cache, that's really tedious and messy though.
There's one D package, Lix, which embedded everything and added a bunch of hacks, but an approach like this isn't really scalable and a lot of tedious work to keep up to date (which is probably why the package is dropped from Debian releases and didn't receive much maintenance): https://salsa.debian.org/games-team/lix/-/tree/master/

However, it does have an add-local command. I'm not sure if it is suitable or not, though.

Would be cool if we could set that globally somehow - keeping modifyable state is always quite messy.

I think that's how Dub already works, it creates a ./.dub directory for temporary files and puts the final binary in the current directory.

My knowledge might be super obsolete here now, but last time I tried this, Dub attempted to write into locations in /usr/share where its system-wide packages were.

That sounds like something that may make sense to push to upstream CI systems. If we can catch changes that break Debian packaging before they are merged or at least released, I think that would make everyone happier.

Absolutely! I have this on my TODO list, but it needs to work again first - merging a patch that intentionally breaks CI would be a bit odd ^^ - At Debian we also continuously test if dub still works, which is quite useful and has found bugs in the past: https://ci.debian.net/packages/d/dub/testing/amd64/ (it also shows when code.dlang.org is down ^^)

I don't know if this is helpful at all, but I played a bit with a docker.io/debian:latest container to see how far I can get. I got my program to build with two methods, both of which I think satisfy some of the above constraints.

You would need to have dpkg build a package to really see what works and what doesn't. The first method may be viable actually, if dub does not write into the locations of its dependencies. It would still be nice if one wouldn't need to manually have to specify the locations of each dependency, but it would work.

If you had a small-ish example, maybe a D project with just one or two dub dependencies, I could maybe rig up a small packaging example. Debian packaging actually doesn't need a ton of files or code to just get a working package out (but people new to it are usually intimidated by the many files, which are needed to get the package into Debian proper. Most of them are for QA and license review though).

BTW, are you in contact with the D foundation? I think they are interested in allocating resources for D packaging.

No, I have been much less involved with D recently (even ported a large project away from it, to Python & C++, unfortunately) - but I think improving packaging would be a boon for D, increasing its visibility and strengthening the ecosystem. Anything that makes D more accessible and easy to use in any form would be a lot of help :-)

Btw, one thing that's obvious but also needs mentioning: When configuring dub to work with apps & libraries packaged for Debian/Ubuntu, we of course must not break it for people who need it to download from online repositories. So ideally, any solution would work for both :)

The key difference is that these packages are natively installed into /usr/lib/python3/dist-packages/ by whatever tool the Python community has come up with this time to handle packaging (setuptools and things like poetry seem to be the hot new thing right now...), and the packages in the location are natively picked up by any Python package before they even try to reach out to the web.

That does sound aligned to what dub --cache=system does, I think... so then the next question is, what options are there for downloading the package? Does it have to be a Debian-provided tool or can it be a language-provided tool (in our case, dub fetch)?

It is not, no internet access is permitted at build-time at all. This ensures that we can reproducibly build all packages in the distribution and that the whole thing depends just on itself with no external service outages influencing anything, and no security issues in external tools having any impact.

OK, then can we use dub fetch to just get the package, save / package it in Debian, and then continue using that for building other packages? So, dub fetch would then be the only step requiring Internet access.

Having multiple versions of a dependency is usually avoided, but occasionally we do have multiple ones (a very notable example is C++ Boost libraries, or even LLVM where multiple versions are shipped).

What is done for languages like Rust/Cargo, is it the same approach? I think a packaging scheme which does not allow more than one version of a package is incompatible with the idea of lock files, which are used by pretty much all language package managers. D (or at least Dub) is much closer to Rust/Node/etc. in that it has strong dependency declaration via lock files (though in case of Dub, they are optional, but generally strongly encouraged). So, Debian would need to either decide to disregard project lock files (and deal with the fallout of possible incompatibilities) or package each version individually.

There's an Upstream Guide, but that one is quite extensive and only a few sections may be relevant here: https://wiki.debian.org/UpstreamGuide

Thank you, I will have a look.

Would be cool if we could set that globally somehow - keeping modifyable state is always quite messy.

add-local actually does have a global effect. However, it is stored in a single configuration file (local-packages.json), so you can't really include its result in a package.

If for whatever reason we decide that the --cache=system cache is unusable for Debian packages, perhaps we could just store Dub packages in an arbitrary location and have dependents use add-local to build them?

My knowledge might be super obsolete here now, but last time I tried this, Dub attempted to write into locations in /usr/share where its system-wide packages were.

OK, I reproduced it! Looks like this is still an issue, with --cache=system it will try to write there. Will file an issue / poke at Dub to see what can be done about this.

Btw, one thing that's obvious but also needs mentioning: When configuring dub to work with apps & libraries packaged for Debian/Ubuntu, we of course must not break it for people who need it to download from online repositories. So ideally, any solution would work for both :)

OK... but I think we need to figure out some conceptual details, like the versioning thing... like, if Debian installs one version of a Dub package, but I want to build a program that wants a different version of that package, what happens?

That does sound aligned to what dub --cache=system does, I think... so then the next question is, what options are there for downloading the package? Does it have to be a Debian-provided tool or can it be a language-provided tool (in our case, dub fetch)?

No downloads of anything are allowed at built-time at all. Internet access is disabled at that stage. So any dependency has to be installed before the build, and as a Debian package, for dub to find.
So the only thing dub fetch could do is copy source code from a system location somewhere else (but at that point, it might just as well use the code directly, without making a copy).

What is done for languages like Rust/Cargo, is it the same approach? I think a packaging scheme which does not allow more than one version of a package is incompatible with the idea of lock files, which are used by pretty much all language package managers. D (or at least Dub) is much closer to Rust/Node/etc. in that it has strong dependency declaration via lock files (though in case of Dub, they are optional, but generally strongly encouraged). So, Debian would need to either decide to disregard project lock files (and deal with the fallout of possible incompatibilities) or package each version individually.

The latter is exactly what happens: When packaging so much software, keeping each small version of each package around is just insane, especially if you also need to provide security support for all this stuff. So, lockfiles are just completely ignored for all languages, Python, Go, Rust, etc. There are some notable exceptions however: If something does change API/ABI in an incompatible way, and it isn't easy to patch in support, we will ship with multiple versions of the same software and allow to install them in parallel. Usually this will not be 10 versions though, ideally the amount of copies would be fairly low (2-3, possibly).
An example of this is the Boost libraries: https://packages.debian.org/search?searchon=sourcenames&keywords=boost
Currently we have two versions of them in the same release (1.81 and 1.74).

So, tl;dr: While over-eager locking of versions can be addressed in theory, it is so much work to maintain this that distro developers will try to avoid duplicating versions and try to build with the latest version of a dependency, and only package two versions if absolutely necessary.
This really isn't a Debian-specific thing, it definitely applies to every distribution, Ubuntu, Fedora/RHEL and SUSE as well.

add-local actually does have a global effect. However, it is stored in a single configuration file (local-packages.json), so you can't really include its result in a package.

What I meant is that it's a bit annoying to have to run this command first before every package build - if dub could just find things automatically, without needing to be configured in every package first, that would be nice. Definitely not a must-have, but a nice to have :-)
(without it, we could just make dh-dlang do this automatically though, I guess)

OK, I reproduced it! Looks like this is still an issue, with --cache=system it will try to write there. Will file an issue / poke at Dub to see what can be done about this.

Awesome! This is the only real dealbreaker that I remember was left from back then, and it really smells a lot more like a bug than any design decision :-)
Would be awesome to have this fixed! :)
If dub also had a means to install the whole source code into the system location, that would be neat too, but it's also truly trivial to do in a package, as long as we have a standardized path specification that we can follow :-)

OK... but I think we need to figure out some conceptual details, like the versioning thing... like, if Debian installs one version of a Dub package, but I want to build a program that wants a different version of that package, what happens?

In this case I would expect dub to realize "local version is not recent enough" and go to code.dlang.org to download a more recent version and use that instead. The "no internet" rule only applies to Debian packages, when users use dub, it can download all it wants - and also write into HOME, of course :-)
(and if it tries that during a package build, it will fail and we can fix the packaging)

Just want to reply to this:

No downloads of anything are allowed at built-time at all. Internet access is disabled at that stage. So any dependency has to be installed before the build, and as a Debian package, for dub to find.

Sorry, we're talking past each other. Don't mean during the build... but... the package build script, or the person running it, has to obtain the upstream source code somehow. So, for example on Arch, that mechanism is the source array, which is a list of URLs pointing to upstream source tarballs or such, and makepkg will fetch it using the user-configured download agents (which could be defined in the package if there is a need).

For the typical case of the package source being distributed as a tarball on the author's website, the download tool would probably be curl or wget, plus tar to unpack it. So the question is, could dub fetch be used here instead of curl/wget?

Ah, right - Debian has uscan and the debian/watch files for that which will autodetect upstream changes. This works with tarballs anywhere, Git repositories, PyPI, ... It wouldn't call dub fetch, but could grab dub packages from GitHub or the location dub itself pulls them from. See https://wiki.debian.org/debian/watch for reference. These checks are run regularly, and people are alerted if new upstream versions are present and need packaging. The uscan tool is also integrated with other packaging tools and the Git-based packaging workflow, so in many cases a single command can update a distro package.
See for example https://tracker.debian.org/pkg/dub - the "action needed" box says that there's a new version that we should update to ;-)

Fedora has a similar-in-concept system called Anitya while is mapped into their RPM packaging workflows (can be found @ https://release-monitoring.org/).

I see, thanks! So then I think we do need something like dub install. I think this plan should work? Or I guess an alternative would be to have the registry provide download links in a format that can be easily put into Dub's system cache.

Awesome! This is the only real dealbreaker that I remember was left from back then, and it really smells a lot more like a bug than any design decision :-)

Good news on this front, turns out only two small things were missing:

1. --temp-build flag at dependent build time;
2. A tiny fix to remove the last bit of writing to the system cache.

With both, I can successfully run dub fetch as root (yes, we still need to figure out a replacement) and dub build as unprivileged user.

If you have bandwidth to continue collaborating, perhaps there is a better place to continue this discussion?