dlang-packaging-issues.md

D Distribution Issue List

This is a list of issues found while making the D ecosystem available on Linux distributions such as Debian and Fedora. It is a smaller, but priorized list of the general list of issues I find while working with D that can be viewed at dlang-quirks.md.

This list is ordered by priority, more pressing issues being at the top.

LDC, DMD, DUB, DLang

1. D has no stable ABI. This kind of enforces recompiling all D shared libraries and binaries in a Linux distribution with every new compiler version. This is possible, but a major pain which requires quite some amount of manpower (Debian does this for OCaml and Haskell) - getting people to use D if it is that much extra work is harder. It also means that you can not use the libraries which were compiled with LDC in an application that is compiled with GDC and vice versa. This is a real PITA for distros and users who might have binaries compiled with the "wrong" compiler. It reduces the usefulness of development D packages in distributions a lot, if you need to recompile everything again anyway. Static linking is also frowned upon by distributions, because it triggers rebuild cascades on every bugfix that we introduce, making devlivering security updates much harder and error-prone (embedded code copies are very difficult to handle). A stable D ABI, or at least a guarantee that the ABI will not change unless there is a good reason (and the ABI change will be announced in the release notes) would help a lot. Having a D ABI that is shared between DMD, LDC and GDC would be even better.

2. Dub doesn't use packages installed to system locations (dlang/dub#838) - this renders dub pretty much unusable for using it as part of the Linux distribution package tooling. Dub also writes into $HOME as part of the build process which is considered bad practice and is disallowed on build daemons at Debian (you need to add a hack for tools which expect $HOME to be available)

3. Dub: Can not version dependencies on other libraries (dlang/dub#906) - this makes it hard to determine compatibility if we package software written in D.

4. dub test overrides the binary created by dub build (prevents automatically running tests as part of the build process) (dlang/dub#840)

5. No working Make/Ninja compatible depfile generation in LDC, DMD or GDC - this is especially important when not using dub (see the dub issues) and some other build system (Automake/CMake/Meson/..) is used instead. See ldc-developers/ldc#1802 for the LDC bug report.

6. There is no dub install command (dlang/dub#839)

7. LDC doesn't support a lot of architectures / architecture support breaks from time to time. It would be quite nice to have D tools available on as many architectures as possible. (see ldc-developers/ldc#1636 for a bug report, build logs are here: https://buildd.debian.org/status/package.php?p=ldc)

GDC Packaging Issues

The GDC issues have been separated out to not clutter the priorized overview.

1. GDC is not part of official GCC. This makes it difficult (or actually sometimes impossible) to ship it with some Linux distributions, e.g. RHEL and Fedora. On Debian, we inject GDC into the GCC build, which is also a bit weird. Having GDC in GCC proper would also give D much higher visibility.

2. GDC only supports an ancient version of the D standard library, which has many nice classes and also bugfixes missing. Because of that, it is almost impossible to compile modern D applications with GDC, and when developing a new application, support for GDC needs to be added explicitly and handled with a lot of care. This is an issue LDC doesn't have, and which is fragmenting the D ecosystem.

3. GDC does not support creating shared libraries at time, which is a big deal for distros which need it to reduce duplicate code and make security fixes easier.

Changes

• DMD is completely free software now, there is no obstacle to ship it in distributions anymore.

That does sound aligned to what dub --cache=system does, I think... so then the next question is, what options are there for downloading the package? Does it have to be a Debian-provided tool or can it be a language-provided tool (in our case, dub fetch)?

No downloads of anything are allowed at built-time at all. Internet access is disabled at that stage. So any dependency has to be installed before the build, and as a Debian package, for dub to find.
So the only thing dub fetch could do is copy source code from a system location somewhere else (but at that point, it might just as well use the code directly, without making a copy).

What is done for languages like Rust/Cargo, is it the same approach? I think a packaging scheme which does not allow more than one version of a package is incompatible with the idea of lock files, which are used by pretty much all language package managers. D (or at least Dub) is much closer to Rust/Node/etc. in that it has strong dependency declaration via lock files (though in case of Dub, they are optional, but generally strongly encouraged). So, Debian would need to either decide to disregard project lock files (and deal with the fallout of possible incompatibilities) or package each version individually.

The latter is exactly what happens: When packaging so much software, keeping each small version of each package around is just insane, especially if you also need to provide security support for all this stuff. So, lockfiles are just completely ignored for all languages, Python, Go, Rust, etc. There are some notable exceptions however: If something does change API/ABI in an incompatible way, and it isn't easy to patch in support, we will ship with multiple versions of the same software and allow to install them in parallel. Usually this will not be 10 versions though, ideally the amount of copies would be fairly low (2-3, possibly).
An example of this is the Boost libraries: https://packages.debian.org/search?searchon=sourcenames&keywords=boost
Currently we have two versions of them in the same release (1.81 and 1.74).

So, tl;dr: While over-eager locking of versions can be addressed in theory, it is so much work to maintain this that distro developers will try to avoid duplicating versions and try to build with the latest version of a dependency, and only package two versions if absolutely necessary.
This really isn't a Debian-specific thing, it definitely applies to every distribution, Ubuntu, Fedora/RHEL and SUSE as well.

add-local actually does have a global effect. However, it is stored in a single configuration file (local-packages.json), so you can't really include its result in a package.

What I meant is that it's a bit annoying to have to run this command first before every package build - if dub could just find things automatically, without needing to be configured in every package first, that would be nice. Definitely not a must-have, but a nice to have :-)
(without it, we could just make dh-dlang do this automatically though, I guess)

OK, I reproduced it! Looks like this is still an issue, with --cache=system it will try to write there. Will file an issue / poke at Dub to see what can be done about this.

Awesome! This is the only real dealbreaker that I remember was left from back then, and it really smells a lot more like a bug than any design decision :-)
Would be awesome to have this fixed! :)
If dub also had a means to install the whole source code into the system location, that would be neat too, but it's also truly trivial to do in a package, as long as we have a standardized path specification that we can follow :-)

OK... but I think we need to figure out some conceptual details, like the versioning thing... like, if Debian installs one version of a Dub package, but I want to build a program that wants a different version of that package, what happens?

In this case I would expect dub to realize "local version is not recent enough" and go to code.dlang.org to download a more recent version and use that instead. The "no internet" rule only applies to Debian packages, when users use dub, it can download all it wants - and also write into HOME, of course :-)
(and if it tries that during a package build, it will fail and we can fix the packaging)

Just want to reply to this:

No downloads of anything are allowed at built-time at all. Internet access is disabled at that stage. So any dependency has to be installed before the build, and as a Debian package, for dub to find.

Sorry, we're talking past each other. Don't mean during the build... but... the package build script, or the person running it, has to obtain the upstream source code somehow. So, for example on Arch, that mechanism is the source array, which is a list of URLs pointing to upstream source tarballs or such, and makepkg will fetch it using the user-configured download agents (which could be defined in the package if there is a need).

For the typical case of the package source being distributed as a tarball on the author's website, the download tool would probably be curl or wget, plus tar to unpack it. So the question is, could dub fetch be used here instead of curl/wget?

Ah, right - Debian has uscan and the debian/watch files for that which will autodetect upstream changes. This works with tarballs anywhere, Git repositories, PyPI, ... It wouldn't call dub fetch, but could grab dub packages from GitHub or the location dub itself pulls them from. See https://wiki.debian.org/debian/watch for reference. These checks are run regularly, and people are alerted if new upstream versions are present and need packaging. The uscan tool is also integrated with other packaging tools and the Git-based packaging workflow, so in many cases a single command can update a distro package.
See for example https://tracker.debian.org/pkg/dub - the "action needed" box says that there's a new version that we should update to ;-)

Fedora has a similar-in-concept system called Anitya while is mapped into their RPM packaging workflows (can be found @ https://release-monitoring.org/).

I see, thanks! So then I think we do need something like dub install. I think this plan should work? Or I guess an alternative would be to have the registry provide download links in a format that can be easily put into Dub's system cache.

Awesome! This is the only real dealbreaker that I remember was left from back then, and it really smells a lot more like a bug than any design decision :-)

Good news on this front, turns out only two small things were missing:

1. --temp-build flag at dependent build time;
2. A tiny fix to remove the last bit of writing to the system cache.

With both, I can successfully run dub fetch as root (yes, we still need to figure out a replacement) and dub build as unprivileged user.

If you have bandwidth to continue collaborating, perhaps there is a better place to continue this discussion?