Bindists for the Haskell toolchain have been produced by upstream (the developers of each respective tool) for a long time and many tools rely on these "official" bindists (e.g. GHCup and stack).
We propose here that bindists are additionally built and maintained in joint effort by the projects that provide installation experiences, removing the dependency on upstream bindists entirely.
Historically, installers like GHCup and stack have used upstream bindists for mainly one reason: it's easy to do so and doesn't require further efforts.
However, using upstream bindists directly is extremely rare in the Linux world of distribution. Most distributions build, package, test and curate binary packages themselves, not only because they have custom formats, but for reasons of control, trust and quality.
From the perspective of a GHCup developer, there are several issues with relying on upstream bindists.
GHC and other tools have in the past dropped support for certain platforms either entirely or requested the community to step up and do the work (e.g. on GHC CI).
E.g. GHC ARMv7 support was dropped silently without any call for help. Similarly, FreeBSD support just ceased to exist when the GHC FreeBSD CI stopped working. Later the community asked for a revival, but nothing signifcant has happened so far.
Similarly, stack used to have issues with Aarch64 bindists:
- commercialhaskell/stack#5709
- commercialhaskell/stack#5854
- commercialhaskell/stack#5540
- commercialhaskell/stack#5610
- commercialhaskell/stack#5619
- commercialhaskell/stack#6141
- commercialhaskell/stack#6142
Recently, cabal-install had issues with i386 binaries and alpine, delaying a GHCup metadata PR:
These issues are frequent and so far the GHCup developers used to single handedly fix all those missing bindists manually and provide them here: https://downloads.haskell.org/~ghcup/unofficial-bindists/
This is unfunded and significant work and GHCup has decided not to pursue this "closing the gap" approach anymore.
Sometimes, bindists are broken, e.g. for GHC there are a couple of instances:
- 9.0.2 shipping without profiling info: https://gitlab.haskell.org/ghc/ghc/-/issues/21841
- DESTDIR variable ignored by
make install
: https://gitlab.haskell.org/ghc/ghc/-/issues/19646
Sometimes, bindists have been built for very old version of linux distros and won't run well on newer linux versions. This is currently a problem since Debian has removed ncurses5:
- haskell/ghcup-hs#902
- https://answers.launchpad.net/ubuntu/+source/ncurses/+question/707838
- https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1025964
For cabal, there has been the infamous hLock issue:
This shows that bindists, for current and historical versions, need continuous maintenance. However, upstream developers so far have very rarely engaged in this type of maintenance work, pushing it down to GHCup. As an example, here are all the manually patched and re-packaged bindists that fix the DESTDIR bug outlined above: https://downloads.haskell.org/ghcup/unofficial-bindists/ghc/curated/
This type of work requires significant time.
There's something pleasing about upstream providing bindists: there's a perceived trust about it in the community, e.g. the haskell.org committee expressed concerns about GHCup changing bindists in the past.
However, rarely do upstream developers have signifcant experience in redistribution, nor do they have the time to focus on all the issues that come with it. Bindists are mostly provided "as-is" and support beyond what the release CI outputs is left to midstream (GHCup, stack, ...).
Conceptually, it is a good idea to separate concerns: upstream provides the sources and has tested it. Distributions build the binaries, validate that the program can be successfully built from source and make sure that the final artifacts pass the test suite. This is good, because we want to know whether end users can build e.g. a functioning GHC from source. If only the release CI outputs a GHC that passes the test suite, then something is fundamentally broken.
Distributors are often closer to the end-users and can provide additional support and efforts for the installation experience.
As a special case, I want to point out that GHC nightlies have been frequently broken beyond repair:
- https://gitlab.haskell.org/ghc/ghcup-metadata/-/issues/2
- https://gitlab.haskell.org/ghc/ghc/-/issues/24000
The breakage was left unattended and some bindists completely vanished from gitlab CI artifacts, because of misconfiguration. Here's a graph of nightlies availability: https://grafana.gitlab.haskell.org/d/ab109e66-a8a1-4ae9-b976-40e2dfe281ab/availabilitie-of-ghc-nightlies-via-ghcup?orgId=2
So far, GHCup developers have tried to close the gap, doing signifcant work on tooling CIs and building bindists manually. This is problematic, because it's volunteer effort and unsustainable. GHCup developers have stopped doing this type of work.
We propose here to create a joint project of "installation experience" developers to get funding and maintain all tooling bindists centrally and autonomically.
This will allow:
- greater quality assurance
- more unofficial platform support (e.g. FreeBSD, i386, ...)
- more bindists for uncommon linux distros
- continuously updated and maintained bindists (including historical GHC versions)
- actually working nightlies
One major concern that was brought up about building bindists midstream was if/how the testing will be done. Building bindists midstream is a great opportunity to put more pressure on e.g. GHC HQ to fix issues with the test bindist:
- https://gitlab.haskell.org/ghc/ghc/-/issues/22726
- https://gitlab.haskell.org/ghc/ghc/-/issues/22723
- https://gitlab.haskell.org/ghc/ghc/-/issues/22727
The idea is that bindists should be primarily tested on the users system, because that is where they're going to run. It is great to know that e.g. the test suite passes on GHC CI, but that may have little value in different environments.
Additionally, issues with tests can flow back to upstream developers and we may develop workflows and processes to streamline this type of feedback. Early release candidates can assist with this workflow.
GHCup will require to implement revisions to make updated bindists feasible: haskell/ghcup-hs#361
- 3 months: proof of concept of a central GitHub CI building the entire toolchain
- 6 months: support for uncommon platforms, bindist testing workflow, automated integration with GHCup, etc.
- 9 months: nightlies working for GHC and cabal
This might require a full time developer for at least half a year and potentially aid from the Haskell Foundation devops, as well as help from volunteers.
- GHC developers
- cabal developers
- stack developers
- HLS developers
- VSCode Haskell developers
- GHCup developers
- Haskell toolchain end users
- reliable, continuously maintained bindists, readily available