TL;DR: We piggy-back on the "code-generators" stanza in the cabal format to declare a build driver. The driver looks-up its own custom stanza, takes the options it needs and starts the build before the component (lib, exec, etc). The resulting artifacts are then put in the build directory and can be depended on through "extra-libraries" and "ext…

0-custom_builder.cabal

-- Inspired by Elixir's Rustler build tool: https://hexdocs.pm/rustler/0.18.0/basics.html
library
    hs-source-dirs:     src
    default-language:   Haskell2010
    ghc-options:        -Wall
    build-depends:
        base >=4.10.0.0 && <5,
        bytestring >=0.11.0.0
   
    extra-bundled-libraries:    rustLibrary -- the rust library that we are building
    code-generators:
      cargo-driver -- cargo-driver then parses the Cabal file, extracts its stanza. It's entirely external.
                   -- The data could be stored in a .TOML file for all we care.
    x-cargo-driver:
      path: native/rust-library
      cargo: -- cargo options
        system -- or
        rustup: "rust_version" -- or
        path: "path/to/cargo/binary"
      default-features: boolean -- default cargo "features", which are flags
      features: [feature1, feature2]
      mode: release | debug

1-FAQ.md

FAQ

These are real questions I've received, if you have more that I haven't answered here, please ask them to me.

Q: Cabal is going to have to learn how to configure rust/zig/D/C++/meson/bazel projects?

A: No. The logic is offloaded to build drivers (like cargo-driver) that get the necessary information from a custom stanza

---

Q: Cabal already knows how to do it, it's called Setup.hs

A: Setup.hs' unrestricted code execution proves to be a liability. The ability here is to provide a clear and official path to allow the transition from this Setup.hs usecase. One build driver can be analysed for n projects, but n Setup.hs files must be analysed for n projects.

---

Q: This cargo-driver of yours, who's going to be in charge?

A: A healthy mix of stakeholders and community members, the idea being that some central coordination with the cabal team is needed and we can't have this left to the good will of the community altogether, because we want to present a coherent and reliable "Rust integration" story. The Haskell Foundation can have a stake in it, for instance. But anybody can make a driver, it's not a walled-garden. cargo-rust will be maintained in a more “official” capacity due to its ecosystemic importance and significance.

---

Q: This is too heavy/rigid for me, why can't we just let the user figure it out?

A: The goal is to offer a clear and straightforward path to native code integration without having to retain the knowlege of those systems in Cabal. At the same time, more flexible tools are an open door to unregulated code execution.

---

Q: This is the wrong approach, you should put that knowledge in something higher-level like Bazel

A: This doesn't prevent you from using Bazel but forcing people to learn Bazel in order to do what other languages have asking too much for too few benefits.

---

Q: You're bloating Cabal, external dependencies shouldn't be necessary to use external dependencies

A: We think that the weight of having a build driver as a build dependency to handle other build dependencies is an acceptable compromise.

---

Q: Realistically, what system only knows how to do cabal build? Can't you run a command before cabal build that takes care of everything?

A: Not every system is able to run arbitrary command when compilling a Haskell package. More specifically each time a package X depends on a package Y, cabal will build the dependency with a standard cabal build. Build platforms that support cabal shouldn't have to adapt to something too flexible when it comes to building cabal packages.

As discussed,

• it's relatively easy to support encoding package case (custom Setup.hs) via something similar to:

build-drivers:
  mapping-generator

• and point it out to Preprocessor module.
• then Cabal would perform actual transition of .mapping files to produce Autogen-Modules.

Worth noting here is that the problem (broadly) that this seeks to address (namely, cross-language integration) has three aspects:

1. Dependencies
2. Calling other build systems
3. Linking

In particular, I see this as addressing problem 2 only. Problem 1 is much broader (and is essentially ecosystemic), while problem 3 is something that GHC (and only GHC) can solve.

Alright, so here's my take on this. Let's take a step back:

If we deal with external dependencies (I've usually referred to them a system libraries, though that's not really what they are), we have tools like pkg-config and similar for those. I understand this can be awkward for more tightly coupled development.

So we have a Haskell project H, and a foreign project F. Both come with package managers P_h, P_f. P_h is mostly cabal-install or stack. Though we could argue that stack delegates a lot to lib:Cabal as does cabal-install, so they are fairly similar.

I think this should be simplified a lot. I don't think turning cabal into a meta-build system is a good idea. Do we expect cabal to have proper tracking logic when to rebuild dependencies or not?

I think we should condense this to a pre-build (or should it even be pre-conifgure?) script that can be called. Calling this code-generator or what not, is ok, we have alex and happy and other tools as well.

But I'm having quite an issue with this the x-cargo-driver section, so let's dissect this line by line:

      path: native/rust-library

This is generic enough that I can see how this might be a good idea to have. Though if we delegate this to a pre-build script this would be even better. If we are concerned that shell scripts suck on e.g. Windows, we could also start allowing this to be a haskell script, we run with runhaskell, and allow to pass some dependencies similar to setup-depends.

      cargo: -- option

having this, and the other flags in the cabal file seems wrong.

        system -- or

Same as above, I don't think this should be in the cabal file at all. Not even under an x- flag.

        rustup: "rust_version" -- or

this hardcodes some assumption about rustup. It shouldn't be in here.

        path: "path/to/rustup/binary"

this is a very bad idea; hardcoding paths into the cabal file is probably going to be a red line I'd draw.

      default-features: boolean -- default cargo "features", which are flags

again, why is this in the cabal file?

      features: [feature1, feature2]

same

      mode: release | debug

same.

Now on to the slightly more interesting question: if you have a package that has a native library dependency. And this ends up being a library, you need to tell cabal somehow to package the library alongside your package. We do have extra-bundled-libraries for this, and it work. We use it for the rts, and you can use it with great success to marry ghcjs + emscripten. You can even abuse it to stick metainformation alongside you cabal package.

extra-libraries, tells us which -l flags to pass. If you consume a library it doesn't tell you where to find that library really; and if that foreign library is part of your build, you'll need to somehow ship it along side. See extra-bundled-lirbaries.

What I think this should be is some pre-build or pre-configure hook like thing that can extend the local package description if needed. Permit to have a pre-build/pre-configure haskell script to be run similar to Setup.hs, and allow it to produce a .buildinfo like file that is use to augment the cabal file. That way you can

• call out to an arbitrary build system (your cargo-driver would just be a haskell library in your pre-build-depends and the pre-build some haskell file. You don't have any of this extra logic in cabal. You could specify all your driver specific stuff in the haskell file. We could have --pre-build-options=... that are fed into your pre-build driver. The pre-build driver could also look at env vars (RUSTUP?), ...

• the pre-build driver could (based on that build system) then produce the extra-bundled-lirbaries and extra-lirbaries in a .buildinfo like file, and we'd still mostly retain a clean cabal file that knows very little about a foreign build system (as it should be IMO, it should just treat it as black box).

I agree with @kozross's points, and I with we'd really delegate linking outright to ghc, but even cabal tries to do linking, and arguably has in a few cases, as only cabal knows final resting locations of libraries, even though I still think we should completley delegate linking to ghc, and ensure that this information flow is passed to ghc accordingly. This has lead to bugs in lirbaries built and linked with cabal, that ghc would link properly.

One more note: why is cabal a {asm,c,cmm} package manager then? That's simply because there wasn't any when cabal was conceived, and even today there isn't any. Another reason is that GHC's FFI is effectively a C FFI, so you'll need (for now) C ffi bindings to effectively interact with foreign code.

Here is a solution how to build and link C files with emscripten into a ghcjs project. There are a few special complications with this:

• emscripten needs to run its own linker on intermediate libraries. As such we have to figure out all emscripten compiled libraries, and for executables run a separate linking phase prior to linking it with ghcjs. We also need to do this in a ad-hoc manner for each intermediate library we build, as we might need to load some pre-linked library for Template Haskell. Most of this logic does not need to be in Cabal at all. And it shouldn't it is orthogonal to cabal. If you have an option to call some pre-build script, and that can inspect the cabal project to some degree and feed back some build info augmentation (.buildinfo is just a poor text based solution for this), this would be enough. And I imagine it would be enough, minimal and sufficiently flexible to cover pretty much any external build system.

@angerman Hi! I've made a couple of edits (typos & all) to the gist before answering to your questions. :)

Do we expect cabal to have proper tracking logic when to rebuild dependencies or not?

When it comes to this type of dependencies, no. As Koz said, this is mostly about concerning ourself with calling other build systems without using the Custom setups.

having this, and the other flags in the cabal file seems wrong.

It's not an obligation, I'm just taking inspiration from how other languages do it

this is a very bad idea; hardcoding paths into the cabal file is probably going to be a red line I'd draw.

Ok sure, I'm not wedded to this idea.

again, why is this in the cabal file?

I figured out this would be a more straightforward path in terms of UX for the developer, than writing an extra TOML file or w/e.

We do have extra-bundled-libraries for this, and it work.

Agreed, I've fixed this in the cabal file.

so you'll need (for now) C ffi bindings to effectively interact with foreign code.

This is a restriction I've talked about with Koz, C++ linking is one hell of a beast and I'm definitely sure that I don't want to get to it right now.

I think we should condense this to a pre-build (or should it even be pre-conifgure?) script that can be called.

The idea is to get rid of custom setups, so generic "pre-build" or "pre-configure" steps are not quite the kind of hooks I'm aiming to have here. ;)