Proposal for making third-party libs easier to use

include_magic.md

Magic #includes (and eventually imports)

This is an alternative to a proposal that I'm sure was offered with good intentions, but that I and several others shuddered at the sight of:

// Oh, the horror!
#include <https://github.com/someone/something/path/to/some/header.h>

Instead, I propose letting the first component of a /-separated #include <.../...> (or similar for module imports) be a "tag" that can be specified in a project-wide tags.yaml file:

#include <boost/some/boost/header.h> // tag = boost
#include <catch2/some/catch/header.h> // tag = catch2

// If the would-be tag isn't declared, it would be found as it
// is today with the normal system paths.
#include <linux/futex.h>

// no / so don't use tags
#include <map>

// don't use tags with "" includes
#include "no/tags/here.h"

(Assume all names and even the format are strawmen for now. We can bikeshed those details later, once we agree on the overall concept of prefix-tag-replacement)

# tags.yml
catch2:
  repo: https://github.com/catchorg/Catch2
  subdir: include/
  hash: deadbeef
    
boost:
  repo: http://github.com/some/url
  hash: badf00d

benchmark:
  path: third_party/benchmark/
  # If path is present, remaining fields are informational only.
  # They could be used by tools to simplify upgrading of vendored deps.
  repo: http://github.com/google/benchmark
  hash: d00dface

The top-level project can provide overrides to force child projects to use specific dependencies, or local checkouts. In general, only exactly matching git hashes are ever used. If two children in the same project specify different hashes for the same repo, the top-level project must use an override to to force a common version. Note that there is only a two-level hierarchy, the top-level project, and all dependencies.

Overrides can also be used by distros that demand that builds be done using the system version of libraries rather than vendored versions. Tools should accept an additional override file provided by the invoking user to let them overrule the project's defaults.

# These overrride everything using this tag.
GLOBAL:
  boost:
    # use the system version
    path: /usr/include/
  nifty_lib:
    # use my working copy since I'm making changes to both
    path: ~/develop/nifty_lib/
  benchmark:
    # Use a fork rather than the mainline branch
    repo: http://github.com/hooli/benchmark
    hash: feedf00d

# You can also override for a single project
silly_project:
  boost_1_56:
    use_tag: boost # replacement by reference
  catch2:
    # Just force it to use a different hash
    repo: https://github.com/catchorg/Catch2
    subdir: include/
    hash: deadc0de

The C++ standard already allows a lot of implementation flexibility in how it interprets #include paths, so no change to the core standard is needed to allow for this.

This is almost achievable with just build-system work today using fake include paths that just hold symlinks to the real projects. The only part that would require compiler changes is to make the include paths configurable based on where the including file is located, rather than using the same paths for the whole TU. But since that should only be needed in the (hopefully) rare case of two projects using the same tag for unrelated dependencies, this should be usable for most (sane) projects today.

Pros:

1. It doesn't put github links in every file!
2. See 1!
3. Provides a single place to specify (and change!) which version should be used for each dependency
4. Makes it easy to do local development on a dependency
5. Git hashes don't lie (at least once they finish moving off of SHA1)
6. It uses only exact versioning to always get reproducible builds with the tested versions
7. Easily extended to non-git repos and source archives (with hashes or signatures)
8. Works even in projects that vendor in dependencies, but want to support building with other versions
9. Plays nice with distros that require use of system libs
10. Doesn't require any central repo (even github)
11. Most people consider YAML human readable and editable
12. Doesn't require boiling the ocean: any project can start using this with existing code, even before dependencies do

Cons:

1. It requires an extra file in each project
2. It requires top-level projects to pick compatible versions of common libs

Open Questions:

(These can be addressed if we like the overall direction)

1. How to handle projects like boost that are generally treated as multiple packages in subdirectories.
   • Could require something like <boost.math/path/to/header.h> so the first / separates tag from path
   • Could abandon that simplicity and just let tags contain / and match each #include prefix against all tags
2. Windows \ vs Unix / vs modules .
   • Simplest is to treat all the same inside the paths, but that depends on the outcome of 1
3. Case sensitivity
   • I think the best option is to do case folding before matching
   • Disallows using tags that differ only in case, but is a bad idea that isn't portable anyway
   • May require compiler support
4. Do we even need to support the case where multiple projects use the same tag to refer to different deps
   • This is probably the hardest part of this proposal
   • On the other hand this almost seems inevitable in the real world as long as some packages use common names like benchmark
5. Should the format support a field like build commands to describe how to build the dependency, or should that be a separate tool/format?
   • Certainly not a version 1 feature, I think