Draft of the upcoming RubyGems external_depenencies RFC

issue.md

[RFC] Allow specifying and installing external dependencies

The Problem

Some gems require external dependencies, such as gems containing C extensions that compile against external C libraries (ex: pg) or gems that wrap around external command-line utilities (ex: graphviz). Presently there is no mechanism to automatically install these external dependencies. Instead, it is the user's responsibility to figure out the correct package name and install the external dependency via their system's package manager (ex: apt or brew) and attempt installing the gem again. This often results in users becoming frustrated when a C extension does not successfully build and then having to search google and StackOverflow for which package needs to be installed.

Very rarely do gem authors list external dependencies via the gemspec requirements attribute, and if so, it usually lists the canonical project name of the external dependency (ex: sqlite3) and not the package name (ex: libsqlite3-dev).

The Workaround

To workaround the problem of external dependencies some popular gems (ex: nokogiri) have begun vendoring their external dependencies directly into the gem and building both the C library and any C extensions during installation. There are two downsides to this approach:

1. Increased compilation time.
2. Security Advisories: each time a new security advisory is published for the vendored library, the gem maintainer has to update the vendored copy of the C library, publish an updated version of the gem, and then publish their own security advisory warning users that the previous versions of their gem contains a vulnerable copy of the vendored library.

The Proposal

It should be possible to list the external dependencies required by a gem and have rubygems automatically install them from the system's package manager during the installation process.

This external_dependencies metadata could be embedded in the gemspec's metadata attribute. In order to support naming differences between different package managers, the package names for the external dependencies would need to be listed for each popular package manager.

Example:

  gemspec.metadata['external_dependencies'] = {
    'apt'  => %w[libsqlite3-dev],
    'dnf'  => %w[sqlite-devel],
    'brew' => %w[sqlite],
    # ...
  }

Sub-Proposal 1

If only one external dependency needs to be specified, then the values of the external_dependencies Hash could also be single package name Strings which would later be automatically coerced into an Array:

Example:

  gemspec.metadata['external_dependencies'] = {
    'apt'  => 'libsqlite3-dev',
    'dnf'  => 'sqlite-devel',
    'brew' => 'sqlite',
    # ...
  }

However, this might be a bit confusing?

Sub-Proposal 2

If all of the package names are the same for each package manager, then external_dependencies could be specified as an Array of Strings:

Example:

  gemspec.metadata['external_dependencies'] = %w[nmap]

Caveats

Detecting the System's Package Manager

Multiple package managers may be installed on the same system. In order to determine the primary system package manager, the system's package manager can be selected based on the OS/distro/flavor.

macOS is a unique edge-case since it does not have a default package manager. So there should be a prioritized list of macOS package managers to check for in order of popularity:

1. brew
2. ports

Security Concerns

If gems can specify packages names that will be installed via an apt-get install or brew install command, special concern should be made to prevent arbitrary command injection. system() with multiple arguments (ex: system('apt-get','install',...) or Shellwords.shellescape must be used to prevent command injection.

In order to prevent option injection via the gemspec's package names, an argument of '--' or '-' can be specified before the package names to prematurely terminate option parsing and prevent the package names, so that they are not accidentally parsed as options.

Configurability

Some users may wish to customize which package manager is used on their system, if they have multiple package managers installed alongside each other. It may be necessary to add a configuration option or environment variable to control which package manager is used by default. Although, this feature request seems to be very rare.

Possible Deprecations

If we allow annotating the external dependencies and automatically installing them along with the gem, the gemspec's requirements attribute might no longer be necessary and could be deprecated in the future?

Previous Work

• ruby-install
  • https://github.com/postmodern/ruby-install/blob/0.9.0/share/ruby-install/package_manager.sh
  • https://github.com/postmodern/ruby-install/blob/0.9.0/share/ruby-install/system.sh#L22-L62
• command_kit
• ruby-gnome/native-package-installer

Note for posterity: psych 5 no longer bundles libyaml, and the absence of a libyaml-dev distro package in many CI images is causing builds to fail. See ruby/setup-ruby#409 for background. A proposal like this, if it is adopted and used, could prevent breakage in similar scenarios in the future.

I DMed with @postmodern and I'm going to try to turn this into a real RFC in the next few weeks.