envs.md

Proposal: Introduction of Feature Sets

Objective

The aim is to introduce a feature set mechanism in the pixi package manager. This mechanism will enable clear, conflict-free management of dependencies tailored to specific environments, while also maintaining the integrity of fixed lockfiles.

Motivating Example: Test Dependencies and Multiple Python Versions

Consider a scenario where a project needs to be tested across multiple Python versions, each requiring a different set of dependencies. In this case, defining separate feature sets for each Python version (like py39, py310, etc.) allows for easy switching between environments without conflicts. Similarly, for development purposes, a test feature set can include dependencies necessary for testing and linting, which are not required in the production environment.

Design Considerations

1. Non-Combinatorial: To ensure the dependency resolution process remains manageable, the solution should avoid a combinatorial explosion of dependency sets.
2. Single Feature Activation: The design should allow only one feature set to be active at any given time, simplifying the resolution process and preventing conflicts.
3. Fixed Lockfiles: It's crucial to preserve fixed lockfiles for consistency and predictability. Solutions must ensure reliability not just for authors but also for end-users, particularly at the time of lockfile creation.

Proposed Solution

Feature Set Definitions

Introduce feature sets in the pixi.toml configuration file, with each set comprising dependencies specific to a given environment or use case. For instance, a test feature set may include dependencies like pytest and pre-commit, essential for development but not for production.

pixi.toml Example:

[features]
test = ["pytest", "pre-commit"]
py39 = ["py39"]
py310 = ["py310"]

[dependencies]
requests = "*"
pytest = { version = ">= 1.2", optional = true }
pre-commit = { version = ">= 2", optional = true }
py39 = { package = "python", version = "3.9", optional = true}
py310 = { package = "python", version = "3.10", optional = true}

Lockfile Structure

Within the pixi.lock file, a package may now include an additional feature field, specifying the feature set to which it belongs. This structure ensures clarity and prevents unnecessary duplication of dependencies across different environments.

Feature Set Activation

Users can manually activate the desired feature set via command line or configuration. This approach guarantees a conflict-free environment by allowing only one feature set to be active at a time.

Command Configuration

Commands defined in pixi.toml can specify the supported feature sets. For example, a testing command can be linked to the test feature set, ensuring it runs with the correct dependencies.

Command Configuration Examples:

[tasks.test]
cmd = "pytest"
feature_set = ["test"]

[tasks.test]
cmd = "pytest"
feature_set = ["py39", "py310"]

Benefits

• Simplicity: Clear and straightforward dependency management is achieved by making each feature set mutually exclusive.
• Consistency: The solution upholds the principle of fixed lockfiles, ensuring stable and predictable dependency management across different project stages.

Drawbacks

In the proposed feature set mechanism, users can activate only one feature set at a time. This design decision simplifies the dependency resolution process and prevents conflicts. However, it can be limiting in certain scenarios:

Scenario with Orthogonal Feature Sets

• Orthogonal feature sets are sets that could theoretically be combined because they don't interfere with each other. For example, one set might pertain to linting tools, while another might specify a Python version.
• The limitation arises when a user wants to activate multiple such orthogonal sets simultaneously. For instance, they might want to run linting tools (lint feature set) under a specific Python version (py39 feature set).
• In the current design, they would have to define a new feature set that combines both sets of dependencies. This becomes cumbersome if there are many such orthogonal sets, as it requires defining every possible combination.

Another idea that came to my mind which could be useful:

A use-case that occurs quite often for me is that i want my dev dependencies to be a strict superset of my prod dependencies. This way, one can actually make sure that the tests that one executed actually matter in production and don't produce different results just because the prod environment was solved differently.

I'm not sure what the best way to integrate this into pixi.toml is. Does anybody have suggestions?
Maybe something like this?

[environments]
py39 = ["py39"]
py39test = ["test", "py39"]
py310 = ["py310"]
py310test = ["test", "py310"]
py311 = ["py311"]
py311test = ["test", "py311"]
prod = ["py311"]
prod-test = [{environment = "prod"}, "test"]
# or alternatively
# prod-test = ["evironment:prod", "test"]
# or
# prod = [{constraints = "env:prod-test"}, "py311"]
# prod-test = ["py311", "test"]

[feature.py39.dependencies]
python = "3.9.*"

[feature.py310.dependencies]
python = "3.10.*"

[feature.py311.dependencies]
python = "3.11.*"

[feature.test.dependencies]
pre-commit = "*"
pytest = "*"

[feature.test.tasks]
test = "pytest --color"

is this the official discussion thread? i've created "hydraconda" so i have much to contribute.

There was also some discussions on prefix-dev/pixi#584

now, there is already quite a lot implemented in the latest pixi build on main (not released yet, check the build artifacts)

Yeah end of this week we should have the feature released in an MVP state. I think it's better to start a PR on the design docs if you have additional ideas or a discussion on pixi's discussion board.