_Distfile.lock.json

"Automatically Generated, Not for Editing"
"Acts as a Lock file"
"Uses a List, but is actually a set"
"All packages without a group are in the default group"
{
    "_meta": {
        "sources": [
            {"url": "https://simple.crate.io/"},
            {"url": "https://pypi.python.org/simple/", "verify_ssl": false},
        ]
     },
    "default": [
        {"name": "requests", "version": "0.11.2", "hash": "...."},
        {"name": "Django", "version": "1.4", "hash": "..."},
        {"name": "pinax", "git": "git://....", "branch": "1.4"},
        {"name": "crate", "path": "~/blech", "editable": true}
    ],
    "development": [
        {"name": "test", "version": "0.1", "hash": "..."},
        {"name": "test2", "version": "2.0", "hash": "..."},
        {"name": "another thing", "version": "3.5", "hash": "..."}
    ],
    "testing": [
        {"name": "test", "version": "0.1", "hash": "..."}
    ]
}

Distfile.py

# This is designed to be edited by hand by python developers
# --index-url and friends look like command options, non inutive. No extra metadata available
source("https://simple.crate.io/")
source("https://pypi.python.org/simple/", verify_ssl=False)


# Design:
#  - Use real datastructures, make things clearer
#  - Seperate package name from version, using real strings.
#      Django==1.4 is a valid PyPI package, uninstallable from current files
#  - using kwargs creates a great way to provide optional options on a line by line basis
#     that python programmers are already familar with
#  - People should only have to worry about the things they depend on, the installer
#      should do the right thing
#  - People have different dependency based on environment
#  - Allow advanced usage for "wierd use cases" or "patterns not anticipated"

# Concerns:
#  - Using Python file might cause the same problems as setup.py
#    - This File not designed to be directly executed
#    - setup.py is typically sent to other people, requirements are typically for internal use
#    - Final result is still deterministic
#  - Somewhat more verbose
#    - Uses a syntax familar with all python programmers


dist("requests")  # Install the latest version of requests, and all dependency's
dist("Django", "==1.4")  # Install a version of Django matching the "==1.4" version spec and all dependencies
dist("pinax", git="git://github.com/pinax/pinax.git", branch="1.4") # Install pinax, using git and the 1.4 branch
dist("crate", path="~/blech", editable=True) # Install crate from the supplied path, and install it as an editable
dist("test", group="development")  # install test, but only if the development group is passed
dist("test2", group=["development", "testing"])  # install test2, but only if the development or testing group is passed

with group("development"):  # start a context that is equivilant to passing everything a certain group
    dist("another thing")   # install ``another thing`` if the development group is passed

with source("https://inhouse.example.com/"):
    # Things in here MUST be installed from the above source, the idea being that if you have forked, e.g. django-model-utils
    # you can uplaod to an inhouse server and force django-model-utils to install from this source, even in the case
    # there is another version higher then yours, or even the same version.
    # Additionally if packages installed inside of a with source(): have dependencies their deps are searched for
    # on source, and installed from there if possible. However if they are not available dependencies will fall back
    # to the global source()es.
    dist("django-model-utils")  # Not shown in the json lockfile.

# Details
#  - This file does NOT cause anything to directly get installed, the program uses it to build up an internal list of requirements
#  - All dist()'s are considered for the requirements, even if they are not going to get installed (e.g. they are in a not currently active group)
#    - This will allow things to work smoothly where the requirement in a group might modify the final installed version (e.g. a group might pin to Django<1.4)
#  - This file will be "compiled" down to json.
#    - When the compiled json exists, there is no need to do any sort of version resolving. We know exactly what it is we want to install. (e.g. --no-deps etc)
#    - We still need to find out where to download the packages from? (Unless we encode that in the json. Might be risky)
#  - If there's a corner case not thought of, file is still Python and allows people to easily extend
#  - You don't need to pin to exact versions, you just need to pin to what you want to support. e.g. if a pacakge follows semver, you could do package>1.2.1,<1.3 (you first started using the 1.2.1 version, and you expect 1.2.X to always be good for you.
#    - Exact versions are pinned automatically in the json file

pypa/pip#1795

Is the idea that this is actually Python, or just Python-like

@Julian I guess it's at least Python-parseable.

Although it looks a bit too verbose to me, eh?

I was thinking about something more like package.json style actually:

dependencies:
  six: ^1.10.0  # semver extended syntax, finally in the python world
  graphql-core: >=1.0
  graphql-relay: >=0.4.4
  promise: *
  # These two packages are installed from git:
  somepackage: iamale/somepackage  # shorthand for GitHub
  anotherpkg: git+https://bitbucket.org/iamale/anotherpkg

dev_dependencies:
  another-thing: *

test_dependencies:
  pytest: >=2.7.2

extras:
  django:
    graphene-django: *
  sqlalchemy:
    graphene-sqlalchemy: *

I'm toying with the possibility of making a Pip wrapper that reads such syntax, but I guess it's not what Python community needs right now (we already have setup.py, requirements.txt and now also Pipfile and pyproject.toml — already too many tools for such a simple task).

with source( and with group( are literally the most wanted things for me in recent years. Thanks!

I like the idea of @iamale with static files. It does all that original gist does but without executing of the Pipfile. Do we really need yet another setup.py with well-known flaws?

I really don't like the idea of it being a python script. I see so many setup.py scripts that are unreadable or unstable because people don't understand Setuptools or think they need something special and write the craziest script logic around something that (in my eyes) should be declarative.

Shouldn't implicit dependencies be included in the lock files as well à la pip freeze?

@charettes You bet! At least yarn.lock works this way, and I guess other lockfiles do, too.

@nils-werner

[...] should be declarative.

And just a few days later pypa/setuptools#862 has appeared %)