Skip to content

Instantly share code, notes, and snippets.

@ken-reitz ken-reitz/Distfile.py Secret

forked from dstufft/Distfile.py
Created Nov 18, 2016
Embed
What would you like to do?
"Automatically Generated, Not for Editing"
"Acts as a Lock file"
"Uses a List, but is actually a set"
"All packages without a group are in the default group"
{
"_meta": {
"sources": [
{"url": "https://simple.crate.io/"},
{"url": "https://pypi.python.org/simple/", "verify_ssl": false},
]
},
"default": [
{"name": "requests", "version": "0.11.2", "hash": "...."},
{"name": "Django", "version": "1.4", "hash": "..."},
{"name": "pinax", "git": "git://....", "branch": "1.4"},
{"name": "crate", "path": "~/blech", "editable": true}
],
"development": [
{"name": "test", "version": "0.1", "hash": "..."},
{"name": "test2", "version": "2.0", "hash": "..."},
{"name": "another thing", "version": "3.5", "hash": "..."}
],
"testing": [
{"name": "test", "version": "0.1", "hash": "..."}
]
}
# This is designed to be edited by hand by python developers
# --index-url and friends look like command options, non inutive. No extra metadata available
source("https://simple.crate.io/")
source("https://pypi.python.org/simple/", verify_ssl=False)
# Design:
# - Use real datastructures, make things clearer
# - Seperate package name from version, using real strings.
# Django==1.4 is a valid PyPI package, uninstallable from current files
# - using kwargs creates a great way to provide optional options on a line by line basis
# that python programmers are already familar with
# - People should only have to worry about the things they depend on, the installer
# should do the right thing
# - People have different dependency based on environment
# - Allow advanced usage for "wierd use cases" or "patterns not anticipated"
# Concerns:
# - Using Python file might cause the same problems as setup.py
# - This File not designed to be directly executed
# - setup.py is typically sent to other people, requirements are typically for internal use
# - Final result is still deterministic
# - Somewhat more verbose
# - Uses a syntax familar with all python programmers
dist("requests") # Install the latest version of requests, and all dependency's
dist("Django", "==1.4") # Install a version of Django matching the "==1.4" version spec and all dependencies
dist("pinax", git="git://github.com/pinax/pinax.git", branch="1.4") # Install pinax, using git and the 1.4 branch
dist("crate", path="~/blech", editable=True) # Install crate from the supplied path, and install it as an editable
dist("test", group="development") # install test, but only if the development group is passed
dist("test2", group=["development", "testing"]) # install test2, but only if the development or testing group is passed
with group("development"): # start a context that is equivilant to passing everything a certain group
dist("another thing") # install ``another thing`` if the development group is passed
with source("https://inhouse.example.com/"):
# Things in here MUST be installed from the above source, the idea being that if you have forked, e.g. django-model-utils
# you can uplaod to an inhouse server and force django-model-utils to install from this source, even in the case
# there is another version higher then yours, or even the same version.
# Additionally if packages installed inside of a with source(): have dependencies their deps are searched for
# on source, and installed from there if possible. However if they are not available dependencies will fall back
# to the global source()es.
dist("django-model-utils") # Not shown in the json lockfile.
# Details
# - This file does NOT cause anything to directly get installed, the program uses it to build up an internal list of requirements
# - All dist()'s are considered for the requirements, even if they are not going to get installed (e.g. they are in a not currently active group)
# - This will allow things to work smoothly where the requirement in a group might modify the final installed version (e.g. a group might pin to Django<1.4)
# - This file will be "compiled" down to json.
# - When the compiled json exists, there is no need to do any sort of version resolving. We know exactly what it is we want to install. (e.g. --no-deps etc)
# - We still need to find out where to download the packages from? (Unless we encode that in the json. Might be risky)
# - If there's a corner case not thought of, file is still Python and allows people to easily extend
# - You don't need to pin to exact versions, you just need to pin to what you want to support. e.g. if a pacakge follows semver, you could do package>1.2.1,<1.3 (you first started using the 1.2.1 version, and you expect 1.2.X to always be good for you.
# - Exact versions are pinned automatically in the json file
@ken-reitz

This comment has been minimized.

Copy link
Owner Author

ken-reitz commented Nov 18, 2016

@Julian

This comment has been minimized.

Copy link

Julian commented Nov 20, 2016

Is the idea that this is actually Python, or just Python-like

@notpushkin

This comment has been minimized.

Copy link

notpushkin commented Nov 20, 2016

@Julian I guess it's at least Python-parseable.

Although it looks a bit too verbose to me, eh?

@notpushkin

This comment has been minimized.

Copy link

notpushkin commented Nov 20, 2016

I was thinking about something more like package.json style actually:

dependencies:
  six: ^1.10.0  # semver extended syntax, finally in the python world
  graphql-core: >=1.0
  graphql-relay: >=0.4.4
  promise: *
  # These two packages are installed from git:
  somepackage: iamale/somepackage  # shorthand for GitHub
  anotherpkg: git+https://bitbucket.org/iamale/anotherpkg

dev_dependencies:
  another-thing: *

test_dependencies:
  pytest: >=2.7.2

extras:
  django:
    graphene-django: *
  sqlalchemy:
    graphene-sqlalchemy: *

I'm toying with the possibility of making a Pip wrapper that reads such syntax, but I guess it's not what Python community needs right now (we already have setup.py, requirements.txt and now also Pipfile and pyproject.toml — already too many tools for such a simple task).

@proofit404

This comment has been minimized.

Copy link

proofit404 commented Nov 21, 2016

with source( and with group( are literally the most wanted things for me in recent years. Thanks!

@9seconds

This comment has been minimized.

Copy link

9seconds commented Nov 21, 2016

I like the idea of @iamale with static files. It does all that original gist does but without executing of the Pipfile. Do we really need yet another setup.py with well-known flaws?

@nils-werner

This comment has been minimized.

Copy link

nils-werner commented Nov 21, 2016

I really don't like the idea of it being a python script. I see so many setup.py scripts that are unreadable or unstable because people don't understand Setuptools or think they need something special and write the craziest script logic around something that (in my eyes) should be declarative.

@charettes

This comment has been minimized.

Copy link

charettes commented Nov 21, 2016

Shouldn't implicit dependencies be included in the lock files as well à la pip freeze?

@notpushkin

This comment has been minimized.

Copy link

notpushkin commented Nov 21, 2016

@charettes You bet! At least yarn.lock works this way, and I guess other lockfiles do, too.

@idlesign

This comment has been minimized.

Copy link

idlesign commented Jan 27, 2017

@nils-werner

[...] should be declarative.

And just a few days later pypa/setuptools#862 has appeared %)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.